Four runners, one $HOME

I build a fleet of C++ projects on a self-hosted Windows runner in a Proxmox VM. For a long time it was one runner service and a queue. Then the queue got long, I registered three more services on the same VM, and three problems appeared that had been impossible with one.

None of them were runner bugs. All of them were assumptions that hold for one process and stop holding for four.

1. Four services, one profile directory

All four runner services run as NETWORK SERVICE. That account has exactly one profile, so $HOME for every runner is C:\Windows\ServiceProfiles\NetworkService.

One workflow step wrote a submodule deploy key to $HOME\.ssh\ and a later cleanup step deleted it. With one runner that is a tidy pair. With two jobs starting at the same moment, job A writes the key, job B writes the same key, job A finishes and deletes it, and job B fails to clone with:

[email protected]: Permission denied (publickey).

It looks exactly like a broken deploy key. It is a race, so it is sporadic, and it only ever hits one of the two jobs. I spent the first hour on the key.

The fix was one path in twenty workflows: Join-Path $env:RUNNER_TEMP ".ssh" instead of $HOME. RUNNER_TEMP is per job and cleaned afterwards. The rule generalises: anything a job puts on disk outside its workspace goes under RUNNER_TEMP, no exceptions.

2. Every job is a full build, and that is correct

The audit that preceded all this said “ImGui recompiles on every run”, estimated 15 to 20 seconds per job, and proposed a prebuilt static library checked into the tree.

Measured on an idle runner with six cores:

full build                 41 s
perfect incremental build  39 s
the seven ImGui units       ~3 s

The estimate was off by an order of magnitude, and the reason ImGui recompiles is that everything recompiles. actions/checkout runs with clean: true, which executes git clean -ffdx. The -x removes ignored files, which is the whole x64\ output directory including the tlogs. A prebuilt library in the tree would be deleted by the same command.

Turning clean off does not help either. An incremental Release build against a stale .ipdb from a previous LTCG run dies reproducibly with C1001 or C1354 corrupt ipdb. I reproduced it three times. The git clean is what keeps the pipeline green.

So the bottleneck was never the compile. It was the number of runners. The build was two seconds slower than it could be and the queue was twenty minutes long.

3. Disk fills with history nobody reads

A week after going to four runners, the VM went from 88 GB free to 25 GB. Each runner directory was 27 to 29 GB, almost all of it _work. Build output was only 2.4 GB per runner.

The rest was .git. Every workspace had about 710 MB of git history, because git submodule update --init --recursive pulls the full history of the shared SDK submodule and its own submodules. Twenty-seven repos times four runners times 0.7 GB is 75 GB of packfiles that no job ever looks at.

--depth 1 on the submodule step, in twenty-four workflows. After emptying the workspaces once: 68 MB of .git per workspace instead of 710, workspaces at 450 to 650 MB instead of 1.2 to 1.5 GB. Five projects went red after the change, and all five had been red before it for unrelated reasons, which is its own lesson about reading the previous run before blaming the current one.

Getting in without a password

A small thing that made all of the above faster. The VM has no WinRM password I care to type, but it has the QEMU guest agent, and the Proxmox host accepts my key. So:

function Invoke-Runner([string]$Script, [int]$Timeout = 120) {
    $b64 = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($Script))
    $raw = (ssh proxmox "qm guest exec 302 --timeout $Timeout -- powershell.exe -NoProfile -EncodedCommand $b64") -join "`n"
    ($raw | ConvertFrom-Json).'out-data'
}

The -EncodedCommand is the point. Without it, the chain PowerShell → ssh → bash → qm shreds every quote. Two traps: qm guest exec blocks until the command ends and times out on anything build-sized, so long commands go into Start-Process with a log file and you poll the file. And under build load the agent reports “not running” for a while. It is running. Ask again.

What I would tell myself before scaling

Scaling a runner from one to N is not a capacity change. It turns every “there is only one” assumption into a race, and every “this is small” assumption into a multiplication. Measure the build before optimising it, put job state under RUNNER_TEMP, and clone shallow.