Repository navigation
Parallel host-tool installs race while cleaning the shared sysroot #372
Description
Description:
Host-tool packages are installed in parallel with make -j$(nproc), but package install rules also remove files from the shared host sysroot, including share/man. One install can therefore remove a file another still needs. Deferring this cleanup until the parallel build finishes should avoid the race.
Log:
2026-10-25T08:33:34.7580706Z target:.../work/build/zstd/zstd-1.5.7.install
2026-10-25T08:33:34.7582041Z run:env CC=aarch64-linux-gnu-gcc make -C .../zstd-1.5.7/. install ... PREFIX=/usr DESTDIR=.../work/build/host/aarch64-linux-gnu
2026-10-25T08:33:34.7583486Z Installing includes
2026-10-25T08:33:34.7583717Z Installing static library
2026-10-25T08:33:34.7583960Z Installing shared library
2026-10-25T08:33:34.7584477Z Installing binaries
2026-10-25T08:33:34.7584678Z Installing man pages
2026-10-25T08:33:34.7585406Z install: cannot create regular file '.../work/build/host/aarch64-linux-gnu/usr/share/man/man1/zstd.1': No such file or directory
2026-10-25T08:33:34.7586199Z make[3]: *** [Makefile:423: install] Error 1
2026-10-25T08:33:34.7586495Z make[2]: *** [Makefile:185: install] Error 2
2026-10-25T08:33:34.7587719Z make: *** [.../work/build/pkgdeps/zstd.deps.mk:25: .../work/build/zstd/zstd-1.5.7.install] Error 2
2026-10-25T08:33:34.7588598Z make: *** Waiting for unfinished jobs....
2026-10-25T08:33:37.4320704Z CONFIGURE genimage 20
2026-10-25T08:33:40.0701211Z BUILD genimage 20
2026-10-25T08:33:40.2726702Z INSTALL genimage 20
2026-10-25T08:33:40.3785271Z Error: ... make -s -j20 -C .../package bdebstrap genimage zstd ... (2)
2026-10-25T08:33:40.3803215Z ##[error]Process completed with exit code 1.
Reproduction:
We build rpi-image-gen v2.9.0 inside an LXC container on an NVIDIA DGX Spark, which has a 20-core Grace CPU. The race occurs roughly once every few dozen runs in this setup. We haven’t reproduced it on a Raspberry Pi; its lower core count likely makes the race less likely.
Activity
learmj commentedon Oct 26, 2026
Hi @kszonek Thank you for the report and feedback. The environment description is helpful. The bug is genuine - I can see that the common delete of
share/man,share/doc, etc per-build scheme could easily conflict with another pkg's install ops, eg zstd creates$(MAN1DIR), installs$(DESTDIR)$(MAN1DIR)/zstd.1but another pkg's cleanup could run between those too and removeshare/man. I think that's the error in your snippet. I have run builds on a couple of different machines, both with with more cores than an pi5, and I've never seen this before. However, parallel Make has a way of surfacing real edge cases and the gap you've highlighted is definitely something that needs to be fixed.I've got a few commits already lined up for the package mini-buildsys that we need for other things, so I will fix this on top of those. However, while I appreciate your PR, it's not the right way to address the problem. The buildsys itself should take care of all operations - it shouldn't be for a 'caller' (ie lib/tools.sh) to clean up after the build has completed. I won't be accepting the PR because of this. I think the buildsys needs to gain an internal 'finalise' stage, eg something that waits for all stamps and then cleans up.
kszonek commentedon Oct 26, 2026
Thanks for fast reply. I needed a way to stop CI from failing, for now i am using my fork for the builds, but looking forward for a proper upstream fix :)
I imagined my solution is not a proper way, but only one I could think off to move on.
About repro:
I see this issue on 20 core 2Ghz arm64 (Nvidia DGX Spark). I know it a crime to use it as a build host for rpi images, but it is doing some other work too :) My first solution was to decrease LXC core count to 1, but it had side effect of taking way too much time (for a pipline to release the runner) and this host can not be hijact for too long.
For others experiencing this issue: if lowering core count is acceptable for you, it will allow to work around this issue while staying on upstream.
@learmj I will watch your work closely :)
learmj commentedon Oct 26, 2026
@kszonek I have something that I think works. It would be great if you could test it on your Spark build box.
I would like to credit you on the public commit, particularly as (1) you highlighted the issue, (2) you suggested the fix, and (3) I didn't take your PR.
Is there a particular address/id you'd like me to use?
kszonek commentedon Oct 26, 2026
Sure, I can switch to different branch and easily rollback in case of issues.
You can credit me as
kszonek <may-not-reply@kszonek.pl>Thanks.
package: Prune runtime tree only after all pkgs have installed
learmj commentedon Oct 26, 2026
Thanks. Please could you try the tip of https://github.com/learmj/rpi-image-gen/tree/feat/srcbuild and see if you can reproduce the problem?
learmj commentedon Oct 27, 2026
@kszonek Did you manage to run a test build on your Spark?
learmj commentedon Oct 27, 2026
@kszonek Did you manage to run a test build on your Spark?
kszonek commentedon Oct 27, 2026
@learmj
Yes, I deployed your branch yesterday.
I looked through logs and there was around 100 jobs for different targets:
One failure - failed download of library sources. Probably some infra internal network issue
Several failures - I accidentaly did not reprovision one of the runners - this feature branch now requires cmake. so this was expected, that one runner gave me lots of red.
So far so good. ~90 jobs succesfull, no failures caused by concurency bug.
learmj commentedon Oct 27, 2026
Thanks @kszonek Let's see how things are in a couple of days time, or when you are satisfied it's fixed.
learmj commentedon Oct 30, 2026
@kszonek I'd like to merge #375 today if possible. Any objections? I assume no news is good news..?
kszonek commentedon Oct 30, 2026
@learmj No news is good news indeed! No failures observed so no objections from me.