(that is, call the build hook with a certain interval until it
accepts the build).
* build-remote.pl was totally broken: for all system types other than
the local system type, it would send all builds to the *first*
machine of the appropriate type.
poll for it (i.e. if we can't acquire the lock, then let the main
select() loop wait for at most a few seconds and then try again).
This improves parallelism: if two nix-store processes are both
trying to build a path at the same time, the second one shouldn't
block; it should first see if it can build other goals. Also, it
prevents the deadlocks that have been occuring in Hydra lately,
where a process waits for a lock held by another process that's
waiting for a lock held by the first.
The downside is that polling isn't really elegant, but POSIX doesn't
provide a way to wait for locks in a select() loop. The only
solution would be to spawn a thread for each lock to do a blocking
fcntl() and then signal the main thread, but that would require
pthreads.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
hook. This fixes a problem with log files being partially or
completely filled with 0's because another nix-store process
truncates the log file. It should also be more efficient.
the DerivationGoal runs. Otherwise, if a goal is a top-level goal,
then the lock won't be released until nix-store finishes. With
--keep-going and lots of top-level goals, it's possible to run out
of file descriptors (this happened sometimes in the build farm for
Nixpkgs). Also, for failed derivation, it won't be possible to
build it again until the lock is released.
* Idem for locks on build users: these weren't released in a timely
manner for failed top-level derivation goals. So if there were more
than (say) 10 such failed builds, you would get an error about
having run out of build users.
scan for runtime dependencies (i.e. the local machine shouldn't do a
scan that the remote machine has already done). Also pipe directly
into `nix-store --import': don't use a temporary file.
(e.g. an SSH connection problem) and permanent failures (i.e. the
builder failed). This matters to Hydra (it wants to know whether it
makes sense to retry a build).
closure of the inputs. This really enforces that there can't be any
undeclared dependencies on paths in the store. This is done by
creating a fake Nix store and creating bind-mounts or hard-links in
the fake store for all paths in the closure. After the build, the
build output is moved from the fake store to the real store. TODO:
the chroot has to be on the same filesystem as the Nix store for
this to work, but this isn't enforced yet. (I.e. it only works
currently if /tmp is on the same FS as /nix/store.)
bind-mounts we do are only visible to the builder process and its
children. So accidentally doing "rm -rf" on the chroot directory
won't wipe out /nix/store and other bind-mounted directories
anymore. Also, the bind-mounts in the private namespace disappear
automatically when the builder exits.
necessary that at least one build hook doesn't return "postpone",
otherwise nix-store will barf ("waiting for a build slot, yet there
are no running children"). So inform the build hook when this is
the case, so that it can start a build even when that would exceed
the maximum load on a machine.
subtle and often hard-to-reproduce bugs where programs in pipes
either barf with a "Broken pipe" message or not, depending on the
exact timing conditions. This particularly happened in GNU M4 (and
Bison, which uses M4).
disasters involving `rm -rf' on bind mounts. Will try the
definitive fix (per-process mounts, apparently possible via the
CLONE_NEWNS flag in clone()) some other time.
particular, dietlibc cannot figure out the cwd because the inode of
the current directory doesn't appear in .. (because getdents returns
the inode of the mount point).
need /etc in the chroot (in particular, /etc/resolv.conf for
fetchurl). Not having /etc/resolv.conf in the chroot is a good
thing, since we don't want normal derivations to download files.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
/tmp/nix-<pid>-<counter> for temporary build directories. This
increases purity a bit: many packages store the temporary build path
in their output, causing (generally unimportant) binary differences.
Eelco Dolstra
Ludovic Courtès
Michael Raskin