Previously, we'd spawn 'guix authenticate' once for each item that has
to be signed (when exporting) or authenticated (when importing). Now,
we spawn it once for all and then follow a request/reply protocol. This
reduces the wall-clock time of:
guix archive --export -r $(guix build coreutils -d)
from 30s to 2s.
* guix/scripts/authenticate.scm (sign-with-key): Return the signature
instead of displaying it. Raise a &formatted-message instead of calling
'leave'.
(validate-signature): Likewise.
(read-command): New procedure.
(define-enumerate-type, reply-code): New macros.
(guix-authenticate)[send-reply]: New procedure.
Change to read commands from current-input-port.
* nix/libstore/local-store.cc (runAuthenticationProgram): Remove.
(authenticationAgent, readInteger, readAuthenticateReply): New
functions.
(signHash, verifySignature): Rewrite in terms of the agent.
* tests/store.scm ("import not signed"): Remove 'pk' call.
("import signed by unauthorized key"): Check the error message of C.
* tests/guix-authenticate.sh: Rewrite using the new protocol.
fixlet
* nix/libstore/build.cc (DerivationGoal::tryBuildHook): Add "offload" to
'args' and pass settings.guixProgram as the first argument to
Agent::Agent.
(pathNullDevice, commonChildInit, Agent, Agent::Agent)
(Agent::~Agent): Move to...
* nix/libutil/util.cc: ... here.
* nix/libutil/util.hh (struct Agent, commonChildInit): New
declarations.
* nix/libstore/local-store.cc (signHash, verifySignature): New
functions.
(LocalStore::exportPath): Use 'signHash' instead of inline code.
(LocalStore::importPath): Use 'verifySignature' instead of inline code.
* nix/libstore/build.cc (HookInstance): Rename to...
(Agent): ... this. Rename 'toHook' and 'fromHook' similarly and update
users. Change constructor to require a command and an argument list.
(DerivationGoal::tryBuildHook): Pass arguments to the 'Agent'
constructor.
There's no reason at this point to mimic the calling convention of the
'openssl' command.
* nix/libstore/local-store.cc (LocalStore::exportPath): Add only "sign"
and HASH to ARGS. Remove 'tmpDir' and 'hashFile'.
(LocalStore::importPath): Add only "verify" and SIGNATURE to
* guix/scripts/authenticate.scm (guix-authenticate): Adjust
accordingly; remove the OpenSSL-style clauses.
(read-hash-data): Remove.
(sign-with-key): Replace 'port' with 'sha256' and adjust accordingly.
(validate-signature): Export SIGNATURE to be a canonical sexp.
* tests/guix-authenticate.sh: Adjust tests accordingly.
Suggested by Caleb Ristvedt <caleb.ristvedt@cune.org>.
* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Save errno
from 'rename' before calling 'unlink'.
Remove .gitignore entries where they match source files that are tracked in
Git. This is relevant to me at least, as some code searching tools use
.gitignore files and will ignore matched files.
Fixes <https://bugs.gnu.org/41607>.
Reported by Stephen Scheck <singularsyntax@gmail.com>.
* nix/libstore/gc.cc (LocalStore::deletePathRecursive): When we try to
move a dead directory into the trashDir using rename(2) but it returns
an EXDEV error, just delete the directory instead. This can happen in a
Docker container when the directory is not on the "top layer".
This follows up on 1a1faa78b0, and avoids
the (non-fatal) error seen in <https://issues.guix.gnu.org/41356>.
/gnu/store will remain writable on new foreign distribution
installations until the next release.
* etc/guix-install.sh (sys_enable_guix_daemon): Check for
‘gnu-store.mount’ presence before trying to cp it.
Update forgotten copyright header.
* etc/gnu-store.mount.in: New file.
* nix/local.mk (nodist_systemdservice_DATA): Add it.
(etc/%.mount): New rule for it.
* etc/guix-install.sh (sys_enable_guix_daemon): Install it.
* doc/guix.texi (Binary Installation): Document it.
* .gitignore: Ignore changes to it.
This allows for native builds on the Hurd, doing
sudo ./pre-inst-env guix-daemon --disable-chroot --build-users-group=guixbuild &
./pre-inst-env guix build hello
* nix/libutil/util.cc (killUser)[__GNU__]: Avoid kill -1 bug; kill only
current process and ignore SIGKILL status in parent.
Co-authored-by: Jan Nieuwenhuizen <janneke@gnu.org>
Checking for CLONE_NEWNS is only needed for using tha Linux specific clone(2),
otherwise we can use fork(2). Using clone on the Hurd needs some work, only
support LINUX for now. See
https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00190.html
* nix/libstore/build.cc (CHROOT_ENABLED): Break into CHROOT_ENABLED
and CLONE_ENABLED.
(DerivationGoal::startBuilder): Replace CHROOT_ENABLED with __linux__.
(DerivationGoal::runChild): Only define pivot_root() if SYS_pivot_root is
defined.
Co-authored-by: Jan Nieuwenhuizen <janneke@gnu.org>
Fixes <https://bugs.gnu.org/39727>.
Reported by Paul Garlick <pgarlick@tourbillion-technology.com>.
* nix/libstore/gc.cc (LocalStore::removeUnusedLinks) [HAVE_STATX]: Add
'statx_flags' static variables. Clear 'AT_STATX_DONT_SYNC' flag from
'statx_flags' when 'statx' returns EINVAL.
Previously, a store item that is a regular file would not be accounted
for in the 'bytesFreed' value computed by 'deletePath' because its
'st_nlink' count would always be >= 2. This commit fixes that.
* nix/libutil/util.hh (deletePath): Add optional 'linkThreshold' argument.
* nix/libutil/util.cc (_deletePath): Add 'linkThreshold' argument and
honor it. Pass it down in recursive call.
(deletePath): Add 'linkThreshold' and honor it.
* nix/libstore/gc.cc (LocalStore::deleteGarbage): Pass 'linkThreshold'
argument to 'deletePath', with a value of 2 when PATH is a store item
and deduplication is on.
This is a followup to bc69ea2d60.
* guix/scripts/build.scm (show-build-options-help): Rename
"--no-build-hook" to "--no-offload".
(%standard-build-options): Likewise, and warn when "--no-build-hook" is
passed.
* nix/nix-daemon/guix-daemon.cc (options): Add "--no-offload" and mark
"--no-build-hook" as hidden.
* guix/scripts/offload.scm: Adjust comment.
* doc/guix.texi (Invoking guix-daemon, Common Build Options): Replace
"--no-build-hook" with "--no-offload".
* etc/completion/fish/guix.fish, etc/completion/zsh/_guix: Adjust
accordingly.
* nix/libstore/gc.cc (LocalStore::deletePathRecursive): Display the
percentage reached relative to 'maxFreed', or the total amount of data
deleted when 'maxFreed' is ULLONG_MAX.
As of GNU libc 2.29, <sys/mount.h> declares all the constants and
functions we need, so there's no use in including <linux/fs.h> anymore.
This silences annoying warnings like this one:
In file included from nix/libstore/local-store.cc:32:0:
/gnu/store/…-linux-libre-headers-4.19.56/include/linux/fs.h:108:0: warning: "MS_RDONLY" redefined
#define MS_RDONLY 1 /* Mount read-only */
In file included from nix/libstore/local-store.cc:28:0:
/gnu/store/…-glibc-2.29/include/sys/mount.h:36:0: note: this is the location of the previous definition
#define MS_RDONLY MS_RDONLY
* config-daemon.ac: Remove check for <linux/fs.h>.
* nix/libstore/build.cc: Remove conditional inclusion of <linux/fs.h>.
* nix/libstore/local-store.cc: Remove "#if HAVE_LINUX_FS_H" and
inclusion of <linux/fs.h>.
Fixes <https://bugs.gnu.org/38062>.
This is a followup to ada9a19a2d.
* nix/libstore/build.cc (DerivationGoal::killChild): Add conditional
call to 'worker.childTerminated' for 'hook->pid'.
Fixes <https://bugs.gnu.org/37744>.
Reported at <https://www.openwall.com/lists/oss-security/2019/10/09/4>.
Based on Nix commit 5a303093dcae1e5ce9212616ef18f2ca51020b0d
by Eelco Dolstra <edolstra@gmail.com>.
* nix/libstore/local-store.cc (LocalStore::LocalStore): Set 'perUserDir'
to #o755 instead of #o1777.
(LocalStore::createUser): New function.
* nix/libstore/local-store.hh (LocalStore): Add it.
* nix/libstore/store-api.hh (StoreAPI): Add it.
* nix/nix-daemon/nix-daemon.cc (performOp): In 'wopSetOptions', add
condition to handle "user-name" property and honor it.
(processConnection): Add 'userId' parameter. Call 'store->createUser'
when userId is not -1.
* guix/profiles.scm (ensure-profile-directory): Note that this is now
handled by the daemon.
* guix/store.scm (current-user-name): New procedure.
(set-build-options): Add #:user-name parameter and pass it to the daemon.
* tests/guix-daemon.sh: Test the creation of 'profiles/per-user' when
listening on a TCP socket.
* tests/store.scm ("profiles/per-user exists and is not writable")
("profiles/per-user/$USER exists"): New tests.
This is a followup to 2e3e5d2198.
* build-aux/test-env.in: Remove mentions of 'NIX_ROOT_FINDER'.
* nix/libstore/gc.cc (LocalStore::collectGarbage): Adjust comment
accordingly.
Until now it was up to 'guix offload' to honor timeouts. Unfortunately
it would sometimes fail to do that, for example due to the libssh bug at
<https://bugs.libssh.org/T33>. With this change, 'guix offload' is
automatically killed by the daemon when one of the timeouts expires.
Thus, data transfers performed by 'guix offload' now count as part of
the timeouts, rather than just actual build time.
* nix/libstore/build.cc (DerivationGoal::tryBuildHook): Pass true as the
'respectTimeouts' argument to 'childStarted'.
The daemon had a mechanism that allows it to handle a list of
substituters and try them sequentially; this removes it.
* nix/scripts/substitute.in: Remove.
* nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove.
* config-daemon.ac: Don't output 'nix/scripts/substitute'.
* nix/libstore/build.cc (SubstitutionGoal)[subs, sub, hasSubstitute]:
Remove.
[tryNext]: Make private.
(SubstitutionGoal::SubstitutionGoal, SubstitutionGoal::init): Remove now
unneeded initializers.
(SubstitutionGoal::tryNext): Adjust to assume a single substituter: call
'amDone' upfront when we couldn't find substitutes.
(SubstitutionGoal::tryToRun): Adjust to run 'guix substitute' via
'settings.guixProgram'.
(SubstitutionGoal::finished): Call 'amDone(ecFailed)' upon failure
instead of setting 'state' to 'tryNext'.
* nix/libstore/globals.hh (Settings)[substituters]: Remove.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Adjust to
handle a single substituter.
(LocalStore::startSubstituter): Remove 'path' parameter. Adjust to
invoke 'settings.guixProgram'. Don't refer to 'run.program', which no
longer exists.
(LocalStore::querySubstitutablePaths): Adjust for 'runningSubstituters'
being a singleton instead of a list.
(LocalStore::querySubstitutablePathInfos): Likewise, and remove
'substituter' parameter.
* nix/libstore/local-store.hh (RunningSubstituter)[program]: Remove.
(LocalStore)[runningSubstituters]: Remove.
[runningSubstituter]: New field.
[querySubstitutablePathInfos]: Remove 'substituter' parameter.
[startSubstituter]: Remove 'substituter' parameter.
* nix/nix-daemon/guix-daemon.cc (main): Remove references to
'settings.substituters'.
* nix/nix-daemon/nix-daemon.cc (performOp): Ignore the user's
"build-use-substitutes" value when 'settings.useSubstitutes' is false.
Until now, something like "guix build sed -v1 --check" would not get a
'build-succeeded' event, which in turn meant that the spinner would not
be erased upon build completion.
* nix/libstore/build.cc (DerivationGoal::registerOutputs): When
'buildMode' is bmCheck and 'settings.printBuildTrace' emit a
"@ build-succeeded" trace upon success.
* tests/store.scm ("build-succeeded trace in check mode"): New test.
This makes it easier to run the uninstalled daemon.
* nix/local.mk (libstore_a_CPPFLAGS): Append "/guix" to
NIX_LIBEXEC_DIR.
* build-aux/pre-inst-env.in (NIX_LIBEXEC_DIR): Adjust comment.
* nix/libstore/builtins.cc (builtinDownload): Remove SUBDIR and its
use.
* nix/libstore/local-store.cc (runAuthenticationProgram): Ditto.
* nix/libstore/gc.cc (addAdditionalRoots): Remove "/guix" prefix.
* nix/nix-daemon/guix-daemon.cc (main): Ditto.
* nix/nix-daemon/nix-daemon.cc (performOp) <wopSetOptions>: When
'isRemoteConnection' is true, set 'settings.keepFailed' to zero.
* doc/guix.texi (Common Build Options): Document this behavior.
That way it is handled in the same way as other helper scripts.
* nix/scripts/guix-authenticate.in: Rename to...
* nix/scripts/authenticate.in: ... this.
* config-daemon.ac: Adjust accordingly.
* nix/local.mk (libstore_a_CPPFLAGS): Remove -DOPENSSL_PATH.
(nodist_libexec_SCRIPTS): Remove.
(nodist_pkglibexec_SCRIPTS): New variable.
* nix/nix-daemon/guix-daemon.cc (main): Remove 'setenv' call for
"PATH".
* nix/libstore/local-store.cc (runAuthenticationProgram): New function.
(LocalStore::exportPath, LocalStore::importPath): Use it instead of
'runProgram' and OPENSSL_PATH.
This allows clients to tell whether output comes from the daemon or, if
it comes from a builder, from which builder it comes. The latter is
particularly useful when MAX-BUILD-JOBS > 1.
* nix/libstore/build.cc (DerivationGoal::tryBuildHook)
(DerivationGoal::startBuilder): Print the child's PID in "@ build-started"
traces.
(DerivationGoal::handleChildOutput): Define 'prefix', pass it to
'writeToStderr'.
* nix/libstore/globals.cc (Settings:Settings): Initialize
'multiplexedBuildOutput'.
(Settings::update): Likewise.
* nix/libstore/globals.hh (Settings)[multiplexedBuildOutput]: New field.
Update 'printBuildTrace' documentation.
* nix/libstore/worker-protocol.hh (PROTOCOL_VERSION): Bump to 0.163.
* nix/nix-daemon/nix-daemon.cc (performOp) <wopSetOptions>: Special-case
"multiplexed-build-output" and remove "use-ssh-substituter".
* guix/store.scm (set-build-options): Add #:multiplexed-build-output?
and honor it.
(%protocol-version): Bump to #x163.
* tests/store.scm ("multiplexed-build-output"): New test.
fixlet
This code has never been of any use in Guix.
* nix/libutil/archive.cc (useCaseHack): Remove.
(parse): Keep only the alternate branch in "if (useCaseHack)".
* guix/scripts/perform-download.scm (perform-download): Add
#:print-build-trace? and pass it to 'url-fetch'.
(guix-perform-download): Define 'print-build-trace?' and pass it to
'perform-download'.
* guix/build/download.scm (ftp-fetch): Add #:print-build-trace? and
honor it.
(url-fetch): Likewise.
* nix/libstore/builtins.cc (builtinDownload): Set _NIX_OPTIONS
environment variable.
* nix/libstore/build.cc (canBuildLocally): Allow building armhf-linux
builds on aarch64-linux.
(DerivationGoal::runChild) Throw error if attempting to build for
armhf-linux on an unsupported platform.
* doc/guix.texi (Invoking guix build): Document how to build natively
for armhf-linux on aarch64-linux. Add note that on some aarch64
machines this is unsupported.
* nix/nix-daemon/nix-daemon.cc (isRemoteConnection): New variable.
(performOp): For wopCollectGarbage, throw an error when isRemoteConnection
is set.
(acceptConnection): Set isRemoteConnection when connection is not AF_UNIX.
* tests/guix-daemon.sh: Add a test for the new behavior.
* config-daemon.ac: Don't bail out when libbz2 is missing. Define
'HAVE_LIBBZ2' Automake conditional.
* nix/libstore/build.cc: Wrap relevant bits in '#if HAVE_BZLIB_H'.
* nix/libstore/globals.cc (Settings::Settings): 'logCompression'
defaults to COMPRESSION_GZIP when HAVE_BZLIB_H is false.
* nix/libstore/globals.hh (CompressionType): Make 'COMPRESSION_BZIP2'
conditional on HAVE_BZLIB_H.
* nix/local.mk (guix_register_LDADD, guix_daemon_LDADD): Add -lbz2 only
when HAVE_LIBBZ2.
* nix/nix-daemon/guix-daemon.cc (parse_opt): Ignore "bzip2" when not
HAVE_BZLIB_H.
Fixes <https://bugs.gnu.org/29862>.
Reported by Danny Milosavljevic <dannym@scratchpost.org>.
* nix/scripts/list-runtime-roots.in (canonicalize-store-item): Define
'store' with a trailing "/". Have the 'string-prefix?' call match the
'string-drop' call.
This allows running as non-root.
Fixes a regression introduced in b8f59cdc20.
* nix/scripts/list-runtime-roots.in (referenced-files): Handle EACCES in
addition to ENOENT.
Looking at 'addAdditionalRoots' in libstore/gc.cc, it looks like it
should always have been this way. In practice it probably doesn't make
much of a difference.
* nix/scripts/list-runtime-roots.in (canonicalize-store-item): New
procedure.
<top level>: Use it.
This makes things a bit faster (0.8s instead of 1.4s on my laptop).
* nix/scripts/list-runtime-roots.in (lsof-roots): Remove.
(proc-fd-roots): Return the empty list when 'scandir' returns #f.
(referenced-files): New procedure.
Use it at the top level.
Prior to this change, errors such as wrong permissions on
/etc/guix/signing-key.sec would give:
guix-daemon: nix/libutil/serialise.cc:15: virtual nix::BufferedSink::~BufferedSink(): Assertion `!bufPos' failed.
This patch correctly propagates the error to the client and thus changes
that to:
error: build failed: file `/etc/guix/signing-key.sec' should be secret (inaccessible to everybody else)!
* nix/nix-daemon/nix-daemon.cc (performOp): Wrap 'exportPath' call in
'try' block.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>
Reported by Ricardo Wurmus.
* nix/nix-daemon/nix-daemon.cc (acceptConnection): Remove intermediate
'inaddr' variables that relied on implicit casts. This fixes
compilation with GCC 4.9.
* nix/nix-daemon/guix-daemon.cc (DEFAULT_GUIX_PORT): New macro.
(listen_options): New variable.
(parse_opt): Push back '--listen' options to LISTEN_OPTIONS.
(open_unix_domain_socket, open_inet_socket)
(listening_sockets): New functions.
(main): Use it. Pass SOCKETS to 'run'.
* nix/nix-daemon/nix-daemon.cc (matchUser): Remove.
(SD_LISTEN_FDS_START): Remove.
(acceptConnection): New function.
(daemonLoop): Rewrite to take a vector of file descriptors, to select(2)
on them, and to call 'acceptConnection'.
(run): Change to take a vector of file descriptors.
* tests/guix-daemon.sh: Add test.
Otherwise, users will be stuck running an old copy of guix and the guix-daemon
if they copy the service files instead of symlinking them.
* etc/guix-daemon.conf.in, etc/guix-daemon.service.in, etc/guix-publish.conf.in,
etc/guix-publish.service.in: Expand @localstatedir@ instead of @bindir@.
* nix/local.mk (etc/guix-%.service, etc/guix-%.conf): Use @localstatedir@
instead of @bindir@.
Reported by rohit yadav <rohityadav@utexas.edu>
at <https://lists.gnu.org/archive/html/guix-devel/2017-02/msg00191.html>.
This fixes a regression whereby 'guix perform-download' would always see
NIX_STORE as unset and thus use "/gnu/store", leading it to miscompute
the hydra.gnu.org content-addressed URLs when the store file name is not
"/gnu/store".
* nix/libstore/builtins.cc (builtinDownload): Add 'setenv' call for
'NIX_STORE'.
Fixes <http://bugs.gnu.org/20217>.
* nix/libstore/worker-protocol.hh (PROTOCOL_VERSION): Bump to 0x161.
* nix/nix-daemon/nix-daemon.cc (performOp): "build-max-jobs",
"build-max-silent-time", and "build-cores" are no longer read upfront;
instead, read them from the key/value list at the end.
* nix/nix-daemon/guix-daemon.cc (main): Explicitly set
'settings.maxBuildJobs'.
* guix/store.scm (%protocol-version): Bump to #x161.
(set-build-options): #:max-build-jobs, #:max-silent-time, and
#:build-cores now default to #f. Adjust handshake to new protocol.
* tests/store.scm ("build-cores"): New test.
* tests/guix-daemon.sh: Add test for default "build-cores" value.
Fixes <http://bugs.gnu.org/25242>.
Reported by Leo Famulari <leo@famulari.name>.
The regression was introduced in 94d92c7796.
* nix/libstore/build.cc (DerivationGoal::startBuilder): Set 'useChroot'
as a function 'of isBuiltin(drv)'.
(DerivationGoal::runChild): Use 'useChroot' instead of
'useChroot && !isBuiltin(drv)'.
Fixes <http://bugs.gnu.org/15890>.
* nix/libstore/globals.hh (Settings) Add clientUid and clientGid.
* nix/nix-daemon/nix-daemon.cc (daemonLoop] Store UID and GID of the
caller in settings.
* nix/libstore/build.cc (_chown): New function.
(DerivationGoal::deleteTmpDir): Use it, change ownership of build
directory if it is kept and the new owner is not root.
Before that we'd have STDERR_WRITE round trips for very small amounts of
data, ranging from a few bytes for the metadata of nars to the size of
one file being exported.
With this change, something like:
guix archive --export /gnu/store/5rrsbaghh5ix1vjcicsl60gsxilhjnf2-coreutils-8.25 | dd of=/dev/null
reports a throughput of 35 MB/s instead of 25 MB/s before.
* nix/nix-daemon/nix-daemon.cc (TunnelSink): Inherit from 'BufferedSink'
rather than 'Sink'. Rename 'operator ()' to 'write'.
(performOp) <wopExportPath>: Add 'sink.flush' call.
* .gitignore: add etc/guix-publish.conf and /etc/guix-publish.service.
* etc/guix-publish.conf.in: New file.
* etc/guix-publish.service.in: New file.
* nix/local.mk (etc/guix-%.service, etc/guix-%.conf): Generalized former
build-rules for by using patterns.
(nodist_systemdservice_DATA): Add etc/guix-publish.service, update
comment.
(nodist_upstartjob_DATA): Add etc/guix-publish.conf, update comment.
* doc/guix.texi (Invoking guix publish): Add description for enabling
"guix publish" on host distros using the new files.
This solves a problem whereby if /gnu/store/.links had enough entries,
ext4's directory index would be full, leading to link(2) returning
ENOSPC.
* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
ENOSPC from link(2), print a message and return instead of throwing a
'SysError'.
In particular, this eliminates a bunch of boilerplate code.
Also integrates these Nix commits:
80da7a6 Probably fix SQLITE_BUSY errors
37a337b throwSQLiteError(): Check for SIGINT so we don't loop forever
Co-authored-by: Ludovic Courtès <ludo@gnu.org>
