Commit graph

3692 commits

Author SHA1 Message Date
Eelco Dolstra a98fa2d9e2 Allow regular files as GC roots
If a root is a regular file, then its name must denote a store
path. For instance, the existence of the file

  /nix/var/nix/gcroots/per-user/eelco/hydra-roots/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

would cause

  /nix/store/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

to be a root.

This is useful because it involves less I/O (no need for a readlink()
call) and takes up less disk space (the symlink target typically takes
up a full disk block, while directory entries are packed more
efficiently). This is particularly important for hydra.nixos.org,
which has hundreds of thousands of roots, and where reading the roots
can take 25 minutes.
2015-05-11 17:13:24 +02:00
Eelco Dolstra 4ab4b0c109 Remove some dead code 2015-05-11 17:11:38 +02:00
Eelco Dolstra 1cffdf5847 nix-daemon: Less verbosity 2015-05-11 17:11:27 +02:00
Eelco Dolstra bb07dfe96f nix-daemon: Simplify stderr handling 2015-05-11 17:11:14 +02:00
Ludovic Courtès 766481d606 Merge commit 'fdee1ced43fb495d612a29e955141cdf6b9a95ba' into nix 2015-05-11 17:04:26 +02:00
Ludovic Courtès c69944c511 Merge commit '8e9140cfdef9dbd1eb61e4c75c91d452ab5e4a74' into nix
Conflicts:
	Makefile.config.in
	configure.ac
	dev-shell
	doc/manual/builtins.xml
	doc/manual/conf-file.xml
	doc/manual/local.mk
	doc/manual/nix-instantiate.xml
	doc/manual/nix-store.xml
	doc/manual/writing-nix-expressions.xml
	misc/emacs/nix-mode.el
	perl/lib/Nix/CopyClosure.pm
	release.nix
	scripts/nix-build.in
	scripts/nix-copy-closure.in
	src/download-via-ssh/download-via-ssh.cc
	src/libexpr/common-opts.cc
	src/libexpr/common-opts.hh
	src/libexpr/eval.cc
	src/libexpr/eval.hh
	src/libexpr/local.mk
	src/libexpr/nixexpr.cc
	src/libexpr/nixexpr.hh
	src/libexpr/parser.y
	src/libexpr/primops.cc
	src/libexpr/symbol-table.hh
	src/libmain/shared.cc
	src/libstore/local.mk
	src/nix-env/nix-env.cc
	src/nix-instantiate/nix-instantiate.cc
	src/nix-store/local.mk
	src/nix-store/nix-store.cc
	src/nix-store/serve-protocol.hh
	tests/lang.sh
	tests/lang/eval-okay-context.nix
	tests/lang/eval-okay-search-path.exp
	tests/lang/eval-okay-search-path.nix
	tests/local.mk
	tests/nix-copy-closure.nix
2015-05-06 23:22:04 +02:00
Ludovic Courtès a1dd396cc0 Merge commit '8d5f472f2c49c79a0d3ae2e506f4d4d76224b328' into nix
Conflicts:
	.gitignore
	Makefile
	doc/manual/installation.xml
	doc/manual/introduction.xml
	doc/manual/local.mk
	doc/manual/manual.xml
	doc/manual/nix-collect-garbage.xml
	doc/manual/nix-env.xml
	doc/manual/nix-install-package.xml
	doc/manual/nix-store.xml
	doc/manual/quick-start.xml
	doc/manual/release-notes.xml
	local.mk
	misc/emacs/nix-mode.el
	mk/functions.mk
	mk/install.mk
	mk/lib.mk
	mk/libraries.mk
	mk/patterns.mk
	mk/programs.mk
	nix.spec.in
	release.nix
	scripts/install-nix-from-closure.sh
	scripts/nix-build.in
	src/libexpr/eval-inline.hh
	src/libexpr/eval.cc
	src/libexpr/eval.hh
	src/libexpr/get-drvs.cc
	src/libexpr/nixexpr.cc
	src/libexpr/nixexpr.hh
	src/libexpr/parser.y
	src/libexpr/primops.cc
	src/libstore/local.mk
	src/nix-daemon/local.mk
	src/nix-env/nix-env.cc
	src/nix-env/user-env.cc
	src/nix-instantiate/nix-instantiate.cc
	src/nix-store/nix-store.cc
	tests/init.sh
	tests/nix-copy-closure.nix
	tests/remote-builds.nix
	version
2015-01-04 23:01:14 +01:00
Ludovic Courtès 0a751260ae nix: Adjust code for Guix.
* nix/libstore/gc.cc (addAdditionalRoots): Refer to 'list-runtime-roots'
  instead of 'find-runtime-roots.pl'.
* nix/libutil/hash.cc, nix/libutil/hash.hh: Change 'union Ctx' to
  'struct Ctx', like 'nix/sync-with-upstream' did.
* nix/AUTHORS: New file.
* nix/COPYING: New file, from upstream Nix.
* nix/libutil/md32_common.h, nix/libutil/md5.c, nix/libutil/md5.h,
  nix/libutil/sha1.c, nix/libutil/sha1.h, nix/libutil/sha256.c,
  nix/libutil/sha256.h: Remove.
2014-12-17 22:57:33 +01:00
Ludovic Courtès d4e18b05e0 Keep only libstore, nix-daemon, and related stuff. 2014-12-17 22:57:04 +01:00
Eelco Dolstra fdee1ced43 startProcess: Make writing error messages from the child more robust 2014-07-23 19:11:26 +02:00
Eelco Dolstra 5989966ed3 Remove dead code 2014-07-23 14:46:28 +02:00
Eelco Dolstra ee3c5d7916 Revert old useBuildHook behaviour 2014-07-19 02:25:47 +02:00
Eelco Dolstra 2e77bd70fa Better fix for strcasecmp on Darwin 2014-07-18 12:54:30 +02:00
Eelco Dolstra f609eec71a Bump 2014-07-18 00:01:06 +02:00
Eelco Dolstra 8ddffe7aac Ugly hack to fix building on old Darwin
http://hydra.nixos.org/build/12580878
2014-07-17 23:57:17 +02:00
Eelco Dolstra 049c0eb49c nix-daemon: Add trusted-users and allowed-users options
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.

‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
2014-07-17 16:57:07 +02:00
Eelco Dolstra 0c730887c4 nix-daemon: Show name of connecting user 2014-07-17 15:49:33 +02:00
Eelco Dolstra 77c972c898 nix-daemon: Only print connection info if we have SO_PEERCRED 2014-07-17 15:44:51 +02:00
Eelco Dolstra 8f72e702a1 nix-daemon: Fix compat with older clients 2014-07-17 15:23:31 +02:00
Eelco Dolstra 2304a7dd21 Get rid of a compiler warning 2014-07-16 16:32:26 +02:00
Eelco Dolstra 985f1595fe Be more strict about file names in NARs 2014-07-16 16:30:50 +02:00
Eelco Dolstra 276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra bb65460feb Make dev-shell script work on Darwin 2014-07-16 11:53:47 +02:00
Eelco Dolstra de8be7c3e0 Install systemd and Upstart stuff only on Linux 2014-07-16 11:53:47 +02:00
Shea Levy 048be62484 Pass *_proxy vars to bootstrap fetchurl 2014-07-16 11:21:45 +02:00
Eelco Dolstra a2c85b2ef8 Manual: Typo 2014-07-16 11:21:44 +02:00
Eelco Dolstra 5bcb982711 Remove cruft 2014-07-14 12:39:53 +02:00
Eelco Dolstra fa13d3f4f3 build-remote.pl: Fix building multiple output derivations
We were importing paths without sorting them topologically, leading to
"path is not valid" errors.

See e.g. http://hydra.nixos.org/build/12451761
2014-07-14 12:19:27 +02:00
Eelco Dolstra b2e0293f02 build-remote.pl: Don't keep a shell process around 2014-07-12 00:43:28 +02:00
Eelco Dolstra a00a98548e build-remote.pl: Fix build log 2014-07-12 00:09:43 +02:00
Eelco Dolstra 838138c5c4 Fix test 2014-07-11 16:22:24 +02:00
Eelco Dolstra a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra b8f24f2535 Fix closure size display 2014-07-11 14:27:17 +02:00
Eelco Dolstra e196eecbe6 Allow $NIX_BUILD_HOOK to be relative to Nix libexec directory 2014-07-11 13:55:06 +02:00
Eelco Dolstra d0eb970fb4 Fix broken Pid constructor 2014-07-10 21:48:21 +02:00
Eelco Dolstra edbfe2232e Replace message "importing path <...>" with "exporting path <...>"
This causes nix-copy-closure to show what it's doing before rather
than after.
2014-07-10 21:30:22 +02:00
Eelco Dolstra 42d91b079c Fix use of sysread 2014-07-10 21:17:21 +02:00
Eelco Dolstra 7bb632b024 nix-copy-closure -s: Do substitutions via ‘nix-store --serve’
This means we no longer need an SSH master connection, since we only
execute a single command on the remote host.
2014-07-10 20:43:04 +02:00
Eelco Dolstra 7c3a5090bf nix-copy-closure: Fix --dry-run 2014-07-10 17:44:18 +02:00
Eelco Dolstra 43b64f5038 Remove tabs 2014-07-10 17:32:21 +02:00
Eelco Dolstra 8e9140cfde Refactoring: Move all fork handling into a higher-order function
C++11 lambdas ftw.
2014-07-10 16:58:09 +02:00
Eelco Dolstra 1114c7bd57 nix-copy-closure: Restore compression and the progress viewer 2014-07-10 14:15:12 +02:00
Eelco Dolstra 7911e4c27a Remove maybeVfork 2014-07-10 13:35:44 +02:00
Eelco Dolstra 04170d06bf nix-copy-closure: Fix race condition
There is a long-standing race condition when copying a closure to a
remote machine, particularly affecting build-remote.pl: the client
first asks the remote machine which paths it already has, then copies
over the missing paths. If the garbage collector kicks in on the
remote machine between the first and second step, the already-present
paths may be deleted. The missing paths may then refer to deleted
paths, causing nix-copy-closure to fail. The client now performs both
steps using a single remote Nix call (using ‘nix-store --serve’),
locking all paths in the closure while querying.

I changed the --serve protocol a bit (getting rid of QueryCommand), so
this breaks the SSH substituter from older versions. But it was marked
experimental anyway.

Fixes #141.
2014-07-10 11:58:59 +02:00
Eelco Dolstra 2c3a8f787b Fix security hole in ‘nix-store --serve’
Since it didn't check that the path received from the client is a
store path, the client could dump any path in the file system.
2014-07-10 11:46:01 +02:00
Eelco Dolstra 66dbc0fdee Add a test for the SSH substituter 2014-07-10 01:53:56 +02:00
Eelco Dolstra 0e5d0c1543 Fix compilation error on some versions of GCC
src/libexpr/primops.cc:42:8: error: looser throw specifier for 'virtual nix::InvalidPathError::~InvalidPathError()'
src/libexpr/nixexpr.hh:12:1: error:   overriding 'virtual nix::EvalError::~EvalError() noexcept (true)'

http://hydra.nixos.org/build/12385750
2014-07-09 12:14:40 +02:00
Eelco Dolstra beac05c206 Don't build on Ubuntu 10.10
Its C++ compiler is too old.

http://hydra.nixos.org/build/12385722
2014-07-08 20:41:25 +02:00
Eelco Dolstra beaf3e90af Add builtin function ‘fromJSON’
Fixes #294.
2014-07-04 13:34:15 +02:00
Eelco Dolstra e82951fe23 Manual: html -> xhtml 2014-07-03 12:36:58 +02:00