XZ compresses significantly better than bzip2. Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):
bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
xz -9: total compressed size 588.18 MiB, 21.4% [7m40]
Note that compression takes much longer. More importantly, however,
decompression is much faster:
bzip2: 1m47.274s
xz -6: 0m55.446s
xz -7: 0m54.119s
xz -8: 0m52.388s
xz -9: 0m51.842s
The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
Manifests are a huge pain, since users need to run nix-pull directly
or indirectly to obtain them. They tend to be large and lag behind
the available binaries; also, the downloaded manifests in
/nix/var/nix/manifest need to be in sync with the Nixpkgs sources. So
we want to get rid of them.
The idea of manifest-free operation works as follows. Nix is
configured with a set of URIs of binary caches, e.g.
http://nixos.org/binary-cache
Whenever Nix needs a store path X, it checks each binary cache for the
existence of a file <CACHE-URI>/<SHA-256 hash of X>.narinfo, e.g.
http://nixos.org/binary-cache/bi1gh9...ia17.narinfo
The .narinfo file contains the necessary information about the store
path that was formerly kept in the manifest, i.e., (relative) URI of
the compressed NAR, references, size, hash, etc. For example:
StorePath: /nix/store/xqp4l88cr9bxv01jinkz861mnc9p7qfi-neon-0.29.6
URL: 1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
CompressedHash: sha256:1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd
CompressedSize: 202542
NarHash: sha256:1af26536781e6134ab84201b33408759fc59b36cc5530f57c0663f67b588e15f
NarSize: 700440
References: 043zrsanirjh8nbc5vqpjn93hhrf107f-bash-4.2-p24 cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 ...
Deriver: 4idz1bgi58h3pazxr3akrw4fsr6zrf3r-neon-0.29.6.drv
System: x86_64-linux
Nix then knows that it needs to download
http://nixos.org/binary-cache/1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
to substitute the store path.
Note that the store directory is omitted from the References and
Deriver fields to save space, and that the URL can be relative to the
binary cache prefix.
This patch just makes nix-push create binary caches in this format.
The next step is to make a substituter that supports them.
other simplifications.
* Use <nix/...> to locate the corepkgs. This allows them to be
overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
to create a temporary file.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
derivation should be a source rather than a derivation dependency of
the call to the NAR derivation. Otherwise the derivation (and all
its dependencies) will be built as a side-effect, which may not even
succeed.
dependency. `storePath /nix/store/bla' gives exactly the same
result as `toPath /nix/store/bla', except that the former includes
/nix/store/bla in the dependency context of the string.
Useful in some generated Nix expressions like nix-push, which now
finally does the right thing wrt distributed builds. (Previously
the path to be packed wasn't an explicit dependency, so it wouldn't
be copied to the remote machine.)
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
users.
If the configure flag `--enable-setuid' is used, the Nix programs
nix-env, nix-store, etc. are installed with the setuid bit turned on
so that they are executed as the user and group specified by
`--with-nix-user=USER' and `--with-nix-group=GROUP', respectively
(with defaults `nix' and `nix').
The setuid programs drop all special privileges if they are executed
by a user who is not a member of the Nix group.
The setuid feature is a quick hack to enable sharing of a Nix
installation between users who trust each other. It is not
generally secure, since any user in the Nix group can modify (by
building an appropriate derivation) any object in the store, and for
instance inject trojans into binaries used by other users.
The setuid programs are owned by root, not the Nix user. This is
because on Unix normal users cannot change the real uid, only the
effective uid. Many programs don't work properly when the real uid
differs from the effective uid. For instance, Perl will turn on
taint mode. However, the setuid programs drop all root privileges
immediately, changing all uids and gids to the Nix user and group.
Nix expressions.
To subscribe to a channel (needs to be done only once):
nix-channel --add \
http://catamaran.labs.cs.uu.nl/dist/nix/channels/nixpkgs-unstable
This just adds the given URL to ~/.nix-channels (which can also be
edited manually).
To update from all channels:
nix-channel --update
This fetches the latest expressions and pulls cache manifests. The
default Nix expression (~/.nix-defexpr) is made to point to the
conjunction of the expressions downloaded from all channels.
So to update all installed derivations in the current user
environment:
nix-channel --update
nix-env --upgrade '*'
If you are really courageous, you can put this in a cronjob or
something.
You can subscribe to multiple channels. It is not entirely clear
what happens when there are name clashes between derivations from
different channels. From nix-env/main.cc it appears that the one
with the lowest (highest?) hash will be used, which is pretty
meaningless.
environment variable. This is useful for passing authentication
information (it won't show up in `ps'). Hacky - nix-push should
abstract over the use of Curl.
Eelco Dolstra
Eelco Dolstra
Armijn Hemel