Commit graph

107 commits

Author SHA1 Message Date
Eelco Dolstra 066da4ab85 * Really fix the substitute mechanism, i.e., ensure the closure
invariant by registering references through the manifest.
* Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
Eelco Dolstra c6290e42bc * Fix the `--fallback' switch.
* Fix the substitutes tests.
2005-01-25 13:00:12 +00:00
Eelco Dolstra 581fc47783 * Fix the build hook mechanism; pass the pointer graph to the hook. 2005-01-25 11:55:43 +00:00
Eelco Dolstra 52bf9b86bb * In nix-store: added query `--referers-closure' that returns the
closure of the referers relation rather than the references
  relation, i.e., the set of all paths that directly or indirectly
  refer to the given path.  Note that contrary to the references
  closure this set is not fixed; it can change as paths are added to
  or removed from the store.
2005-01-25 11:18:03 +00:00
Eelco Dolstra 80faa2f98a * In nix-store: change --build' back to --realise'. Also brought
back the query flag `--force-realise'.
* Fixed some of the tests.
2005-01-25 10:55:33 +00:00
Eelco Dolstra 6a0a2d5593 * Terminology fixes. 2005-01-20 16:01:07 +00:00
Eelco Dolstra 05f0430de1 * Another change to low-level derivations. The last one this year, I
promise :-) This allows derivations to specify on *what* output
  paths of input derivations they are dependent.  This helps to
  prevent unnecessary downloads.  For instance, a build might be
  dependent on the `devel' and `lib' outputs of some library
  component, but not the `docs' output.
2005-01-20 14:10:19 +00:00
Eelco Dolstra e0f4e587c3 * Nix-store queries --references' and referers' to query the pointer
graph.  That is, `nix-store --query --references PATH' shows the set
  of paths referenced by PATH, and `nix-store --query --referers PATH'
  shows the set of paths referencing PATH.
2005-01-19 16:59:56 +00:00
Eelco Dolstra 96de272b48 * Renamed normalise.cc' -> build.cc', `storeexprs.cc' ->
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
  they have been built/added.  This is so that we can check whether
  the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
  holds.
2005-01-19 16:39:47 +00:00
Eelco Dolstra ef5f254a55 * `nix-store --build' now builds its arguments in parallel instead of
sequentially (within the limits set by `--jobs').  This should
  greatly improve the utilisation of the build farm when doing Nixpkgs
  builds.
2005-01-19 15:02:02 +00:00
Eelco Dolstra 06c77bf7a8 * Change extension .store' to .drv'.
* Re-enable `nix-store --query --requisites'.
2005-01-19 14:36:00 +00:00
Eelco Dolstra 863dcff6c5 * Started removing closure store expressions, i.e., the explicit
representation of closures as ATerms in the Nix store.  Instead, the
  file system pointer graph is now stored in the Nix database.  This
  has many advantages:

  - It greatly simplifies the implementation (we can drop the notion
    of `successors', and so on).

  - It makes registering roots for the garbage collector much easier.
    Instead of specifying the closure expression as a root, you can
    simply specify the store path that must be retained as a root.
    This could not be done previously, since there was no way to find
    the closure store expression containing a given store path.
    
  - Better traceability: it is now possible to query what paths are
    referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
Eelco Dolstra 32aac8748a * Actually check that the result of fixed-output derivations matches
the specified hash.
2005-01-17 19:01:48 +00:00
Eelco Dolstra f3dc231250 * Removed the `id' attribute hack.
* Formalise the notion of fixed-output derivations, i.e., derivations
  for which a cryptographic hash of the output is known in advance.
  Changes to such derivations should not propagate upwards through the
  dependency graph.  Previously this was done by specifying the hash
  component of the output path through the `id' attribute, but this is
  insecure since you can lie about it (i.e., you can specify any hash
  and then produce a completely different output).  Now the
  responsibility for checking the output is moved from the builder to
  Nix itself.

  A fixed-output derivation can be created by specifying the
  `outputHash' and `outputHashAlgo' attributes, the latter taking
  values `md5', `sha1', and `sha256', and the former specifying the
  actual hash in hexadecimal or in base-32 (auto-detected by looking
  at the length of the attribute value).  MD5 is included for
  compatibility but should be considered deprecated.

* Removed the `drvPath' pseudo-attribute in derivation results.  It's
  no longer necessary.

* Cleaned up the support for multiple output paths in derivation store
  expressions.  Each output now has a unique identifier (e.g., `out',
  `devel', `docs').  Previously there was no way to tell output paths
  apart at the store expression level.

* `nix-hash' now has a flag `--base32' to specify that the hash should
  be printed in base-32 notation.

* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
  `md5'.

* `nix-prefetch-url' now prints out a SHA-1 hash in base-32.  (TODO: a
  flag to specify the hash.)
2005-01-17 16:55:19 +00:00
Eelco Dolstra d58a11e019 * Shorten SHA-256 hashes used in store path name generation to 160
bits, then encode them in a radix-32 representation (using digits
  and letters except e, o, u, and t).  This produces store paths like
  /nix/store/4i0zb0z7f88mwghjirkz702a71dcfivn-aterm-2.3.1.  The nice
  thing about this is that the hash part of the file name is still 32
  characters, as before with MD5.

  (Of course, shortening SHA-256 to 160 bits makes it no better than
  SHA-160 in theory, but hopefully it's a bit more resistant to
  attacks; it's certainly a lot slower.)
2005-01-14 16:04:03 +00:00
Eelco Dolstra 9530cc3170 * Start move towards SHA-256 hashes instead of MD5.
* Start cleaning up unique store path generation (they weren't always
  unique; in particular the suffix ("-aterm-2.2", "-builder.sh") was
  not part of the hash, therefore changes to the suffix would cause
  multiple store objects with the same hash).
2005-01-14 13:51:38 +00:00
Eelco Dolstra 7e8961f720 * Added SHA-1 support. nix-hash' now has an option --type sha1' to
select SHA-1 hashing.
2005-01-13 17:39:26 +00:00
Eelco Dolstra 73992371a3 * Refactoring to support SHA-1. 2005-01-13 15:44:44 +00:00
Eelco Dolstra a03397be4c * Cygwin compatibility. 2005-01-04 17:38:26 +00:00
Eelco Dolstra 8b9697e575 * An operation `nix-store --clear-substitutes' to remove all
registered substitute mappings.
2004-12-20 14:16:55 +00:00
Eelco Dolstra fa9259f5f5 * Simplify the substitute mechanism:
- Drop the store expression.  So now a substitute is just a
    command-line invocation (a program name + arguments).  If you
    register a substitute you are responsible for registering the
    expression that built it (if any) as a root of the garbage
    collector.
  - Drop the substitutes-rev DB table.
2004-12-20 13:43:32 +00:00
Eelco Dolstra 13f77276d1 * utime() follows symlinks, so don't change the mtime if the file is a
symlink.
2004-11-29 19:22:16 +00:00
Eelco Dolstra f17553a212 * Remove debug statement. 2004-11-29 15:30:44 +00:00
Eelco Dolstra 4115d8d8ce * Canonicalise metadata of all files in store paths (set the mtime to
0, set the mode to either 444 or 555, set the group to the default).
2004-11-29 15:09:29 +00:00
Eelco Dolstra cb7ccb528b * string2ATerm -> overloaded toATerm. 2004-11-03 18:12:03 +00:00
Eelco Dolstra a69534fc21 * Drop ATmake / ATMatcher also in handling store expressions. 2004-10-29 11:22:49 +00:00
Eelco Dolstra f4d44a0026 * Allow certain operations to succeed even if we don't have write
permission to the Nix store or database.  E.g., `nix-env -qa' will
  work, but `nix-env -qas' won't (the latter needs DB access).  The
  option `--readonly-mode' forces this mode; otherwise, it's only
  activated when the database cannot be opened.
2004-10-25 14:38:23 +00:00
Eelco Dolstra 1317242780 * Make store objects created by substitutes read-only. 2004-09-22 12:15:04 +00:00
Eelco Dolstra b357284a32 * Fallback didn't work for subderivations of an unnormalised the main
derivation, since NormalisationGoal would first run a
  NormalisationGoal on the subderivation (a no-op, since in a
  situation where we need fallback the successor is known), and then
  runs a RealisationGoal on the normal form, which then cannot do a
  fallback because it doesn't know the derivation expression for which
  it is a normal form.

  Tossed out the 2-phase normalisation/realisation in
  NormalisationGoal and SubstitutionGoal since it's no longer needed -
  a RealisationGoal will run a NormalisationGoal if necessary.
2004-09-12 19:08:57 +00:00
Eelco Dolstra c16be6ac92 * Remove write permission from store objects after they have been
added to the store.  Bug reported by Martin.
2004-09-09 21:19:20 +00:00
Eelco Dolstra 47f87072ad * A very dirty hack to make setuid installations a bit nicer to use.
Previously there was the problem that all files read by nix-env
  etc. should be reachable and readable by the Nix user.  So for
  instance building a Nix expression in your home directory meant that
  the home directory should have at least g+x or o+x permission so
  that the Nix user could reach the Nix expression.  Now we just
  switch back to the original user just prior to reading sources and
  the like.  The places where this happens are somewhat arbitrary,
  however.  Any scope that has a live SwitchToOriginalUser object in
  it is executed as the original user.

* Back out r1385.  setreuid() sets the saved uid to the new
  real/effective uid, which prevents us from switching back to the
  original uid.  setresuid() doesn't have this problem (although the
  manpage has a bug: specifying -1 for the saved uid doesn't leave it
  unchanged; an explicit value must be specified).
2004-09-09 21:12:53 +00:00
Eelco Dolstra 5c443b6550 * Main the `substitutes-rev' table again, but now in a way that
doesn't take \Theta(n^2) space/time complexity.
2004-08-31 16:13:10 +00:00
Eelco Dolstra c25f2883b1 * Quadruple the Berkeley DB locking limits to get rid of out of memory
errors while running `nix-store --verify'.
2004-08-31 10:50:08 +00:00
Eelco Dolstra fe122c5a15 * Removed nrWaitees field. It was redundant with waitees.size() and
could get out of sync if multiple input derivations mapped to the
  same closure expression (since waitees is a set).
2004-08-30 11:51:36 +00:00
Eelco Dolstra eb233e728f * `--min-age' flag in nix-store and nix-collect-garbage to only delete
unreachable paths that haven't been used for N hours.  For instance,
  `nix-collect-garbage --min-age 168' only deletes paths that haven't
  been accessed in the last week.

  This is useful for instance in the build farm where many derivations
  can be shared between consecutive builds, and we wouldn't want a
  garbage collect to throw them all away.  We could of course register
  them as roots, but then we'd to unregister them at some point, which
  would be a pain to manage.  The `--min-age' flag gives us a sort of
  MRU caching scheme.

  BUG: this really shouldn't be in gc.cc since that violates
  mechanism/policy separation.
2004-08-25 16:54:08 +00:00
Eelco Dolstra 818047881e * Put the garbage collector in nix-store: operation `--gc',
suboperations `--print-live', `--print-dead', and `--delete'.  The
  roots are not determined by nix-store; they are read from standard
  input.  This is to make it easy to customise what the roots are.

  The collector now no longer fails when store expressions are missing
  (which legally happens when using substitutes).  It never tries to
  fetch paths through substitutes.

  TODO: acquire a global lock on the store while garbage collecting.
  
* Removed `nix-store --delete'.
2004-08-25 11:43:49 +00:00
Eelco Dolstra 1eddee59f2 * The default verbosity level of all Nix commands is now lvlInfo.
* Builder output is written to standard error by default.
  * The option `-B' is gone.
  * The option `-Q' suppresses builder output.

The result of this is that most Nix invocations shouldn't need any
flags w.r.t. logging.
2004-08-18 12:19:06 +00:00
Eelco Dolstra 62fe5c4a22 * The predecessor of a successor need not be present. This in
particular happens on distributed builds or when using push/pull.
2004-08-11 19:03:13 +00:00
Eelco Dolstra ae1a1efa41 * Clean up the temporary directory for hook communication (and don't
print out incorrect "build failed" messages).
2004-08-05 14:53:27 +00:00
Eelco Dolstra e3a50f7e25 * Creating a file nix-support/no-scan in the output path of a
derivation disables scanning for dependencies.  Use at your own
  risk.  This is a quick hack to speed up UML image generation (image
  are very big, say 1 GB).

  It would be better if the scanner were faster, and didn't read the
  whole file into memory.
2004-08-04 09:25:21 +00:00
Eelco Dolstra c1a18f543e * Fixed format string error. 2004-07-06 11:21:34 +00:00
Eelco Dolstra 056cd1d3b7 * Don't go into a (sometimes infinite) loop calling the build hook. 2004-07-01 16:24:35 +00:00
Eelco Dolstra b584253af4 * Include some missing headers. 2004-07-01 11:11:16 +00:00
Eelco Dolstra f5d5ffe536 * Write build logs to disk again. 2004-06-29 09:41:50 +00:00
Eelco Dolstra 4d2946c516 * In a realisation goal, check the result of the corresponding
normalisation goal.
2004-06-28 13:51:24 +00:00
Eelco Dolstra 91dc023665 * Added a switch `--fallback'. From the manual:
Whenever Nix attempts to realise a derivation for which a closure is
  already known, but this closure cannot be realised, fall back on
  normalising the derivation.

  The most common scenario in which this is useful is when we have
  registered substitutes in order to perform binary distribution from,
  say, a network repository.  If the repository is down, the
  realisation of the derivation will fail.  When this option is
  specified, Nix will build the derivation instead.  Thus, binary
  installation falls back on a source installation.  This option is
  not the default since it is generally not desirable for a transient
  failure in obtaining the substitutes to lead to a full build from
  source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
Eelco Dolstra b113edeab7 * A flag `--keep-going / -k' to keep building goals if one fails, as
much as possible.  (This is similar to GNU Make's `-k' flag.)

* Refactoring to implement this: previously we just bombed out when
  a build failed, but now we have to clean up.  In particular this
  means that goals must be freed quickly --- they shouldn't hang
  around until the worker exits.  So the worker now maintains weak
  pointers in order not to prevent garbage collection.

* Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
Eelco Dolstra e4883211f9 * Don't throw an exception when a build fails. Just terminate the
goal and allow the problem to be handled elsewhere (e.g., at
  top-level).
2004-06-25 10:21:44 +00:00
Eelco Dolstra 795d9f8b08 * Obsolete. 2004-06-24 14:36:50 +00:00
Eelco Dolstra ec32627621 * Multiple and/or failing substitutes now work. 2004-06-24 13:40:38 +00:00