Commit Graph

54 Commits

Author SHA1 Message Date
Eelco Dolstra b6780b9e10 * Uninitialised variable. Fixes the --delete test. 2006-03-01 14:39:42 +00:00
Eelco Dolstra 651ab439cf * A simple hack to fix NIX-18: the garbage collector cannot run when
the disk is full (because to delete something from the Nix store, we
  need a Berkeley DB transaction, which takes up disk space).  Under
  normal operation, we make sure that there exists a file
  /nix/var/nix/db/reserved of 1 MB.  When running the garbage
  collector, we delete that file before we open the Berkeley DB
  environment.
2006-02-16 13:19:15 +00:00
Eelco Dolstra f96d2dea26 * Added a flag `--ignore-liveness' to `nix-store --delete'. It
deletes a path even if it is reachable from a root.  However, it
  won't delete a path that still has referrers (since that would
  violate store invariants).

  Don't try this at home.  It's a useful hack for recovering from
  certain situations in a somewhat clean way (e.g., holes in closures
  due to disk corruption).
2005-12-23 21:36:44 +00:00
Eelco Dolstra 4b9e7f59ca * Revived the old "nix-store --delete" operation that deletes the
specified paths from the Nix store.  However, this operation is
  safe: it refuses to delete anything that the garbage collector
  wouldn't delete.
2005-12-23 21:08:42 +00:00
Eelco Dolstra 530b27df1e * `nix-store --gc' prints out the number of bytes freed on stdout
(even when it is interrupted by a signal).
2005-12-15 21:11:39 +00:00
Eelco Dolstra ab5c6bb3a3 * Change `referer' to `referrer' throughout. In particular, the
nix-store query options `--referer' and `--referer-closure' have
  been changed to `--referrer' and `--referrer-closure' (but the old
  ones are still accepted for compatibility).
2005-12-13 21:04:48 +00:00
Eelco Dolstra 52a2f41320 * Include some required header files. 2005-05-04 16:33:20 +00:00
Eelco Dolstra c702dfca3f * nix-store: `--substitute' -> `--register-substitutes'. 2005-04-08 13:48:41 +00:00
Eelco Dolstra 4271385a73 * Make `nix-store --query --tree' work on non-derivations (i.e., on
any store path).
2005-04-08 12:57:16 +00:00
Eelco Dolstra f9848d4f31 * Support base-32 hash representations. 2005-04-07 14:33:32 +00:00
Eelco Dolstra c815aff21b * `nix-store --add-fixed' to preload the outputs of fixed-output
derivations.  This is mostly to simplify the implementation of
  nix-prefetch-{url, svn}, which now work properly in setuid
  installations.

* Enforce valid store names in `nix-store --add / --add-fixed'.
2005-04-07 14:01:51 +00:00
Eelco Dolstra 6f788880b6 * Re-enable dot graph generation. 2005-03-26 22:06:57 +00:00
Eelco Dolstra 3f236f01ae * `nix-store --register-validity': allow a path to refer to a path
listed later in the list of new valid paths.
2005-03-23 13:07:28 +00:00
Eelco Dolstra a04c62e0c4 * Canonicalise path meta-data in `nix-store --register-validity'. 2005-03-23 12:06:57 +00:00
Eelco Dolstra f20f081560 * nix-store: `--isvalid' -> `--check-validity', `--validpath' ->
`--register-validity'.
* `nix-store --register-validity': read arguments from stdin, and
  allow the references and deriver to be set.
2005-03-23 11:25:20 +00:00
Eelco Dolstra 07b4399fb6 * `nix-store -q --hash' to quickly query the hash of the contents of a
store path (which is stored in the database).
2005-03-02 15:57:06 +00:00
Eelco Dolstra eda2c3c253 * Compatibility hack so that Nixpkgs can continue to do hash checking
in `fetchurl' in Nix <= 0.7, but doesn't in Nix 0.8.
2005-02-22 15:23:24 +00:00
Eelco Dolstra e0181f56be * `nix-store -q --tree' shows a tree representing the dependency graph
of the given derivation.  Useful for getting a quick overview of how
  something was built.  E.g., to find out how the `baffle' program in
  your user environment was built, you can do

    $ nix-store -q --tree $(nix-store -qd $(which baffle))

  Tree nesting depth is minimised (?) by topologically sorting paths
  under the relation A < B iff A \in closure(B).
2005-02-17 15:57:46 +00:00
Eelco Dolstra 8a3a96dd5b * Switch to the calling user context for some more operations in a
setuid installation.
2005-02-17 13:55:18 +00:00
Eelco Dolstra 8992fce3da * It is now possible to add store derivations or paths directly to a
user environment, e.g.,

    $ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv

  or 

    $ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1

  This is useful because it allows Nix expressions to be bypassed
  entirely.  For instance, if only a nix-pull manifest is provided,
  plus the top-level path of some component, it can be installed
  without having to supply the Nix expression (e.g., for obfuscation,
  or to be independent of Nix expression language changes or context
  dependencies).
2005-02-14 17:35:10 +00:00
Eelco Dolstra 98df735b51 * Propagate the deriver of a path through the substitute mechanism.
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
2005-02-09 12:57:13 +00:00
Eelco Dolstra c547439843 * Subflag in `--verify': `nix-store --verify --check-contents' checks
that the contents of store paths has not changed by comparing hashes
  of their current contents to the hashes stored in the database.
2005-02-08 13:48:53 +00:00
Eelco Dolstra 3d74274b37 * Updated `nix-store --verify' to the new schema. 2005-02-08 13:23:55 +00:00
Eelco Dolstra fbc434ee4c * `nix-store -qb' to query derivation environment bindings. Useful
for finding build-time dependencies (possibly after a build).  E.g.,

    $ nix-store -qb aterm $(nix-store -qd $(which strc))
    /nix/store/jw7c7s65n1gwhxpn35j9rgcci6ilzxym-aterm-2.3.1

* Arguments to nix-store can be files within store objects, e.g.,
  /nix/store/jw7c...-aterm-2.3.1/bin/baffle.

* Idem for garbage collector roots.
2005-02-07 14:32:44 +00:00
Eelco Dolstra 450c358e20 * Maintain a database table (`derivers') that maps output paths to the
derivation that produced them.
* `nix-store -qd PATH' prints out the derivation that produced a path.
2005-02-07 13:40:40 +00:00
Eelco Dolstra 65b6c8ab4c * Move root finding from `nix-collect-garbage' to `nix-store --gc'.
This was necessary becase root finding must be done after
  acquisition of the global GC lock.

  This makes `nix-collect-garbage' obsolete; it is now just a wrapper
  around `nix-store --gc'.

* Automatically remove stale GC roots (i.e., indirect GC roots that
  point to non-existent paths).
2005-02-01 15:05:32 +00:00
Eelco Dolstra 630ae0c9d7 * nix-build: use an indirection scheme to make it easier for users to
get rid of GC roots.  Nix-build places a symlink `result' in the
  current directory.  Previously, removing that symlink would not
  remove the store path being linked to as a GC root.  Now, the GC
  root created by nix-build is actually a symlink in
  `/nix/var/nix/gcroots/auto' to `result'.  So if that symlink is
  removed the GC root automatically becomes invalid (since it can no
  longer be resolved).  The root itself is not automatically removed -
  the garbage collector should delete dangling roots.
2005-02-01 13:48:46 +00:00
Eelco Dolstra dcc37c236c * nix-store, nix-instantiate: added an option `--add-root' to
immediately add the result as a permanent GC root.  This is the only
  way to prevent a race with the garbage collector.  For instance, the
  old style

    ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
      /nix/var/nix/gcroots/result

  has two time windows in which the garbage collector can interfere
  (by GC'ing the derivation and the output, respectively).  On the
  other hand,

    nix-store --add-root /nix/var/nix/gcroots/result -r \
      $(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
        foo.nix)

  is safe.

* nix-build: use `--add-root' to prevent GC races.
2005-02-01 12:36:25 +00:00
Eelco Dolstra c505702265 * Fix and simplify the garbage collector (it's still not concurrent,
though).  In particular it's now much easier to register a GC root.
  Just place a symlink to whatever store path it is that you want to
  keep in /nix/var/nix/gcroots.
2005-01-27 15:21:29 +00:00
Eelco Dolstra a24b78e9f1 * Maintain the references/referers relation also for derivations.
This simplifies garbage collection and `nix-store --query
  --requisites' since we no longer need to treat derivations
  specially.

* Better maintaining of the invariants, e.g., setReferences() can only
  be called on a valid/substitutable path.
2005-01-25 21:28:25 +00:00
Eelco Dolstra 2a2756b856 * Simplification: registerSubstitutes -> registerSubstitute. We no
longer need the former since there we no longer have the
  substitutes-rev table (which triggered a O(n^2) cost in updating
  them).
2005-01-25 20:27:40 +00:00
Eelco Dolstra 066da4ab85 * Really fix the substitute mechanism, i.e., ensure the closure
invariant by registering references through the manifest.
* Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
Eelco Dolstra 52bf9b86bb * In nix-store: added query `--referers-closure' that returns the
closure of the referers relation rather than the references
  relation, i.e., the set of all paths that directly or indirectly
  refer to the given path.  Note that contrary to the references
  closure this set is not fixed; it can change as paths are added to
  or removed from the store.
2005-01-25 11:18:03 +00:00
Eelco Dolstra 80faa2f98a * In nix-store: change `--build' back to `--realise'. Also brought
back the query flag `--force-realise'.
* Fixed some of the tests.
2005-01-25 10:55:33 +00:00
Eelco Dolstra e0f4e587c3 * Nix-store queries `--references' and `referers' to query the pointer
graph.  That is, `nix-store --query --references PATH' shows the set
  of paths referenced by PATH, and `nix-store --query --referers PATH'
  shows the set of paths referencing PATH.
2005-01-19 16:59:56 +00:00
Eelco Dolstra 96de272b48 * Renamed `normalise.cc' -> `build.cc', `storeexprs.cc' ->
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
  they have been built/added.  This is so that we can check whether
  the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
  holds.
2005-01-19 16:39:47 +00:00
Eelco Dolstra ef5f254a55 * `nix-store --build' now builds its arguments in parallel instead of
sequentially (within the limits set by `--jobs').  This should
  greatly improve the utilisation of the build farm when doing Nixpkgs
  builds.
2005-01-19 15:02:02 +00:00
Eelco Dolstra 06c77bf7a8 * Change extension `.store' to `.drv'.
* Re-enable `nix-store --query --requisites'.
2005-01-19 14:36:00 +00:00
Eelco Dolstra 863dcff6c5 * Started removing closure store expressions, i.e., the explicit
representation of closures as ATerms in the Nix store.  Instead, the
  file system pointer graph is now stored in the Nix database.  This
  has many advantages:

  - It greatly simplifies the implementation (we can drop the notion
    of `successors', and so on).

  - It makes registering roots for the garbage collector much easier.
    Instead of specifying the closure expression as a root, you can
    simply specify the store path that must be retained as a root.
    This could not be done previously, since there was no way to find
    the closure store expression containing a given store path.
    
  - Better traceability: it is now possible to query what paths are
    referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
Eelco Dolstra fa9259f5f5 * Simplify the substitute mechanism:
- Drop the store expression.  So now a substitute is just a
    command-line invocation (a program name + arguments).  If you
    register a substitute you are responsible for registering the
    expression that built it (if any) as a root of the garbage
    collector.
  - Drop the substitutes-rev DB table.
2004-12-20 13:43:32 +00:00
Eelco Dolstra dcc433de47 * Operation `--delete-generations' to delete generations of a
profile.  Arguments are either generation number, or `old' to delete
  all non-current generations.  Typical use:

  $ nix-env --delete-generations old
  $ nix-collect-garbage

* istringstream -> string2Int.
2004-09-10 13:32:08 +00:00
Eelco Dolstra 5c443b6550 * Main the `substitutes-rev' table again, but now in a way that
doesn't take \Theta(n^2) space/time complexity.
2004-08-31 16:13:10 +00:00
Eelco Dolstra eb233e728f * `--min-age' flag in nix-store and nix-collect-garbage to only delete
unreachable paths that haven't been used for N hours.  For instance,
  `nix-collect-garbage --min-age 168' only deletes paths that haven't
  been accessed in the last week.

  This is useful for instance in the build farm where many derivations
  can be shared between consecutive builds, and we wouldn't want a
  garbage collect to throw them all away.  We could of course register
  them as roots, but then we'd to unregister them at some point, which
  would be a pain to manage.  The `--min-age' flag gives us a sort of
  MRU caching scheme.

  BUG: this really shouldn't be in gc.cc since that violates
  mechanism/policy separation.
2004-08-25 16:54:08 +00:00
Eelco Dolstra fdec72c6cc * `nix-collect-garbage' now actually performs a garbage collection, it
doesn't just print the set of paths that should be deleted.  So
  there is no more need to pipe the result into `nix-store --delete'
  (which doesn't even exist anymore).
2004-08-25 15:39:13 +00:00
Eelco Dolstra 818047881e * Put the garbage collector in nix-store: operation `--gc',
suboperations `--print-live', `--print-dead', and `--delete'.  The
  roots are not determined by nix-store; they are read from standard
  input.  This is to make it easy to customise what the roots are.

  The collector now no longer fails when store expressions are missing
  (which legally happens when using substitutes).  It never tries to
  fetch paths through substitutes.

  TODO: acquire a global lock on the store while garbage collecting.
  
* Removed `nix-store --delete'.
2004-08-25 11:43:49 +00:00
Eelco Dolstra 91dc023665 * Added a switch `--fallback'. From the manual:
Whenever Nix attempts to realise a derivation for which a closure is
  already known, but this closure cannot be realised, fall back on
  normalising the derivation.

  The most common scenario in which this is useful is when we have
  registered substitutes in order to perform binary distribution from,
  say, a network repository.  If the repository is down, the
  realisation of the derivation will fail.  When this option is
  specified, Nix will build the derivation instead.  Thus, binary
  installation falls back on a source installation.  This option is
  not the default since it is generally not desirable for a transient
  failure in obtaining the substitutes to lead to a full build from
  source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
Eelco Dolstra 2db9748221 * Remove debug output. 2004-06-21 10:01:17 +00:00
Eelco Dolstra daf0a923c7 * Wrap calls to registerSubstitute() in a single transaction to
improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
  Theta(N^2) time and log file consumption when adding N substitutes.
  Maybe we can do without it.
2004-06-21 07:38:17 +00:00
Eelco Dolstra 112ee89501 * Re-enable support for substitutes in the normaliser.
* A better substitute mechanism.

  Instead of generating a store expression for each store path for
  which we have a substitute, we can have a single store expression
  that builds a generic program that is invoked to build the desired
  store path, which is passed as an argument.

  This means that operations like `nix-pull' only produce O(1) files
  instead of O(N) files in the store when registering N substitutes.
  (It consumes O(N) database storage, of course, but that's not a
  performance problem).

* Added a test for the substitute mechanism.
  
* `nix-store --substitute' reads the substitutes from standard input,
  instead of from the command line.  This prevents us from running
  into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
Eelco Dolstra 76c0e85929 * The environment variable NIX_ROOT can now be set to execute Nix in a
chroot() environment.
* A operation `--validpath' to register path validity.  Useful for
  bootstrapping in a pure Nix environment.
* Safety checks: ensure that files involved in store operations are in
  the store.
2004-02-14 21:44:18 +00:00