This is a followup to b0715d7cd2, fixing
builds of grafts on i586-gnu.
* guix/grafts.scm (graft-derivation/shallow)[glibc-locales]: Choose
symbol as a function of ‘target-hurd?’.
Change-Id: I05e50c0ed74a64986a0cea9c6302d1b5592b898d
Fixes a bug that would occur with references to two outputs of the same
derivation, with one of them referring to the other one.
For example, the references of libreoffice include both mariadb:dev and
mariadb:lib; additionally, mariadb:dev refers to mariadb:lib. In this
case, the glibc graft would not be applied on one of the mariadb paths,
and both the grafted and ungrafted glibc would end up in the closure of
libreoffice.
Fixes <https://issues.guix.gnu.org/66662>.
* guix/grafts.scm (non-self-references): Simplify and include references
to outputs of DRV other than OUTPUTS.
(reference-origins): Simplify and possibly return outputs of DRV itself.
(cumulative-grafts)[graft-origin?]: Add OUTPUT parameter and honor it.
[dependency-grafts]: Adjust accordingly.
* tests/grafts.scm ("graft-derivation, multiple outputs need to be replaced"):
New test.
Change-Id: Iac2005024ab7049037537b3af55298696ec90e3c
Fixes <https://issues.guix.gnu.org/55968>.
Reported by Maxime Devos <maximedevos@telenet.be>.
* guix/grafts.scm (%graft-with-utf8-locale?): New parameter.
(graft-derivation/shallow)[glibc-locales, set-utf8-locale]: New
variables.
[build]: Use 'set-utf8-locale'.
* tests/gexp.scm, tests/grafts.scm, tests/packages.scm: Set
'%graft-with-utf8-locale?' to #f.
Fixes <https://issues.guix.gnu.org/58419>.
* guix/grafts.scm (graft-derivation/shallow): Rewrite using gexps and
remove 'store' parameter.
(graft-derivation/shallow*): New variable.
(cumulative-grafts): Use it instead of 'graft-derivation/shallow'.
The goal is to allow (guix grafts) to use (guix gexp) without
introducing a cycle between these two modules.
* guix/grafts.scm (%graft?, call-without-grafting, without-grafting)
(set-grafting, grafting?): Move to...
* guix/store.scm: ... here.
Note: this merge actually changes the 'curl' and 'python-attrs' derivations,
as part of solving caf4a7a277 and
12964df69a respectively.
4604d43c0e (gnu: gnutls@3.6.16: Fix cross-compilation.) was ignored because it
cannot currently be tested.
Conflicts:
gnu/local.mk
gnu/packages/aidc.scm
gnu/packages/boost.scm
gnu/packages/curl.scm
gnu/packages/nettle.scm
gnu/packages/networking.scm
gnu/packages/python-xyz.scm
gnu/packages/tls.scm
Partly fixes <https://bugs.gnu.org/41702>.
Reported by Lars-Dominik Braun <ldb@leibniz-psychology.org>.
Previously, 'graft-derivation' would start anew at every call. When
creating a profile with lots of packages, it would potentially do the
same work multiple times. The per-session cache addresses this. It
increases the derivation-graft-cache hit rate from 77.9% to 80.1% on:
GUIX_PROFILING="derivation-graft-cache" ./pre-inst-env \
guix environment --ad-hoc libreoffice inkscape krita darktable -n
The effect is more visible on the pathological case below, where cache
hit rate goes from 75% to 87% and wall-clock time from 5.0s to 3.5s:
GUIX_PROFILING="derivation-graft-cache" ./pre-inst-env \
guix environment --ad-hoc r-learnr --search-paths
* guix/grafts.scm (%graft-cache): New variable.
(graft-derivation): Add calls to 'store-connection-cache' and
'set-store-connection-cache!'.
Partly fixes <https://bugs.gnu.org/41702>.
Reported by Lars-Dominik Braun <ldb@leibniz-psychology.org>.
Previously we'd potentially traverse the same sub-graph of DEPS several
times.
With this patch, command:
guix environment --ad-hoc r-learnr --search-paths
goes from 11.3s to 4.6s.
* guix/grafts.scm (reference-origin): Rename to...
(reference-origins): ... this. Change 'item' parameter to 'items'.
[lookup-derivers]: New procedure.
(cumulative-grafts)[dependency-grafts]: Change 'item' to 'items' and use
'reference-origins'.
Remove 'mapm' around 'dependency-grafts' call.
This is a followup to 710854304b.
This also slightly reduces the number of 'query-references' RPCs, for
instance from 176 to 166 from "guix build emacs -d".
* guix/grafts.scm (references-oracle): Remove.
(non-self-references): Remove 'references' parameter and add 'store'.
Add 'references*' procedure and use it instead of 'references'. Adjust
caller accordingly.
(cumulative-grafts): Remove 'references' parameter and adjust caller
accordingly.
Fixes <https://bugs.gnu.org/22990>.
* guix/grafts.scm (references-oracle)[references*]: Remove call to
'substitution-oracle' and to 'references/substitutes'. Use
'references/cached' and 'build-derivations' right away instead.
That way, if we end up calling 'build-derivations', we'll only build the
outputs that we really need.
* guix/grafts.scm (references-oracle): Rename 'drv' to 'input'.
[output-paths]: Remove.
Adjust accordingly.
(graft-derivation): Adjust call to 'references-oracle'.
Previously we'd make 502 'query-valid-derivers' RPCs for
"guix build vim -d", and after this patch, we don't do any.
Furthermore, the previous strategy was "stateful" in the sense that
'item->deriver' could return a derivation that is not the one that was
actually computed by this process, but an "equivalent" one (due to
fixed-output derivations); which one is chosen would depend on the state
of the store.
This in turn means that we'd have to call 'read-derivation-from-file' to
actually read .drv files (as opposed to getting them from
%DERIVATION-CACHE). This is costly and doesn't work with
GUIX_DAEMON_SOCKET=ssh://….
* guix/grafts.scm (item->deriver): Remove.
(reference-origin): New procedure.
(cumulative-grafts): Use it instead of 'item->deriver'.
* guix/grafts.scm (graft-derivation/shallow): Pass #:properties to
'build-expression->derivation'.
* tests/grafts.scm ("graft-derivation, grafted item is a direct
dependency"): Check the value returned by 'derivation-properties'.
This leads to ~25% improvements on things like:
guix system build desktop.tmpl --no-grafts -d
* guix/store.scm (<nix-server>)[object-cache]: New field.
* guix/store.scm (open-connection): Initialize it.
(cache-object-mapping, lookup-cached-object, %mcached): New procedures.
(mcached): New macro.
* guix/gexp.scm (lower-object): Use it.
* guix/grafts.scm (grafting?): New procedure.
Fixes <https://bugs.gnu.org/19973>.
Reported by Mark H Weaver <mhw@netris.org>.
* guix/build/debug-link.scm: New file.
* guix/build/graft.scm (%graft-hooks): New variable.
(graft): Add #:hooks and honor it.
* guix/grafts.scm (graft-derivation/shallow): Add (guix build
debug-link) and (guix elf) to #:modules.
* tests/debug-link.scm: New file.
* Makefile.am (MODULES): Add guix/build/debug-link.scm.
(SCM_TESTS): Add tests/debug-link.scm.
This avoids the open/fstat/close syscalls upon a cache hit that we had
with the previous idiom:
(call-with-input-file file read-derivation)
where caching happened in 'read-derivation' itself.
* guix/derivations.scm (%read-derivation): Rename to...
(read-derivation): ... this.
(read-derivation-from-file): New procedure.
(derivation-prerequisites, substitution-oracle)
(derivation-prerequisites-to-build):
(derivation-path->output-path, derivation-path->output-paths):
(derivation-path->base16-hash, map-derivation): Use
'read-derivation-from-file' instead of (call-with-input-file …
read-derivation).
* guix/grafts.scm (item->deriver): Likewise.
* guix/scripts/build.scm (log-url, options->things-to-build): Likewise.
* guix/scripts/graph.scm (file->derivation): Remove.
(derivation-dependencies, %derivation-node-type): Use
'read-derivation-from-file' instead.
* guix/scripts/offload.scm (guix-offload): Likewise.
* guix/scripts/perform-download.scm (guix-perform-download): Likewise.
* guix/scripts/publish.scm (load-derivation): Remove.
(narinfo-string): Use 'read-derivation-from-file'.
Fixes <http://bugs.gnu.org/24886>.
Previously, the grafting derivation of, say, brdf-explorer would pull in
qt:doc even though brdf-explorer depends only on qt:out, not qt:doc.
* guix/grafts.scm (with-cache): Use 'vhash-assoc' and 'vhash-cons'
instead of 'vhash-assq' and 'vhash-consq'.
(cumulative-grafts): Pass #:outputs to 'graft-derivation/shallow'. Use
OUTPUTS instead of (derivation-output-names drv).
(graft-derivation): Add #:outputs parameter; pass it to
'cumulative-grafts'.
* tests/grafts.scm (make-derivation-input): New variable.
("graft-derivation, replaced derivation has multiple outputs"): Make
sure P2:zzz is not part of the outputs of P3D.
("graft-derivation with #:outputs")
("graft-derivation, unused outputs not depended on"): New tests.
Before this commit, we'd lose the cache across recursive calls to
'cumulative-grafts', which isn't great performance-wise.
This bug was already present before
d38bc9a9f6.
* guix/grafts.scm (with-cache): In the miss case, call 'current-state'
after EXP has been evaluated.
This fixes a regression introduced in
d38bc9a9f6 whereby the cache was
effectively disabled.
Reported by Thomas Danckaert <thomas.danckaert@gmail.com>.
* guix/grafts.scm (with-cache): In the cache miss case, wrap body in
'mbegin'.
Fixes <http://bugs.gnu.org/24712>.
* guix/grafts.scm (cumulative-grafts): Add grafts for all the outputs of
DRV.
* tests/grafts.scm ("graft-derivation, replaced derivation has multiple
outputs"): New test.
Partly fixes <http://bugs.gnu.org/24418>.
* guix/grafts.scm (cumulative-grafts)[graft-origin?]: New procedure.
[dependency-grafts]: Use it in new 'if' around recursive call.
* tests/grafts.scm ("graft-derivation, grafts are not shadowed"): New test.
Currently, with several grafts applicable to Inkscape, this makes:
guix gc -R $(guix build inkscape -d) | wc -l
go from 2376 to 2266 (4.6%).
* guix/grafts.scm (cumulative-grafts): Pass 'graft-derivation/shallow'
the subset of GRAFTS that applies to DRV.
The time for:
guix build inkscape -n --no-substitutes
goes down by 30% (in the presence of 3 replacements among all the
packages.)
* guix/grafts.scm (cumulative-grafts): Turn into a monadic procedure in
%STATE-MONAD. Use the current state as a derivation-to-graft cache.
(graft-derivation): Call 'cumulative-grafts' within 'run-with-state'.
This avoids starting derivation builds just for the sake of knowing the
references of their outputs, thereby restoring the expected behavior of
--dry-run when substitutes are available.
* guix/grafts.scm (non-self-references): Remove 'store' parameter, and
add 'references'. Use it. Update caller.
(references-oracle): New variable.
(cumulative-grafts): Add 'references' parameter and use it. Update
callers.
(graft-derivation): Remove 'build-derivations' call. Add call to
'references-oracle'.
Fixes <http://bugs.gnu.org/22139>.
* guix/grafts.scm (graft-derivation): Rename to...
(graft-derivation/shallow): ... this.
(graft-origin-file-name, item->deriver, non-self-references)
(cumulative-grafts, graft-derivation): New procedures
* tests/grafts.scm ("graft-derivation, grafted item is a direct
dependency"): Clarify title. Use 'grafted' instead of 'graft' to refer
to the grafted derivation.
("graft-derivation, grafted item is an indirect dependency")
("graft-derivation, no dependencies on grafted output"): New tests.
* guix/packages.scm (input-graft): Change to take a package instead of
an input.
(input-cross-graft): Likewise.
(fold-bag-dependencies): New procedure.
(bag-grafts): Rewrite in terms of 'fold-bag-dependencies'.
* tests/packages.scm ("package-derivation, indirect grafts"): Comment out.
* doc/guix.texi (Security Updates): Mention run-time dependencies and
recursive grafting.
Before that, outputs of a derivation could be left referring to the
ungrafted version of the derivation.
* guix/grafts.scm (graft-derivation)[outputs]: Change to a list of
name/file pairs.
* guix/grafts.scm (graft-derivation)[build]: Add 'old-outputs' variable
and use it when computing 'mapping'. Use 'mapping' directly.
* tests/grafts.scm ("graft-derivation, multiple outputs"): New test.
Ludovic Courtès
Marius Bakke
Ludovic Courtès