Commit Graph

89 Commits

Author SHA1 Message Date
Graham Christensen
817c933878
check-env: don't execute check-meta.nix 15,000 times
Generated from https://github.com/NixOS/nix/pull/2761:

```
                                                                                          ns     calls ns/call
- /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 591200 15026 39.3451
+ /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 8744   308   28.3896
```

more, generated by:

```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > before 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' before | sort  > before.list
```

applying this patch, then:

```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > after 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' after | sort  > after.list
```

and then diffing before.list and after.list to get:

```
                                                                                                        calls
- :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1               7513
+ :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1               154

- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23                7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23                154

- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26                7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26                154

- onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 15026
+ onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 308
```

The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true`:

| stat                       | before         | after          | Δ               | Δ%      |
|:---------------------------|---------------:|---------------:|:----------------|--------:|
| **cpuTime**                |        179.915 |        145.543 | 🡖 34.372        | -19.10% |
| **envs-bytes**             |  3,900,878,824 |  3,599,483,208 | 🡖 301,395,616   |  -7.73% |
| **envs-elements**          |    214,426,071 |    185,881,709 | 🡖 28,544,362    | -13.31% |
| **envs-number**            |    136,591,891 |    132,026,846 | 🡖 4,565,045     |  -3.34% |
| **gc-heapSize**            | 11,400,048,640 | 12,314,890,240 | 🡕 914,841,600   |   8.02% |
| **gc-totalBytes**          | 25,976,902,560 | 24,510,740,176 | 🡖 1,466,162,384 |  -5.64% |
| **list-bytes**             |  1,665,290,080 |  1,665,290,080 | 0               |         |
| **list-concats**           |      7,264,417 |      7,264,417 | 0               |         |
| **list-elements**          |    208,161,260 |    208,161,260 | 0               |         |
| **nrAvoided**              |    191,359,386 |    179,693,661 | 🡖 11,665,725    |  -6.10% |
| **nrFunctionCalls**        |    119,665,062 |    116,348,547 | 🡖 3,316,515     |  -2.77% |
| **nrLookups**              |     80,996,257 |     76,069,825 | 🡖 4,926,432     |  -6.08% |
| **nrOpUpdateValuesCopied** |    213,930,649 |    213,930,649 | 0               |         |
| **nrOpUpdates**            |     12,025,937 |     12,025,937 | 0               |         |
| **nrPrimOpCalls**          |     88,105,604 |     86,451,598 | 🡖 1,654,006     |  -1.88% |
| **nrThunks**               |    196,842,044 |    175,126,701 | 🡖 21,715,343    | -11.03% |
| **sets-bytes**             |  7,678,425,776 |  7,285,767,928 | 🡖 392,657,848   |  -5.11% |
| **sets-elements**          |    310,241,340 |    294,373,227 | 🡖 15,868,113    |  -5.11% |
| **sets-number**            |     29,079,202 |     27,601,310 | 🡖 1,477,892     |  -5.08% |
| **sizes-Attr**             |             24 |             24 | 0               |         |
| **sizes-Bindings**         |              8 |              8 | 0               |         |
| **sizes-Env**              |             16 |             16 | 0               |         |
| **sizes-Value**            |             24 |             24 | 0               |         |
| **symbols-bytes**          |     16,474,666 |     16,474,676 | 🡕 10            |   0.00% |
| **symbols-number**         |        376,426 |        376,427 | 🡕 1             |   0.00% |
| **values-bytes**           |  6,856,506,288 |  6,316,585,560 | 🡖 539,920,728   |  -7.87% |
| **values-number**          |    285,687,762 |    263,191,065 | 🡖 22,496,697    |  -7.87% |

The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-instantiate ./nixos/release-combined.nix -A tested`:

| stat                       | before         | after          | Δ               | Δ%     |
|:---------------------------|---------------:|---------------:|:----------------|-------:|
| **cpuTime**                |        256.071 |        237.531 | 🡖 18.54         | -7.24% |
| **envs-bytes**             |  7,111,004,192 |  7,041,478,520 | 🡖 69,525,672    | -0.98% |
| **envs-elements**          |    346,236,940 |    339,588,487 | 🡖 6,648,453     | -1.92% |
| **envs-number**            |    271,319,292 |    270,298,164 | 🡖 1,021,128     | -0.38% |
| **gc-heapSize**            |  8,995,291,136 | 10,110,009,344 | 🡕 1,114,718,208 | 12.39% |
| **gc-totalBytes**          | 37,172,737,408 | 36,878,391,888 | 🡖 294,345,520   | -0.79% |
| **list-bytes**             |  1,886,162,656 |  1,886,163,472 | 🡕 816           |  0.00% |
| **list-concats**           |      6,898,114 |      6,898,114 | 0               |        |
| **list-elements**          |    235,770,332 |    235,770,434 | 🡕 102           |  0.00% |
| **nrAvoided**              |    328,829,821 |    326,618,157 | 🡖 2,211,664     | -0.67% |
| **nrFunctionCalls**        |    240,850,845 |    239,998,495 | 🡖 852,350       | -0.35% |
| **nrLookups**              |    144,849,632 |    142,126,339 | 🡖 2,723,293     | -1.88% |
| **nrOpUpdateValuesCopied** |    251,032,504 |    251,032,504 | 0               |        |
| **nrOpUpdates**            |     17,903,110 |     17,903,110 | 0               |        |
| **nrPrimOpCalls**          |    140,674,913 |    139,485,975 | 🡖 1,188,938     | -0.85% |
| **nrThunks**               |    294,643,131 |    288,678,022 | 🡖 5,965,109     | -2.02% |
| **sets-bytes**             |  9,464,322,192 |  9,456,172,048 | 🡖 8,150,144     | -0.09% |
| **sets-elements**          |    377,474,889 |    377,134,877 | 🡖 340,012       | -0.09% |
| **sets-number**            |     50,615,607 |     50,616,875 | 🡕 1,268         |  0.00% |
| **sizes-Attr**             |             24 |             24 | 0               |        |
| **sizes-Bindings**         |              8 |              8 | 0               |        |
| **sizes-Env**              |             16 |             16 | 0               |        |
| **sizes-Value**            |             24 |             24 | 0               |        |
| **symbols-bytes**          |      3,147,102 |      3,147,064 | 🡖 38            | -0.00% |
| **symbols-number**         |         82,819 |         82,819 | 0               |        |
| **values-bytes**           | 11,147,448,768 | 10,996,111,512 | 🡖 151,337,256   | -1.36% |
| **values-number**          |    464,477,032 |    458,171,313 | 🡖 6,305,719     | -1.36% |
2019-04-11 19:21:38 -04:00
Frederik Rietdijk
2fcb11a244 Merge staging-next into master 2019-03-01 09:06:20 +01:00
Matthew Bauer
b86e62d30d llvm: support cross compilation with useLLVM flag
You can build (partially) with LLVM toolchain using the useLLVM flag.
This works like so:

  nix-build -A hello --arg crossSystem '{ system =
    "aarch64-unknown-linux-musl"; useLLVM = true }'

also don’t separate debug info in lldClang

It doesn’t work currently with that setup hook. Missing build-id?
2019-02-26 19:45:35 -05:00
Daiderd Jordan
7ec53a932f
stdenv: only set __darwinAllowLocalNetworking on darwin
This is a darwin only nix attribute for sandbox builds, it can be
ignored on other platforms to avoid unnecessary rebuilds.
2019-02-14 21:38:08 +01:00
Matthew Bauer
5c09d977c7 Merge remote-tracking branch 'origin/master' into staging 2019-02-09 12:14:06 -05:00
Vladimír Čunát
ff101946a8
Merge branch 'master' into staging-next
Hydra nixpkgs: ?compare=1503899
2019-02-06 14:24:46 +01:00
Matthew Bauer
c6f8f8d98d make-derivation: only modify name when name is given
This preserves Nix’s native error handling of missing name:

  error: derivation name missing
2019-02-04 09:15:03 -05:00
Vincent Laporte
0d17ecce2c
mkDerivation: cleaner handling of the name argument 2019-02-01 16:02:42 +00:00
Vladimír Čunát
5effa4e0f9
Merge branch 'master' into staging-next
Comments on conflicts:
- llvm: d6f401e1 vs. 469ecc70 - docs for 6 and 7 say the default is
  to build all targets, so we should be fine
- some pypi hashes: they were equivalent, just base16 vs. base32
2019-02-01 09:22:29 +01:00
Matthew Bauer
8020bd6869
Merge pull request #35884 from dtzWill/fix/man-in-outputsToInstall
default to including "man" in outputsToInstall
2019-01-31 14:55:12 -05:00
Matthew Bauer
e2fe4c2d49 make-derivation: fix ordering of conditionals
cross should have higher precedence
2019-01-28 11:38:30 -05:00
Matthew Bauer
9fd1c170cc make-derivation: try to fix stdenv.cc == null conditional
sometimes this gets an infinite recursion error
2019-01-28 10:41:19 -05:00
Matthew Bauer
7e589e5594 make-derivation: fix position in trace
For a long time now, tracing has been broken in Nixpkgs. So when you
have an eval error you would get something like this:

  error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:11:
  while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
  while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:

This is coming from how Nix handles string context and how
make-derivation messes with the "name" attribute. This commit should
restore the old behavior so you get a nice line number like:

  error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/applications/misc/hello/default.nix:4:3:
  while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
  while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:

NOTE: This will still be broken for cross compilation due to the
prefixes we are adding to name.
2019-01-28 09:25:47 -05:00
Matthew Bauer
ae16dd1a15
stdenv/make-derivation: don't hide broken packages
This behavior ended up breaking the handleEvalIssue functionality by hiding those packages. So something like this:

$ nix-env -iA nixpkgs.zoom-us

would silently fail, without telling the user how to fix it! Regardless, this "bug" should be handled in Nix - not Nixpkgs.

Fixes #38952.
2019-01-27 17:02:17 -05:00
Matthew Bauer
8babcc3d44
Merge branch 'master' into fix/man-in-outputsToInstall 2019-01-26 10:51:12 -05:00
Frederik Rietdijk
a1a5ea5943 stdenv: make checkInputs native
We can't run the checkPhase when build != host, so we may as well make
the checkInputs native.

This signicantly improves the situation of Python packages when enabling
strictDeps.
2019-01-13 14:43:18 +01:00
Jan Tojnar
c5881ec2c9
Merge remote-tracking branch 'upstream/master' into staging 2018-11-30 20:09:45 +01:00
Matthew Bauer
28e2277305 make-derivation: remove selfConsistent check
version is set in lots of places but might not need to be in a name.

Alternative to #50364.
2018-11-30 17:58:33 +01:00
Frederik Rietdijk
1828a5c5ba Merge master into staging-next 2018-11-30 17:46:21 +01:00
Matthew Bauer
e6834171b9
Merge pull request #51183 from matthewbauer/make-derivation-no-assertions
Fix breaking changes to make-derivation
2018-11-29 19:35:55 -06:00
Vaibhav Sagar
b5504e26b0 pkgs/stdenv/generic/make-derivation.nix: s/targetPlatform/hostPlatform/ 2018-11-29 17:54:59 -05:00
Matthew Bauer
a56fe056ec make-derivation: don’t add host suffix if there is no c compiler
Some trivial builders use the name attr to choose the exec name
produced. For example nixos-install,

  nixos-install = makeProg {
    name = "nixos-install";
    src = ./nixos-install.sh;
    nix = config.nix.package.out;
    path = makeBinPath [ nixos-enter ];
  };

When cross compiling, this puts the prog in,

  /bin/nixos-install-powerpc64le-unknown-linux-gnu
2018-11-28 12:44:10 -06:00
Matthew Bauer
d64f288154 make-derivation: don’t assert on separateDebugInfo
When separateDebugInfo = true & !stdenv.hostPlatform.isLinux, we
always gave an error. There is no reason to do this. Instead, just
don’t add separate debug info when we aren’t on Linux.
2018-11-28 12:29:00 -06:00
Matthew Bauer
13e45aeed3 make-derivation: remove selfConsistent check
version is set in lots of places but might not need to be in a name.

Alternative to #50364.
2018-11-28 12:26:21 -06:00
Vladimír Čunát
a5de78b7d7
Merge branch 'master' into staging-next 2018-11-26 10:28:00 +01:00
Matthew Bauer
7eeb02d47b Revert "make-derivation: add disallowedReferences in strictDeps"
This reverts commit 8dbfb61e4617050917ce6bb7c5f4efc902c2a36c.
Also reverts commit fc99c337edd29d9596379667063bc3a8f3cea5b3.

Fixes #50915
2018-11-25 15:53:28 -06:00
Frederik Rietdijk
c31cb577ae Merge master into staging-next 2018-11-22 09:57:08 +01:00
Matthew Bauer
fc99c337ed make-derivation: don’t disallow propagated native inputs
propagateNativeBuildInputs will end up going in the output derivation.
This case is allowed to end up in references because of that. Sorry
for the disruption!

Fixes #50865
2018-11-21 09:39:47 -06:00
Frederik Rietdijk
1d3bff25db Merge staging-next into staging 2018-11-11 14:28:08 +01:00
Frederik Rietdijk
53d00c3351 Merge master into staging-next 2018-11-10 11:08:54 +01:00
Matthew Bauer
6d531f3541 make-derivation: enable pie hardening with musl
Fixes #49071

On ld.gold, we produce broken executables when linking with the Musl
libc. This appears to be a known bug when using ld.gold and Musl. This
thread describes the workaround as enabling PIE when using ld.gold and
Musl:

https://www.openwall.com/lists/musl/2015/05/01/5

By default we don’t enable PIE to avoid breaking things. But in the
Musl case we are breaking things by not enabling PIE. So this adds a
special case for defaultHardeningFlags which keeps the pie hardening
for everything. Any packages that break with PIE can add the pie flag
to disableHardeningFlags array (a no-op for now on anything but Musl).
2018-11-07 15:24:51 +01:00
Patrick Hilhorst
320c9c10de
make-derivation: use pname-version as default name if both are present 2018-11-06 00:04:21 +01:00
Frederik Rietdijk
322f87137c Merge master into staging-next 2018-11-04 11:33:17 +01:00
Frederik Rietdijk
abea6f461a Revert "Merge pull request #49398 from Synthetica9/implement-rfc0035" to fix eval
This reverts commit 3fc7d5eb83804e10ae55b1ae9b102f88b1ea2b08, reversing
changes made to 1fddf2b68996b56804a24b67191e4d883943057d.

The idea is good, however, before enforcing, make sure all occurences
are fixed.
2018-11-04 11:02:43 +01:00
Frederik Rietdijk
cb4ff927a1 Merge master into staging-next 2018-11-04 08:49:24 +01:00
Matthew Bauer
8dbfb61e46 make-derivation: add disallowedReferences in strictDeps
When strictDeps = true, we don’t want native build inputs to end up in
the output. For instance gcc is a builtin native build input and
should only show up in an output if it is also listed in buildInputs.

/cc @ericson2314
2018-11-02 19:31:51 -05:00
Patrick Hilhorst
1f7fc09176
make-derivation: use a more descriptive assert message
As suggested by @Profpatsch
2018-10-30 14:33:14 +01:00
Patrick Hilhorst
c7e026bec4
make-derivation: use lib.assertMsg
As suggested by @Profpatsch
2018-10-29 18:25:59 +01:00
Patrick Hilhorst
5be927db14
make-derivation: use ? instead of builtins.hasAttr
As suggested by @edolstra
2018-10-29 15:17:13 +01:00
Patrick Hilhorst
2962f94fec
make-derivation: add check that the name is consistent with pname and version 2018-10-29 14:58:12 +01:00
Patrick Hilhorst
149a55eca7
make-derivation: get position info from version 2018-10-29 14:51:22 +01:00
Patrick Hilhorst
efca8b4b97
make-derivation: use pname-version as default name if both are present 2018-10-29 14:51:08 +01:00
Matthew Bauer
b3041b4455 make-derivation: set CMAKE_SYSTEM_* when cross compiling
Uses uname data to find what to set these variables:

- CMAKE_SYSTEM_NAME
- CMAKE_SYSTEM_PROCESSOR
- CMAKE_SYSTEM_VERSION
- CMAKE_HOST_SYSTEM_NAME
- CMAKE_HOST_SYSTEM_PROCESSOR
- CMAKE_HOST_SYSTEM_VERSION
2018-10-16 21:50:37 -05:00
John Ericson
7319013ea1 Merge remote-tracking branch 'upstream/master' into staging 2018-09-18 16:55:42 -04:00
John Ericson
2111e7b742 mkDerivation: Make separateDebugInfo assertion lazier to match other assertions
This is needed to access attributes of derivations on platforms where
they cannot be built.
2018-09-18 16:25:19 -04:00
Jan Malakhovski
c63ca0a431 stdenv: implement enableParallelChecking option
Works similarly to `enableParallelBuilding`, but is set by default when
`enableParallelBuilding` is set. In my experience most packages that build
fine in parallel also check fine in parallel.
2018-09-05 01:14:39 +00:00
John Ericson
06cd7c15a3 mkDerivation: Fix cross compilation
Derivations where drawing their `system` attribute from `hostPlatform`
instead of `buildPlatform`. Fix that, and add an explanatory commment.

Fixes #45993
2018-09-03 23:18:24 -04:00
John Ericson
2c2f1e37d4 reewide: Purge all uses stdenv.system and top-level system
It is deprecated and will be removed after 18.09.
2018-08-30 17:20:32 -04:00
Jan Malakhovski
eb5b313b16 stdenv: shadow outputs 2018-08-05 19:24:46 +00:00
Jan Malakhovski
a53504f4a4 stdenv: shadow doCheck and doInstallCheck 2018-08-05 19:24:32 +00:00