Commit Graph

164 Commits

Author SHA1 Message Date
Johan Thomsen
f5db415e2f nixos/tests/dockerTools: add test for running non-root containers with buildLayeredImage
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-07-31 10:14:07 +02:00
Johan Thomsen
9f86685cc7 dockerTools: fix permissions on /nix/store 2020-07-30 16:20:50 +02:00
Robert Hensing
8c0459f611 dockerTools: Always set imageTag attribute
The image tag can be specified or generated from the output hash.
Previously, a generated tag could be recovered from the evaluated
image with some string operations.

However, with the introduction of streamLayeredImage, it's not
feasible to compute the generated tag yourself.

With this change, the imageTag attribute is set unconditionally,
for the buildImage, buildLayeredImage, streamLayeredImage functions.
2020-07-11 16:58:25 +02:00
Jos van Bakel
66737168a4
dockerTools.buildLayeredImage: fix created=now 2020-07-09 09:34:18 +02:00
Utku Demir
06db331922
dockerTools: Verify nix-store contents on buildLayeredImage test 2020-07-06 16:59:58 +12:00
Utku Demir
2ae2186dfd
dockerTools.streamLayeredImage: Correctly copy hardlinks 2020-07-06 16:42:03 +12:00
Utku Demir
cc46362929
dockerTools: Support files directly under /nix/store
Also makes sure that the files inside a layer added in a sorted order
to make the results more deterministic.
2020-07-04 22:00:57 +12:00
Utku Demir
fa8f2bf34f
stream_layered_image: Use pathlib module for directory traversal 2020-06-22 15:11:04 +12:00
Utku Demir
16199521ea
stream_layered_image: Always set uid and gid to root 2020-06-21 12:35:39 +12:00
Utku Demir
fe71abfc12
stream_layered_image: Clarify assertion failure 2020-06-21 12:35:39 +12:00
Utku Demir
87a538e149
stream_layered_image: Use more descriptive variable name 2020-06-21 12:35:39 +12:00
Utku Demir
4bff5b7f3d
dockerTools: Properly quote a shell variable 2020-06-21 12:35:39 +12:00
Utku Demir
f1d7a09146
stream_layered_image: Add docstrings to functions 2020-06-21 12:35:38 +12:00
Utku Demir
26402290bf
stream_layered_image: Add main method 2020-06-21 12:35:38 +12:00
Utku Demir
307804d97d
dockerTools.streamLayeredImage: comments 2020-06-21 12:35:38 +12:00
Utku Demir
4ab7baf6f6
stream_layered_image.py: comments 2020-06-21 12:35:38 +12:00
Utku Demir
a5a611cacb
Set mtime in even more places 2020-06-21 12:35:38 +12:00
Utku Demir
4e0109f873
Set mtime to get more deterministic builds 2020-06-21 12:35:37 +12:00
Utku Demir
560201da66
Implement dockerTools.streamLayeredImage 2020-06-21 12:35:32 +12:00
Robert Hensing
2dd94af186 dockerTools: keep symlinks to nix intact 2020-06-19 10:08:34 +02:00
Robert Hensing
caf47063b4 dockerTools: test that tar keeps nix binary symlinks intact 2020-06-19 10:08:34 +02:00
zowoq
365d07cea0 dockerTools: fix skopeo commands 2020-05-25 14:29:54 +10:00
Alex Biehl
886c92332c dockerTools: Properly add /nix/ and /nix/store/ first to layer.tar
In https://github.com/NixOS/nixpkgs/pull/58431 the authors ensured that
the resulting layer.tar would always list

  /nix/
  /nix/store/

first to fully comply to the tar spec. Various refactorings later it is only
ensured to create /nix/ but NOT /nix/store anymore. Instead tar transformed
them to /nix/nix and /nix/nix/store.
2020-05-24 14:28:35 +02:00
lewo
a498da343a
Merge pull request #87154 from utdemir/buildimage-optimizations
Some performance optimizations to dockerTools.build{,Layered}Image
2020-05-19 15:39:25 +02:00
zowoq
42232493a3 dockerTools: pass insecure-policy and tmpdir to skopeo 2020-05-16 08:46:13 +10:00
lewo
df928fafd4
Merge pull request #87725 from nlewo/fix-dockertools
dockerTools.examples: fix the nginx image
2020-05-15 16:54:09 +02:00
Antoine Eiche
9ac444d80e dockerTools.examples: fix the nginx image
Nginx now requires directories `/var/log/nginx` and `/var/cache/nginx`
to be existing.

The objective is to fix the test `nixosTests.docker-tools`.
2020-05-13 10:29:53 +02:00
Utku Demir
f5a90a7aab
dockerTools.buildImage: Preserve environment variables from the parent image 2020-05-08 21:49:16 +12:00
Utku Demir
f12346d493
dockerTools: Calculate tarsum's on the fly
Calculating the tarsum after creating a layer is inefficient, since
we have to read the tarball we've just written from the disk.

This commit simultaneously calculates the tarsum while creating the
tarball.
2020-05-07 11:50:07 +12:00
Utku Demir
69f6294724
dockerTools.buildLayeredImage: Avoid appending to tarballs when building layers
Appending to an existing tar archive repeatedly seems to be a quadratic
operation, since tar seems to traverse the existing archive even using
the `-r, --append` flag. This commit avoids that by passing the list of
files to a single tar invocation.
2020-05-07 11:50:00 +12:00
adisbladis
fafb127947
dockertools: Add a buildLayeredImageWithNixDb function
This is analogous to buildImageWithNixDb but instead uses
buildLayeredImage under the hood.
2020-05-02 15:59:39 +01:00
Simonas Kazlauskas
dde3e8edf4 nix-prefetch-docker: Add jq into path
Otheriwse running the tool fails with

    line 41: jq: command not found
2020-04-27 22:21:55 +03:00
Robert Hensing
6dab1b50a6 buildLayeredImage: Allow empty store, no paths to add
This is useful when buildLayeredImage is called in a generic way
that should allow simple (base) images to be built, which may not
reference any store paths.
2020-02-28 14:59:04 +01:00
Antoine Eiche
baa78de594 nixosTests.docker-tools: add bulk-layer test
A regression test for https://github.com/NixOS/nixpkgs/issues/78744.
2020-02-14 09:26:26 +01:00
Richard Wallace
3b65b3f6d6 dockerTools.buildLayeredImage: store all paths passed in final layer
Fixes #78744

My previous change broke when there are more packages than the maximum
number of layers. I had assumed that the `store-path-to-layer.sh` was
only ever passed a single store path, but that is not the case if
there are multiple packages going into the final layer. To fix this, we
loop through the paths going into the final layer, appending them to the
tar file and making sure they end up at the right path.
2020-02-14 09:26:26 +01:00
Silvan Mosberger
0a351c3f65
dockerTools.*: Assertion against building for Darwin (#77952)
dockerTools.*: Assertion against building for Darwin
2020-01-31 21:17:40 +01:00
Antoine Eiche
01a68479cc dockerTools.buildLayeredImage: assert maxLayers > 1
Since a layer is reserved for "customization", the image can not
contains less than 2 layers.

The user gets the following message at evaluation:

    nix-instantiate nixos/tests/docker-tools.nix
    trace: the maxLayers argument of dockerTools.buildLayeredImage function must be greather than 1 (current value: 1)
2020-01-30 21:09:52 +01:00
Antoine Eiche
283bcc1003 dockerTools.buildLayeredImage: fix image with only 2 layers
A test is also added to ensure an image with 2 layers can be built.
2020-01-30 21:09:44 +01:00
Silvan Mosberger
b6a9211bf4
dockerTools.*: Assertion against building for darwin
Building a docker image with darwin binaries just yields a confusing
error when ran:

  standard_init_linux.go:211: exec user process caused "exec format error"

This change prevents people from building such images in the first place
2020-01-20 19:14:12 +01:00
Antoine Eiche
da261e3631 dockerTools.buildLayeredImage: fix typo in comments 2020-01-11 09:02:30 +01:00
Richard Wallace
3be767593b dockerTools.buildLayeredImage: fix building layered images in parallel
when tar'ing store paths into layered archives when building layered
images, don't use the absolute nix store path so that tar won't complain
if something new is added to the nix store

when building the final docker image, ignore any file changes tar
detects in the layers. they are all immutable and the only thing that
might change is the number of hard links due to store optimization
2019-12-30 14:47:11 -07:00
Graham Christensen
64453c8dbd
Merge pull request #75781 from grahamc/dockertools/remove-implementation-detail-layers
dockertools.buildLayeredImage: remove implementation detail layers
2019-12-17 16:03:11 -05:00
Graham Christensen
75b8f3820d
Merge pull request #75779 from grahamc/dockertools/fixup-max-layers
dockerTools.buildLayeredImage: update maxlayers from 24 to 100 to match documentation
2019-12-16 14:11:36 -05:00
Graham Christensen
9c02760855
dockerTools.buildLayeredImage: update maxlayers from 24 to 100 to match documentation
mkManyPureLayers already was changed, and this function was not updated.
2019-12-16 13:14:21 -05:00
Graham Christensen
77452740c6
docker examples: Drop unneeded contents list 2019-12-16 13:11:27 -05:00
Graham Christensen
12e2416380
dockerTools.buildLayeredImage: Exclude top level implementation detail layers 2019-12-16 13:03:15 -05:00
Graham Christensen
700f4c5388
dockerTools.buildLayeredImage: prepare to exclude some paths
Without changing behavior, since this code is fiddly, make it possible
to add a filtering step before packaging individual paths.
2019-12-16 12:57:04 -05:00
Graham Christensen
aec80dddc0
dockerTools.buildLayeredImage: pass a list of closures to mkManyPureLayers so it can exclude the top-most level
Before, every docker image had three extra layers:

1. A `closure` layer which is an internal implementation detail of
   calculating the closure of the container
2. a `name-config.json` layer which is the images' run-time
   configuration, and has no business being *in* the image as a layer.
3. a "bulk-layers" layer which is again and implementation detail
   around collecting the image's closure.

None of these layers need to be in the final product.
2019-12-16 12:48:05 -05:00
Graham Christensen
f6d75f550e
dockerTools.buildLayeredImage: tweak formatting on contentsEnv 2019-12-16 12:36:45 -05:00
tomberek
81b0a20dfa buildImageWithNixDb: export USER (#74959)
dockerTools.buildImageWithNixDb: export USER

Changes to Nix user detection (./src/nix-channel/nix-channel.cc#L-166)
cause this function to error. Exporting USER fixes this.
2019-12-07 10:06:42 +01:00