Can't build on Ubuntu?

[a-h]

The README.md has this example:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

  outputs = { nixpkgs, ... }: {
    nixosConfigurations.my-system = nixpkgs.lib.nixosSystem {
      modules = [
        "${nixpkgs}/nixos/maintainers/scripts/ec2/amazon-image.nix"
        {
          virtualisation.diskSize = "auto"; # or gigabytes. default is 3 * 1024
          nixpkgs.hostPlatform = "x86_64-linux";
          services.nginx.enable = true;
        }
      ];
    };
  };
}

So, I try it on my Ubuntu system, with Nix installed:

nix build .#nixosConfigurations.my-system.config.system.build.amazonImage

But I get a hash mismatch error:

warning: creating lock file '"/home/adrian-hesketh/github.com/NixOS/test/flake.lock"': 
• Added input 'nixpkgs':
    'github:NixOS/nixpkgs/2d293cbfa5a793b4c50d17c05ef9e385b90edf6c?narHash=sha256-pp3uT4hHijIC8JUK5MEqeAWmParJrgBVzHLNfJDZxg4%3D' (2025-11-30)
evaluation warning: system.stateVersion is not set, defaulting to 26.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
error: builder for '/nix/store/8srzxbmfmzid7ywd90bvbyika0rnpaiz-nixos-image-amazon-26.05.20251130.2d293cb-x86_64-linux.drv' failed with exit code 1;
       last 25 log lines:
       > copying path '/nix/store/pq7kfn7csd5mraabqqd04r05lr1ys20p-nix.conf' to 'local'...
       > copying path '/nix/store/d8vgpa0xrzn7cs5b2cjps3l7j658l4kg-nixos-configuration-reference-manpage' to 'local'...
       > copying path '/nix/store/gqp0lmydjwxvzh4sz2wxav0bx4kvqmy1-X-Restart-Triggers-nix-daemon' to 'local'...
       > copying path '/nix/store/1fgly246vib7x8lcn75y7wvw3lv2n974-nix-2.31.2-man' to 'local'...
       > copying path '/nix/store/lbhpa2769ibbhq3skcjz17ww0j7r00ns-nixos-manual-html' to 'local'...
       > error: hash mismatch importing path '/nix/store/1fgly246vib7x8lcn75y7wvw3lv2n974-nix-2.31.2-man';
       >          specified: sha256:0av3yx7j9pifilcc8053q544b5zqahclqw6gpq9yz7cysirli66b
       >          got:       sha256:1j4jk200qg2wj4d7dws12zcsph0hd0wj69mh4dz9mr5wl3623b77
       > copying path '/nix/store/ncdy64g32scrjkqmbklsdp08aj9n7p4b-nixos.conf' to 'local'...
       > copying path '/nix/store/m65dckzfgr1lr90g5v6jx0iynnz38nsk-nss-cacert-3.115' to 'local'...
       > copying path '/nix/store/j3kvsaal1g5hk9afc3l1s23abm9mb23q-nss-cacert-3.115-p11kit' to 'local'...
       > copying path '/nix/store/17n5sx566xy7q5rqn1q2v8w1g2kpjajk-other.pam' to 'local'...
       > copying path '/nix/store/0g1mha3mrqspxvxgag3fij2w2jk4m35i-perl-5.40.0-man' to 'local'...
       > copying path '/nix/store/xgr0gawyyjv6jmk25022b1z53zxqhi4j-perl5.40.0-Digest-HMAC-1.05' to 'local'...
       > copying path '/nix/store/lhicc1840hbljj7mjly69xfjpf8a4ga5-perl5.40.0-Encode-Locale-1.05' to 'local'...
       > copying path '/nix/store/6p281mkvcl13fsxwvjrmnn0wldk0ksdc-perl5.40.0-FCGI-ProcManager-0.28' to 'local'...
       > copying path '/nix/store/m6q0acx8mz8gk33lqm0sgadbbqc6jb14-perl5.40.0-File-Copy-Recursive-0.45' to 'local'...
       > copying path '/nix/store/gbsvbq172qrcqrzd2j0rwapf3rrmlg0i-perl5.40.0-File-Slurp-9999.32' to 'local'...
       > copying path '/nix/store/gsz08hb519jijq84r36jagj1a6drirvw-system-shutdown' to 'local'...
       > copying path '/nix/store/xaa75rd44q62nc9mrbvym9d1m6gy0fj8-tzdata-2025b' to 'local'...
       > copying path '/nix/store/lak7173vnnyxxkf9bysyk99gndp1lr7n-udev-rules' to 'local'...
       > copying path '/nix/store/nl8601ic4drz80s2rpdg6w3msw6l2j6f-unit--.slice' to 'local'...
       > copying path '/nix/store/760s2vdy9zcgrhli53rmxklbxr9b1vfm-unit-dbus.socket' to 'local'...
       > copying path '/nix/store/mmby6dldf289kixpb6bnp5vkbw9rqyq1-unit-fs.target' to 'local'...
       > error: build of '/nix/store/c0w392bbhvccg48f5sghsdj2rq4p9w7w-nixos-system-unnamed-amazon-26.05.20251130.2d293cb' failed
       For full logs, run 'nix log /nix/store/8srzxbmfmzid7ywd90bvbyika0rnpaiz-nixos-image-amazon-26.05.20251130.2d293cb-x86_64-linux.drv'.

I can build a normal (non AWS AMI, e.g. QEMU) virtual machine image on Ubuntu without issue.

I can also build the AWS AMI virtual machine image from a NixOS machine.

Is it expected that AMI building only works on NixOS hosts? The problem seems to be somewhere in whatever "${nixpkgs}/nixos/maintainers/scripts/ec2/amazon-image.nix" does.

[arianvp]

That is super weird. What version of nix? I'd almost say this is a nix bug

[a-h]

I tested it in nix (Nix) 2.28.4

That's quite old, so I also tested an upgrade to 2.31.2 (default in 2.31.2) and it was still broken with the same error.

I kept Nix 2.31.2 and I bumped nixpkgs to the latest nixos-25.11 commit, and it still didn't work, so I bumped Nix to 2.32.4 by using pkgs.nixVersions.latest, but still got the same error.

[arianvp]

Seems we've gotten ourselves into a classic frankenbuild Situation.

What i think is happening here; but not 100% sure:

1. Cache.nixos.org caches intermediate dependencies of the image build. Including the closureInfo. The closureInfo includes the content hash of the nix manual.
2. The nix manual is not reproducible. And the hash of the man pages is different in the nix in the nix installer compared to what is in cache.nixos.org
3. When copying the the nix man pages to the image, the hash mismatch occurs

The fix here is: hydra should never cache closureInfo derivations. But currently it is.
The workaround is: build the image without cache.nixos.org enabled

I'm not 100% sure though. As we don't build an image with nginx enabled in hydra. So there shouldn't be a matching closureInfo derivation.

Kind of puzzling

[arianvp]

It's kind of weird as it's happening across nix versions. Which makes me expect something else is going on. As your exact closureInfo derivation shouldn't be cached by our cache

[a-h]

Thanks for looking into it!

I tried disabling documentation, and it then got a hash mismatch on getty...

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

  outputs = { nixpkgs, ... }: {
    nixosConfigurations.my-system = nixpkgs.lib.nixosSystem {
      modules = [
        "${nixpkgs}/nixos/maintainers/scripts/ec2/amazon-image.nix"
        {
          virtualisation.diskSize = "auto"; # or gigabytes. default is 3 * 1024
          nixpkgs.hostPlatform = "x86_64-linux";
          services.nginx.enable = true;
          documentation.enable = false;
        }
      ];
    };
  };
}

error: builder for '/nix/store/z4w01wb0qh3sq1qz1q5h6y13wy8ps25s-nixos-image-amazon-26.05.20260116.e4bae1b-x86_64-linux.drv' failed with exit code 1;
       last 25 log lines:
       > copying path '/nix/store/qzc04a3npl70cyyy6flnnrb2ig3kayxm-python3-3.13.11' to 'local'...
       > copying path '/nix/store/42r0qnkkx9y0wl3vcaa7iyb2cn5bd5qy-dash-0.5.13.1' to 'local'...
       > copying path '/nix/store/wf1b666cp6ahn165bjidlxidnpkrkswx-openssh-10.2p1' to 'local'...
       > copying path '/nix/store/lkxf3cbsd8wd3qqwbz1n50y1lky5gb6a-etc-systemd-system.conf' to 'local'...
       > copying path '/nix/store/ym6hv54llm5ml25v2k66nrxx9395drzd-sshd.conf-final' to 'local'...
       > copying path '/nix/store/d6mnhlvb3kca7hyslp1syhgs51y5kczs-unit-script-print-host-key-start' to 'local'...
       > copying path '/nix/store/y01mq95z92k65wskf4713plx0wdcbcxh-X-Restart-Triggers-sshd' to 'local'...
       > copying path '/nix/store/wdyy8a75j6snjbyazl3qb5mc669z3f29-etc-nanorc' to 'local'...
       > copying path '/nix/store/n6j2r1dca19m5jicq0z6qglx4j060wbc-bcache-tools-1.1' to 'local'...
       > copying path '/nix/store/yi6jwdm6af0dig2krj38zk57a2jiqsrc-etc-shells' to 'local'...
       > copying path '/nix/store/a1yn6mdj5h13m0bjyimwqnf1akda1xci-grub-2.12' to 'local'...
       > copying path '/nix/store/00bl40bym3lshsjhsn2j03sgjbddkdwj-iptables-1.8.11' to 'local'...
       > copying path '/nix/store/gjikaah2apy6s1h03jcyw2fqd11j0nw1-nix-info' to 'local'...
       > copying path '/nix/store/xkx3x7cs6c3b05808wmwjv2zmrxykj1g-nixos-firewall-tool' to 'local'...
       > copying path '/nix/store/3mfj86p2r3vvk8s9wqpsb0ixyfw6ckgv-amazon-ssm-agent-3.3.2299.0' to 'local'...
       > copying path '/nix/store/1li7ip0277i45k8ag3lhsjl7q6c3r5f2-getty' to 'local'...
       > copying path '/nix/store/6c6680vq87lj0qypvrbbc0zslan32v3c-install-grub.pl' to 'local'...
       > copying path '/nix/store/ixbzifh9c92wsymbqzvyn988p31hq987-mdadm-4.4' to 'local'...
       > copying path '/nix/store/nxp3rcl217qi714kng3b5igz8rygplww-security-wrapper-mount-x86_64-unknown-linux-musl' to 'local'...
       > copying path '/nix/store/47gmra3zqg4cgzbr9gcm7pa4hd75phb3-security-wrapper-umount-x86_64-unknown-linux-musl' to 'local'...
       > copying path '/nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty' to 'local'...
       > error: hash mismatch importing path '/nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty';
       >          specified: sha256:0nprbq45lvkjai35f9xdhns7sv6szbqk5wbsx59r7ja4rl00h512
       >          got:       sha256:0gxhb4i0v5kpj2i30bz65mgl7vv31v0kh6pb4hx291pdy2hlpjhm
       > error: build of '/nix/store/pch4yiljpk5w0xmriwfzfhx99ynn0sq9-nixos-system-unnamed-amazon-26.05.20260116.e4bae1b' failed
       For full logs, run 'nix log /nix/store/z4w01wb0qh3sq1qz1q5h6y13wy8ps25s-nixos-image-amazon-26.05.20260116.e4bae1b-x86_64-linux.drv'.

My system is a quite vanilla company laptop, nothing exciting.

[arianvp]

Adding an ubuntu build to CI to further debug this: https://github.com/NixOS/amis/pull/335/changes

[xokdvium]

Sorry if this question is too obvious, but did you rule out legitimate store corruption?

Like with nix-store --verify --check-contents. Before jumping to triaging bugs it seems prudent to rule out the more obvious stuff.

The exportReferencesGraph in closureInfo dumps what nix thinks the hash should be (as is stored the db), so it's either the host that's wrong or that store path doesn't have the expected hash. Can you compute the actual NAR hash that you have on disk and compare that against what you substituted from cache.nixos.org?

One thing that might lead to modified store paths is that (IIRC) nix on non-nixos doesn't have /nix/store mounted as read-only, so root can modify anything. Could you double-check if that's the case?

[arianvp]

I can't reproduce with both DeterminateNix installer and cachix/nix-install-action

https://github.com/NixOS/amis/actions/runs/21268130257/job/61212159251

Both pass on ubuntu

[a-h]

Thanks for taking the time to investigate!

I checked the store:

nix-store --verify --check-contents         
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...

@xokdvium - I'm not sure how to do this:

The exportReferencesGraph in closureInfo dumps what nix thinks the hash should be (as is stored the db), so it's either the host that's wrong or that store path doesn't have the expected hash. Can you compute the actual NAR hash that you have on disk and compare that against what you substituted from cache.nixos.org?

However, it makes sense that there might have genuinely been an issue with the file on disk, so I tried using the refresh and repair flags, but got the same error.

nix build .#nixosConfigurations.my-system.config.system.build.amazonImage --refresh --repair
warning: Git tree '/home/adrian-hesketh/nixostest' is dirty
evaluation warning: system.stateVersion is not set, defaulting to 26.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
error: builder for '/nix/store/i32g53mycn1r42yr2l17c9bd1qvfa5q6-nixos-image-amazon-26.05.20260121.88d3861-x86_64-linux.drv' failed with exit code 1;
       last 25 log lines:
       > copying path '/nix/store/00bl40bym3lshsjhsn2j03sgjbddkdwj-iptables-1.8.11' to 'local'...
       > copying path '/nix/store/gjikaah2apy6s1h03jcyw2fqd11j0nw1-nix-info' to 'local'...
       > copying path '/nix/store/xkx3x7cs6c3b05808wmwjv2zmrxykj1g-nixos-firewall-tool' to 'local'...
       > copying path '/nix/store/zk9bf6xik0360ghfcdxis784v79qpbz0-unit-firewall.service' to 'local'...
       > copying path '/nix/store/3mfj86p2r3vvk8s9wqpsb0ixyfw6ckgv-amazon-ssm-agent-3.3.2299.0' to 'local'...
       > copying path '/nix/store/1li7ip0277i45k8ag3lhsjl7q6c3r5f2-getty' to 'local'...
       > copying path '/nix/store/6c6680vq87lj0qypvrbbc0zslan32v3c-install-grub.pl' to 'local'...
       > copying path '/nix/store/ixbzifh9c92wsymbqzvyn988p31hq987-mdadm-4.4' to 'local'...
       > copying path '/nix/store/nxp3rcl217qi714kng3b5igz8rygplww-security-wrapper-mount-x86_64-unknown-linux-musl' to 'local'...
       > copying path '/nix/store/47gmra3zqg4cgzbr9gcm7pa4hd75phb3-security-wrapper-umount-x86_64-unknown-linux-musl' to 'local'...
       > copying path '/nix/store/85nyy12f8b43vjny4wmyxck3zn8j09y0-udev-path' to 'local'...
       > copying path '/nix/store/jqpr43n8zc4786309xz0kad8s9x88pxv-unit-container-getty-.service' to 'local'...
       > copying path '/nix/store/pgs01bp0qbnxs28jhxmsmhhpi2kh4l5p-unit-save-hwclock.service' to 'local'...
       > copying path '/nix/store/cvxq0y2xnc42s92crsmkxsxln2ri4b8i-unit-script-nixos-activation-start' to 'local'...
       > copying path '/nix/store/f6wdl32150mi9sf10ss2aalz50mjhw5z-unit-serial-getty-.service' to 'local'...
       > copying path '/nix/store/nwvd6v9wvzi30ji2whf9rsb4riyxp89k-unit-systemd-fsck-.service' to 'local'...
       > copying path '/nix/store/05smlnfpc7yp65jq6p5150w9zlnafqiy-unit-systemd-makefs-.service' to 'local'...
       > copying path '/nix/store/gk7cjr1gwnlh7z3z3psm5nlvsjj044h3-unit-systemd-mkswap-.service' to 'local'...
       > copying path '/nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty' to 'local'...
       > copying path '/nix/store/20wccvva11xlxylq9wh0fr4vrwq79h7d-unit-nixos-activation.service' to 'local'...
       > copying path '/nix/store/av10n3ym1pd72iahqcs4rk17jigc19w0-unit-script-suid-sgid-wrappers-start' to 'local'...
       > error: hash mismatch importing path '/nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty';
       >          specified: sha256:0nprbq45lvkjai35f9xdhns7sv6szbqk5wbsx59r7ja4rl00h512
       >          got:       sha256:0gxhb4i0v5kpj2i30bz65mgl7vv31v0kh6pb4hx291pdy2hlpjhm
       > error: build of '/nix/store/2ppzglf3pvndnvy6d60np02h6z41sy7w-nixos-system-unnamed-amazon-26.05.20260121.88d3861' failed
       For full logs, run 'nix log /nix/store/i32g53mycn1r42yr2l17c9bd1qvfa5q6-nixos-image-amazon-26.05.20260121.88d3861-x86_64-linux.drv'.

Given getty was the problem, I figured I'd delete it from the store to make it pull the file from cache.nixos.org - I checked my config, and I only have substituters = https://cache.nixos.org so it's not coming from anywhere else:

0 0 /home/adrian-hesketh/nixostest % nix-store --delete /nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty       
finding garbage collector roots...
deleting '/nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty'
deleting unused links...
note: currently hard linking saves 9901.97 MiB
1 store paths deleted, 0.00 MiB freed

But the build still had the same problem:

0 0 /home/adrian-hesketh/nixostest % nix build .#nixosConfigurations.my-system.config.system.build.amazonImage
warning: Git tree '/home/adrian-hesketh/nixostest' is dirty
evaluation warning: system.stateVersion is not set, defaulting to 26.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
error: builder for '/nix/store/i32g53mycn1r42yr2l17c9bd1qvfa5q6-nixos-image-amazon-26.05.20260121.88d3861-x86_64-linux.drv' failed with exit code 1;
       last 25 log lines:
       > copying path '/nix/store/w603bkk0wbn0x86ap4z7p62ai3i04mc1-less-679' to 'local'...
       > copying path '/nix/store/zj5bwkxqkp53prm4wz8nhlvmk39srajk-nano-8.7' to 'local'...
       > copying path '/nix/store/xw2j3bn2ss92fwmr0ijz4jnyqd21xybc-libedit-20251016-3.1' to 'local'...
       > copying path '/nix/store/2m5f7kh434zcfgj60s9gw22f37hs8c4k-procps-4.0.4' to 'local'...
       > copying path '/nix/store/vksayvpb9qm9h32k0qqg67nclrq8sf79-readline-8.3p1' to 'local'...
       > copying path '/nix/store/i05qa489majbwyhkmyv610h5d4k7rkl0-util-linux-2.41.2-bin' to 'local'...
       > copying path '/nix/store/42r0qnkkx9y0wl3vcaa7iyb2cn5bd5qy-dash-0.5.13.1' to 'local'...
       > copying path '/nix/store/wf1b666cp6ahn165bjidlxidnpkrkswx-openssh-10.2p1' to 'local'...
       > copying path '/nix/store/hkbylipx1iiawqdcjv858p501wv81bpm-bash-interactive-5.3p3' to 'local'...
       > copying path '/nix/store/qzc04a3npl70cyyy6flnnrb2ig3kayxm-python3-3.13.11' to 'local'...
       > copying path '/nix/store/lkxf3cbsd8wd3qqwbz1n50y1lky5gb6a-etc-systemd-system.conf' to 'local'...
       > copying path '/nix/store/ym6hv54llm5ml25v2k66nrxx9395drzd-sshd.conf-final' to 'local'...
       > copying path '/nix/store/d6mnhlvb3kca7hyslp1syhgs51y5kczs-unit-script-print-host-key-start' to 'local'...
       > copying path '/nix/store/y01mq95z92k65wskf4713plx0wdcbcxh-X-Restart-Triggers-sshd' to 'local'...
       > copying path '/nix/store/wdyy8a75j6snjbyazl3qb5mc669z3f29-etc-nanorc' to 'local'...
       > copying path '/nix/store/1li7ip0277i45k8ag3lhsjl7q6c3r5f2-getty' to 'local'...
       > copying path '/nix/store/6c6680vq87lj0qypvrbbc0zslan32v3c-install-grub.pl' to 'local'...
       > copying path '/nix/store/ixbzifh9c92wsymbqzvyn988p31hq987-mdadm-4.4' to 'local'...
       > copying path '/nix/store/nxp3rcl217qi714kng3b5igz8rygplww-security-wrapper-mount-x86_64-unknown-linux-musl' to 'local'...
       > copying path '/nix/store/47gmra3zqg4cgzbr9gcm7pa4hd75phb3-security-wrapper-umount-x86_64-unknown-linux-musl' to 'local'...
       > copying path '/nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty' to 'local'...
       > error: hash mismatch importing path '/nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty';
       >          specified: sha256:0nprbq45lvkjai35f9xdhns7sv6szbqk5wbsx59r7ja4rl00h512
       >          got:       sha256:0gxhb4i0v5kpj2i30bz65mgl7vv31v0kh6pb4hx291pdy2hlpjhm
       > error: build of '/nix/store/2ppzglf3pvndnvy6d60np02h6z41sy7w-nixos-system-unnamed-amazon-26.05.20260121.88d3861' failed
       For full logs, run 'nix log /nix/store/i32g53mycn1r42yr2l17c9bd1qvfa5q6-nixos-image-amazon-26.05.20260121.88d3861-x86_64-linux.drv'.

[arianvp]

nix path-info --json nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty should give you your local narhash

https://cache.nixos.org/ya2ijc7fycg76z89p0dvd6b1jkimc19a.narinfo gives the caches hash

[arianvp]

sha256:0nprbq45lvkjai35f9xdhns7sv6szbqk5wbsx59r7ja4rl00h512 thus is the caches hash.

[xokdvium]

Hm I see, something up with the builder following symlinks (via nix or otherwise).

[xokdvium]

As a sanity check, /nix/store/ya2ijc7fycg76z89p0dvd6b1jkimc19a-getty store path on your host is a symlink to /nix/store/1li7ip0277i45k8ag3lhsjl7q6c3r5f2-getty/bin/getty?

Can you share your nix config and describe the setup more so that we can try to reproduce?

[xokdvium]

At this point I think the most useful thing would be to strace -ff -e newfstatat the nix process and all its children and search for the offending symlinks (or the nix daemon processes if you are running with the daemon). Specifically the one in the build sandbox that copies the NAR into the chroot store under the hood of nixos-install. My best guess is that something must be different about your Ubuntu setup if we can't reproduce it – this is very bizarre. With the strace we could start pinpointing the issue.