diff options
| -rw-r--r-- | blog/i-was-wrong-about-nix-2020-02-10.markdown | 431 | ||||
| -rw-r--r-- | blog/thoughts-on-nix-2020-01-28.markdown | 4 | ||||
| -rw-r--r-- | nix/sources.json | 6 | ||||
| -rw-r--r-- | shell.nix | 5 |
4 files changed, 444 insertions, 2 deletions
diff --git a/blog/i-was-wrong-about-nix-2020-02-10.markdown b/blog/i-was-wrong-about-nix-2020-02-10.markdown new file mode 100644 index 0000000..90a0ef6 --- /dev/null +++ b/blog/i-was-wrong-about-nix-2020-02-10.markdown @@ -0,0 +1,431 @@ +--- +title: I was Wrong about Nix +date: 2020-02-10 +tags: + - nix + - witchcraft +--- + +From time to time, I am outright wrong on my blog. This is one of those times. +In my [last post about Nix][nixpost], I didn't see the light yet. I think I do +now, and I'm going to attempt to clarify below. + +[nixpost]: https://christine.website/blog/thoughts-on-nix-2020-01-28 + +Let's talk about a more simple scenario: writing a service in Go. This service +will depend on at least the following: + +- A Go compiler to build the code into a binary +- An appropriate runtime to ensure the code will run successfully +- Any data files needed at runtime + +A popular way to model this is with a Dockerfile. Here's the Dockerfile I use +for my website (the one you are reading right now): + +``` +FROM xena/go:1.13.6 AS build +ENV GOPROXY https://cache.greedo.xeserv.us +COPY . /site +WORKDIR /site +RUN CGO_ENABLED=0 go test -v ./... +RUN CGO_ENABLED=0 GOBIN=/root go install -v ./cmd/site + +FROM xena/alpine +EXPOSE 5000 +WORKDIR /site +COPY --from=build /root/site . +COPY ./static /site/static +COPY ./templates /site/templates +COPY ./blog /site/blog +COPY ./talks /site/talks +COPY ./gallery /site/gallery +COPY ./css /site/css +HEALTHCHECK CMD wget --spider http://127.0.0.1:5000/.within/health || exit 1 +CMD ./site +``` + +This fetches the Go compiler from [an image I made][godockerfile], copies the +source code to the image, builds it (in a way that makes the resulting binary a +[static executable][staticbin]), and creates the runtime environment for it. + +[godockerfile]: https://github.com/Xe/dockerfiles/blob/master/lang/go/Dockerfile +[staticbin]: https://oddcode.daveamit.com/2018/08/16/statically-compile-golang-binary/ + +Let's let it build and see how big the result is: + +``` +$ docker build -t xena/christinewebsite:example1 . +<output omitted> +$ docker images | grep xena +xena/christinewebsite example1 4b8ee64969e8 24 seconds ago 111MB +``` + +Investigating this image with [dive][dive], we see the following: + +[dive]: https://github.com/wagoodman/dive + +- The package manager is included in the image +- The package manager's database is included in the image +- An entire copy of the C library is included in the image (even though the + binary was _statically linked_ to specifically avoid this) +- Most of the files in the docker image are unrelated to my website's + functionality and are involved with the normal functioning of Linux systems + +Granted, [Alpine Linux][alpine] does a good job at keeping this chaff to a +minimum, but it is still there, still needs to be updated (causing all of my +docker images to be rebuilt and applications to be redeployed) and still takes +up space in transfer quotas and on the disk. + +[alpine]: https://alpinelinux.org + +Let's compare this to the same build process but done with Nix. My Nix setup is +done in a few phases. First I use [niv][niv] to manage some dependencies a-la +git submodules that don't hate you: + +[niv]: https://github.com/nmattia/niv + +``` +$ nix-shell -p niv +[nix-shel]$ niv init +<writes nix/*> +``` + +Now I add the tool [vgo2nix][vgo2nix] in niv: + +[vgo2nix]: https://github.com/adisbladis/vgo2nix + +``` +[nix-shell]$ niv add adisbladis/vgo2nix +``` + +And I can use it in my shell.nix: + +```nix +let + pkgs = import <nixpkgs> { }; + sources = import ./nix/sources.nix; + vgo2nix = (import sources.vgo2nix { }); +in pkgs.mkShell { buildInputs = [ pkgs.go pkgs.niv vgo2nix ]; } +``` + +And then relaunch nix-shell with vgo2nix installed and convert my [go modules][gomod] +dependencies to a Nix expression: + +[gomod]: https://github.com/golang/go/wiki/Modules + +``` +$ nix-shell +<some work is done to compile things, etc> +[nix-shell]$ vgo2nix +<writes deps.nix> +``` + +Now that I have this, I can follow the [buildGoPackage +instructions][buildgopackage] from the upstream nixpkgs documentation and create +`site.nix`: + +[buildgopackage]: https://nixos.org/nixpkgs/manual/#ssec-go-legacy + +``` +{ pkgs ? import <nixpkgs> {} }: +with pkgs; + +assert lib.versionAtLeast go.version "1.13"; + +buildGoPackage rec { + name = "christinewebsite-HEAD"; + version = "latest"; + goPackagePath = "christine.website"; + src = ./.; + + goDeps = ./deps.nix; + allowGoReference = false; + preBuild = '' + export CGO_ENABLED=0 + buildFlagsArray+=(-pkgdir "$TMPDIR") + ''; + + postInstall = '' + cp -rf $src/blog $bin/blog + cp -rf $src/css $bin/css + cp -rf $src/gallery $bin/gallery + cp -rf $src/static $bin/static + cp -rf $src/talks $bin/talks + cp -rf $src/templates $bin/templates + ''; +} +``` + +And this will do the following: + +- Download all of the needed dependencies and place them in the system-level Nix + store so that they are not downloaded again +- Set the `CGO_ENABLED` environment variable to `0` so the Go compiler emits a + static binary +- Copy all of the needed files to the right places so that the blog, gallery and + talks features can load all of their data +- Depend on nothing other than a working system at runtime + +This Nix build manifest doesn't just work on Linux. It works on my mac too. The +dockerfile approach works great for Linux boxes, but (unlike what the me of a +decade ago would have hoped) the whole world just doesn't run Linux on their +desktops. The real world has multiple OSes and Nix allows me to compensate. + +So, now that we have a working _cross-platform_ build, let's see how big it +comes out as: + +``` +$ readlink ./result-bin +/nix/store/ayvafpvn763wwdzwjzvix3mizayyblx5-christinewebsite-HEAD-bin +$ du -hs result-bin/ +89M ./result-bin/ +$ du -hs result-bin/ +11M ./result-bin/bin +888K ./result-bin/blog +40K ./result-bin/css +44K ./result-bin/gallery +77M ./result-bin/static +28K ./result-bin/talks +64K ./result-bin/templates +``` + +As expected, most of the build results are static assets. I have a lot of larger +static assets including an entire copy of TempleOS, so this isn't too +surprising. Let's compare this to on the mac: + +``` +$ du -hs result-bin/ + 91M result-bin/ +$ du -hs result-bin/* + 14M result-bin/bin +872K result-bin/blog + 36K result-bin/css + 40K result-bin/gallery + 77M result-bin/static + 24K result-bin/talks + 60K result-bin/templates +``` + +Which is damn-near identical save some macOS specific crud that Go has to deal +with. + +I mentioned this is used for Docker builds, so let's make `docker.nix`: + +```nix +{ system ? builtins.currentSystem }: + +let + pkgs = import <nixpkgs> { inherit system; }; + + callPackage = pkgs.lib.callPackageWith pkgs; + + site = callPackage ./site.nix { }; + + dockerImage = pkg: + pkgs.dockerTools.buildImage { + name = "xena/christinewebsite"; + tag = pkg.version; + + contents = [ pkg ]; + + config = { + Cmd = [ "/bin/site" ]; + WorkingDir = "/"; + }; + }; + +in dockerImage site +``` + +And then build it: + +``` +$ nix-build docker.nix +<output omitted> +$ docker load -i result +c6b1d6ce7549: Loading layer [==================================================>] 95.81MB/95.81MB +$ docker images | grep xena +xena/christinewebsite latest 0d1ccd676af8 50 years ago 94.6MB +``` + +And the output is 16 megabytes smaller. + +The image age might look weird at first, but it's part of the reproducibility +Nix offers. The date an image was built is something that can change with time +and is actually a part of the resulting file. This means that an image built one +second after another has a different cryptographic hash. It helpfully pins all +images to Unix timestamp 0, which just happens to be about 50 years ago. + +Looking into the image with `dive`, the only packages installed into this image +are: + +- The website and all of its static content goodness +- IANA portmaps that Go depends on as part of the [`net`][gonet] package +- The standard list of [MIME types][mimetypes] that the [`net/http`][gonethttp] + package needs +- Time zone data that the [`time`][gotime] package needs + +[gonet]: https://godoc.org/net +[gonethttp]: https://godoc.org/net/http +[gotime]: https://godoc.org/time + +And that's it. This is _fantastic_. Nearly all of the disk usage has been +eliminated. If someone manages to trick my website into executing code, that +attacker cannot do anything but run more copies of my website (that will +immediately fail and die because the port is already allocated). + +This strategy pans out to more complicated projects too. Consider a case where a +frontend and backend need to be built and deployed as a unit. Let's create a new +setup using niv: + +``` +$ niv init +``` + +Since we are using [Elm][elm] for this complicated project, let's add the +[elm2nix][elm2nix] tool so that our Elm dependencies have repeatable builds, and +[gruvbox-css][gcss] for some nice simple CSS: + +[elm]: https://elm-lang.org +[elm2nix]: https://github.com/cachix/elm2nix +[gcss]: https://github.com/Xe/gruvbox-css + +``` +$ niv add cachix/elm2nix +$ niv add Xe/gruvbox-css +``` + +And then add it to our `shell.nix`: + +``` +let + pkgs = import <nixpkgs> {}; + sources = import ./nix/sources.nix; + elm2nix = (import sources.elm2nix { }); +in +pkgs.mkShell { + buildInputs = [ + pkgs.elmPackages.elm + pkgs.elmPackages.elm-format + elm2nix + ]; +} +``` + +And then enter `nix-shell` to create the Elm boilerplate: + +``` +$ nix-shell +[nix-shell]$ cd frontend +[nix-shell:frontend]$ elm2nix init > default.nix +[nix-shell:frontend]$ elm2nix convert > elm-srcs.nix +[nix-shell:frontend]$ elm2nix snapshot +``` + +And then we can edit the generated Nix expression: + +``` +let + sources = import ./nix/sources.nix; + gcss = (import sources.gruvbox-css { }); +# ... + buildInputs = [ elmPackages.elm gcss ] + ++ lib.optional outputJavaScript nodePackages_10_x.uglify-js; +# ... + cp -rf ${gcss}/gruvbox.css $out/public + cp -rf $src/public/* $out/public/ +# ... + outputJavaScript = true; +``` + +And then test it with `nix-build`: + +``` +$ nix-build +<output omitted> +``` + +And now create a `name.nix` for your Go service like I did above. The real +magic comes from the `docker.nix` file: + +``` +{ system ? builtins.currentSystem }: + +let + pkgs = import <nixpkgs> { inherit system; }; + sources = import ./nix/sources.nix; + backend = import ./backend.nix { }; + frontend = import ./frontend/default.nix { }; +in + +pkgs.dockerTools.buildImage { + name = "xena/complicatedservice"; + tag = "latest"; + + contents = [ backend frontend ]; + + config = { + Cmd = [ "/bin/backend" ]; + WorkingDir = "/public"; + }; +}; +``` + +Now both your backend and frontend services are built with the dependencies in +the Nix store and shipped as a repeatable Docker image. + +Sometimes it might be useful to ship the dependencies to a service like +[Cachix][cachix] to help speed up builds. + +[cachix]: https://cachix.org + +You can install the cachix tool like this: + +``` +$ nix-env -iA cachix -f https://cachix.org/api/v1/install +``` + +And then follow the steps at [cachix.org][cachix] to create a new binary cache. +Let's assume you made a cache named `teddybear`. When you've created a new +cache, logged in with an API token and created a signing key, you can pipe +nix-build to the Cachix client like so: + +``` +$ nix-build | cachix push teddybear +``` + +And other people using that cache will benefit from your premade dependency and +binary downloads. + +To use the cache somewhere, install the Cachix client and then run the +following: + +``` +$ cachix use teddybear +``` + +I've been able to use my Go, Elm, Rust and Haskell dependencies on other +machines using this. It's saved so much extra download time. + +## tl;dr + +I was wrong about Nix. It's actually quite good once you get past the +documentation being baroque and hard to read as a beginner. I'm going to try and +do what I can to get the documentation improved. + +As far as getting started with Nix, I suggest following these posts: + +- Nix Pills: https://nixos.org/nixos/nix-pills/ +- Nix Shorts: https://github.com/justinwoo/nix-shorts +- NixOS: For Developers: https://myme.no/posts/2020-01-26-nixos-for-development.html + +Also, I really suggest trying stuff as a vehicle to understand how things work. +I got really far by experimenting with getting [this Discord bot I am writing in +Rust][withinbot] working in Nix and have been very pleased with how it's turned +out. I don't need to use `rustup` anymore to manage my Rust compiler or the +language server. With a combination of [direnv][direnv] and [lorri][lorri], I +can avoid needing to set up language servers or the like _at all_. I can define +them as part of the _project environment_ and then trust the tools I build on +top of to take care of that for me. + +Give Nix a try. It's worth at least that much in my opinion. diff --git a/blog/thoughts-on-nix-2020-01-28.markdown b/blog/thoughts-on-nix-2020-01-28.markdown index 164e8af..7acdd91 100644 --- a/blog/thoughts-on-nix-2020-01-28.markdown +++ b/blog/thoughts-on-nix-2020-01-28.markdown @@ -9,6 +9,10 @@ tags: # Thoughts on Nix +EDIT(M02 20 2020): I've written a bit of a rebuttal to my own post +[here](https://christine.website/blog/i-was-wrong-about-nix-2020-02-10). I am +keeping this post up for posterity. + I don't really know how I feel about [Nix][nix]. It's a functional package manager that's designed to help with dependency hell. It also lets you define packages using [Nix][nixlang], which is an identically named yet separate thing. diff --git a/nix/sources.json b/nix/sources.json index 8565fcc..225728a 100644 --- a/nix/sources.json +++ b/nix/sources.json @@ -10,5 +10,11 @@ "type": "tarball", "url": "https://github.com/adisbladis/vgo2nix/archive/1288e3dbf23ed79cef237661225df0afa30f8510.tar.gz", "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz" + }, + "xepkgs": { + "ref": "master", + "repo": "https://tulpa.dev/Xe/nixpkgs", + "rev": "71488e7dd46c9530d6781ab7845e6f720591a0b0", + "type": "git" } } @@ -1,5 +1,6 @@ let pkgs = import <nixpkgs> { }; sources = import ./nix/sources.nix; - vgo2nix = (import sources.vgo2nix { }); -in pkgs.mkShell { buildInputs = [ pkgs.go pkgs.niv vgo2nix ]; } + xepkgs = import sources.xepkgs { }; + vgo2nix = import sources.vgo2nix { }; +in pkgs.mkShell { buildInputs = [ pkgs.go pkgs.niv xepkgs.gopls vgo2nix ]; } |