diff options
| author | Xe Iaso <me@christine.website> | 2023-09-30 10:36:37 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-09-30 10:36:37 -0400 |
| commit | ac6a3df0d18cc73524c0096d954a57d24cad5669 (patch) | |
| tree | 81474177d730440657f490ae29892d62392251ea /lume/src/talks | |
| parent | cbdea8ba3fca9a663778af71f8df5965aeb6c090 (diff) | |
| download | xesite-ac6a3df0d18cc73524c0096d954a57d24cad5669.tar.xz xesite-ac6a3df0d18cc73524c0096d954a57d24cad5669.zip | |
Xesite V4 (#723)
* scripts/ditherify: fix quoting
Signed-off-by: Xe Iaso <me@xeiaso.net>
* clean up some old files
Signed-off-by: Xe Iaso <me@xeiaso.net>
* import site into lume
Signed-off-by: Xe Iaso <me@xeiaso.net>
* initial go code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* move vods index to top level
Signed-off-by: Xe Iaso <me@xeiaso.net>
* remove the ads
Signed-off-by: Xe Iaso <me@xeiaso.net>
* internal/lume: metrics
Signed-off-by: Xe Iaso <me@xeiaso.net>
* delete old code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* load config into memory
Signed-off-by: Xe Iaso <me@xeiaso.net>
* autogenerate data from dhall config
Signed-off-by: Xe Iaso <me@xeiaso.net>
* various cleanups, import clackset logic
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update signalboost.dhall (#722)
Added myself, and also fixed someone’s typo
* Add Connor Edwards to signal boost (#721)
* add cache headers
Signed-off-by: Xe Iaso <me@xeiaso.net>
* move command to xesite folder
Signed-off-by: Xe Iaso <me@xeiaso.net>
* xesite: listen for GitHub webhook push events
Signed-off-by: Xe Iaso <me@xeiaso.net>
* xesite: 5 minute timeout for rebuilding the site
Signed-off-by: Xe Iaso <me@xeiaso.net>
* xesite: add rebuild metrics
Signed-off-by: Xe Iaso <me@xeiaso.net>
* xesite: update default variables
Signed-off-by: Xe Iaso <me@xeiaso.net>
* don't commit binaries oops lol
Signed-off-by: Xe Iaso <me@xeiaso.net>
* lume: make search have a light background
Signed-off-by: Xe Iaso <me@xeiaso.net>
* add a notfound page
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fetch info from patreon API
Signed-off-by: Xe Iaso <me@xeiaso.net>
* create contact page
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Toot embedding
Signed-off-by: Xe Iaso <me@xeiaso.net>
* attempt a docker image
Signed-off-by: Xe Iaso <me@xeiaso.net>
* lume: fix deno lock
Signed-off-by: Xe Iaso <me@xeiaso.net>
* add gokrazy post
Signed-off-by: Xe Iaso <me@xeiaso.net>
* cmd/xesite: go up before trying to connect to the saas proxy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* blog: add Sine post/demo
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: bri <284789+b-@users.noreply.github.com>
Co-authored-by: Connor Edwards <38229097+cedws@users.noreply.github.com>
Diffstat (limited to 'lume/src/talks')
| -rw-r--r-- | lume/src/talks/_data.yml | 3 | ||||
| -rw-r--r-- | lume/src/talks/asg-2023-nixos.mdx | 429 | ||||
| -rw-r--r-- | lume/src/talks/conf42-static-analysis.mdx | 422 | ||||
| -rw-r--r-- | lume/src/talks/how-my-website-works.mdx | 437 | ||||
| -rw-r--r-- | lume/src/talks/irc-why-it-failed-2018-05-17.md | 11 | ||||
| -rw-r--r-- | lume/src/talks/nixos-pain-2021-11-10.md | 400 | ||||
| -rw-r--r-- | lume/src/talks/progressive-webapp-conversion-in-5-minutes-2019-01-28.md | 9 | ||||
| -rw-r--r-- | lume/src/talks/rustconf-2022-sheer-terror-pam.mdx | 507 | ||||
| -rw-r--r-- | lume/src/talks/surreal-horror-pam-2021-11-09.md | 265 | ||||
| -rw-r--r-- | lume/src/talks/systemd-the-good-parts-2021-05-16.md | 252 | ||||
| -rw-r--r-- | lume/src/talks/thinking-different-2018-11-03.md | 9 | ||||
| -rw-r--r-- | lume/src/talks/unix-philosophy-logical-extreme-wasm.mdx | 218 | ||||
| -rw-r--r-- | lume/src/talks/virtual-networks-pulumi-tailscale.mdx | 69 | ||||
| -rw-r--r-- | lume/src/talks/wasm-abi.mdx | 938 | ||||
| -rw-r--r-- | lume/src/talks/wazero-lightning-2023.mdx | 165 | ||||
| -rw-r--r-- | lume/src/talks/webassembly-on-the-server-system-calls-2019-05-31.md | 135 |
16 files changed, 4269 insertions, 0 deletions
diff --git a/lume/src/talks/_data.yml b/lume/src/talks/_data.yml new file mode 100644 index 0000000..79702c4 --- /dev/null +++ b/lume/src/talks/_data.yml @@ -0,0 +1,3 @@ +layout: talk.njk +type: talk +index: true
\ No newline at end of file diff --git a/lume/src/talks/asg-2023-nixos.mdx b/lume/src/talks/asg-2023-nixos.mdx new file mode 100644 index 0000000..2329ace --- /dev/null +++ b/lume/src/talks/asg-2023-nixos.mdx @@ -0,0 +1,429 @@ +--- +title: "Making NixOS modules for fun and (hopefully) profit" +date: 2023-09-15 +slides_link: https://drive.google.com/file/d/1h0Y8PUREuSrrSStmJMJ4MFHm0h53tf5a/view?usp=sharing +tags: + - nix + - nixos + - tailscale +--- + +<XeblogConv name="Cadey" mood="coffee" standalone>There was an A/V glitch with the recording, my slides were intended to be black and white, but they somehow came out as purple and green. This couldn't be fixed even when trying several HDMI cables. If this becomes an issue I may re-record this talk in my home studio.</XeblogConv> + +<XeblogVideo path="talks/2023/asg-nixos/video/proper"></XeblogVideo> + +- [Link to the video on the CCC + website](https://media.ccc.de/v/all-systems-go-2023-214-writing-your-own-nixos-modules-for-fun-and-hopefully-profit) +- [YouTube link](https://youtu.be/SzyuLVzS5Fg) + +<XeblogPicture path="talks/2023/asg-nixos/001"></XeblogPicture> + +Good morning everyone! Say this happens to you: you've been coding nonstop on something you want to share with your friends and it works on your MacBook. You want it to stay up when your MacBook goes to sleep or you get on a plane or something, and all you have to do is the easy task of putting it into production. It's just simple, right? + +Just add a Dockerfile, they say! So you do that and then you have a Docker image that you can push to your target machine and then you find out that you can't just push it from machine to machine, you have to push it to a registry. + +So you make an account on the Docker hub only to find out that their rate limits are very aggressive so you have to move to something like GHCR and aggressively cache all your images there so you don't run afoul of the comically small Docker Hub rate limits which will block your attempts to deploy it to your cloud provider of choice. + +So you do that and you pull this on a VM running on someone else's computer, and then you need to figure out the other fun part: + +You need to configure nginx. Of course it uses its own bespoke configuration language that no other program on the planet uses (this is an unfortunately common pattern in our industry) so it's even more googling for that. But then you realize you need to configure the real final boss of the internet: + +<XeblogPicture path="talks/2023/asg-nixos/006"></XeblogPicture> + +DNS. It's never DNS until it's always DNS. So you install the artist formerly known as Terraform, lego, and provision your DNS and HTTPS certificates (because of course nginx doesn't just natively have this support in anno dominium two thousand and twenty three like any sensible HTTP reverse proxy should). And then you're finally done. It's taken you an hour to hack up the service and a whole 8 hours to research and implement everything to deploy it. This is _madness_. Why do we have to put up with this? + +<XeblogPicture path="talks/2023/asg-nixos/007"></XeblogPicture> + +The koolaid runs deep in the cloud too, if you're not careful you'll end up accidentally making an entire event sourcing platform with an unrealistic amount of complexity to manage something as simple as a tamagochi. You're just trying to make an HTTP service show up on the internet, you don't need to know what an ALB, EKS, ECS, IAM, or PFA is. + +Of course, complain about this online and a certain tangerine community funded by big YAML will decry that you should use Kubernetes to simplify all this down to "simple" and "easy to understand" things, conveniently ignoring that they use a _string templating language_ for _structured data_. + +There has _got_ to be something simpler, right? What if you didn't have to deal with nearly any of that? What if you could just push and run your binary on a _home server_ and then access it? No dealing with the cloud. No dealing with security groups or IAM or DNS or HTTPS or any of the slings and arrows of outrageous investment. What if you could just describe the state of the system you want instead of going three layers deep into a side of devops hell that you will never return from unscathed? + +<XeblogPicture path="talks/2023/asg-nixos/010"></XeblogPicture> + +This is the real value of NixOS. Today I'm going to show you how to turn an arbitrary Go program into a NixOS service and then I'll expose it to the world thanks to Tailscale Funnel. This means you can link it to your group chat of friends and restore balance to the force. Or whatever it is you zoomers do in group chats. + +All that said, let me introduce myself. I'm Xe Iaso, I write that one blog that you keep finding when you google Nix and NixOS stuff. I'm a writer, gamer, philosopher of chaos magick, and have a critical blogging addiction. + +<XeblogPicture path="talks/2023/asg-nixos/012"></XeblogPicture> + +Today I'm going to cover a few core things so you can make your own NixOS modules: I'm going to cover what a NixOS module is and why you should care, the parts of one, how to make your own, and then I'm going to tempt the demo gods by doing a live deployment to a virtual machine on my MacBook. + +Before I get started though, let's get some exercise in. Raise your hand if this is your first exposure to Nix and/or NixOS. + +(About half the room raises their hands) + +Alright, thanks. + +Raise your hand if you've ever used it before. + +(The other half of the room raises their hands) + +That's about what I expected. + +How about if you have it installed on a server at home? + +(The same people raise their hands) + +Okay, okay, I see. + +How about if you're one of the lucky few where your employer uses it in production? + +(Only a few of those people raise their hands) + +Oh, wow, okay. That makes sense. You can lower your hands now. + +Just so we're on the same page, Nix is a package manager that lets you declare all of the inputs for a program and get the same output from the build process. + +```go +func Build( + inputs []Package, + steps BuildInfo, +) ( + output Package, + err Error, +) +``` + +One of the main ways that Nix stands out from the crowd is the idea that package builds are functions. They effectively take in inputs, use them against some instructions, and then either return a package or the build fails due to an error. Because there were no other options at the time, Nix uses its own programming language also named Nix to define packages. + +Remember, Nix was the result of lamentations at the state of software and this was the result. + +<XeblogPicture path="talks/2023/asg-nixos/019"></XeblogPicture> + +To help you understand, I've put up this helpful diagram. It uses rainbow comic sans so you know it's legit. Nixpkgs the standard library uses Nix the language, but it is not NixOS the operating system. I like to think about NixOS like this: + +NixOS is the natural consequence of using Nix to build Linux systems. You can think about NixOS as a bunch of prebaked snippets of configuration that you can combine info a running system that does what you want. Each of those snippets is called a module. Nixpkgs (the standard library in Nix land) ships with a bunch of them that do things from compiling systemd to configuring Tailscale for you. Here's a simple NixOS module from my homelab: + +```nix +{ config, pkgs, ... }: + +{ + services.prometheus.exporters.node.enable = true; +} +``` + +A NixOS module is a function that takes the current state of the world and returns things to change in it. The module I'm showing here is from my homelab, specifically the part that enables the prometheus node exporter so that I can report when machines suddenly go offline or their hard drives are going bad. This is a very simple example. When you import it, it always takes effect. There's no flags to enable it or disable it. This is fine for my usecase however, because I want my homelab cluster to always be monitored. Things get a lot more fun when you add options into the mix: + +```nix +{ lib, config, pkgs, ... }: + +with lib; + +let cfg = config.within.vim; +in { + options.within.vim.enable = mkEnableOption "Enables Within's vim config"; + + config = mkIf cfg.enable { + home.packages = [ pkgs.vim ]; + home.file.".vimrc".source = ./vimrc; + }; +} +``` + +Compare it to this module, this is a dot file management module that sets up my vimrc on my machines. I have the option `within.vim.enable`, and if that is set to true, the vim configuration is dropped in place. If it's not set to true, it won't put the vim configuration in the system. NixOS modules have options and configuration outputs. Options let you customize the configuration to meet your spacebar heating needs. + +```nix +{ ... }: + +{ + imports = [ ./vim ]; + within.vim.enable = true; +} +``` + +To use this, you'd add the path to the file to an `imports` output of the module, then add a `within.vim.enable = true` statement inside your home-manager configuration. + +The state of the world is the input, and any new changes are the outputs. This lets you build a Linux system _exactly_ the way you want to. It's just a new and interesting way to write a function. + +```nix +services.nginx.virtualHosts."xeiaso.net" = { + locations."/" = { + proxyPass = "http://unix:${toString cfg.sockPath}"; + proxyWebsockets = true; + }; + forceSSL = cfg.useACME; + useACMEHost = "xeiaso.net"; + extraConfig = '' + access_log /var/log/nginx/xesite.access.log; + ''; +}; +``` + +Above all else: you can configure programs like nginx directly in your NixOS configuration without having to learn how to write nginx config, saving you from having to configure every single program on your system in its own bespoke ways. + +(Pause) + +<xeblog-sticker name="Mara" mood="happy"></xeblog-sticker> + +Of course, things become a lot more fun when you can build your own NixOS modules that have your own programs running on your own machines. Let's do that with an example program that shows quotes from the legendary British science fiction author Douglas Adams. + +<XeblogPicture path="talks/2023/asg-nixos/daquotes"></XeblogPicture> + +This is what the end result will look like. It'll be the quotes on a screen that refreshes every time you press F5. This will let you spread the undeniable wisdom of the late and great Douglas Adams, author of the five part trilogy The Hitchhiker's Guide to the Galaxy. + + + +So overall, the infrastructure setup will look like this: my MacBook and the VM are both connected to each other with Tailscale. When I enable Tailscale Funnel, the VM is going to have its HTTPS port opened up to the public internet so that you can visit this service running on a VM, on my MacBook, on conference Wi-Fi. Let's hope the demo gods are in our favour! + +<XeblogPicture path="talks/2023/asg-nixos/028"></XeblogPicture> + +By the way, Tailscale's gonna take care of the DNS and Let's Encrypt problems for us. No having to figure that out in the conference! + +```nix +{ + description = "Douglas Adams quotes"; + + # Nixpkgs / NixOS version to use. + inputs.nixpkgs.url = "nixpkgs/nixos-unstable"; + + outputs = { self, nixpkgs }: + let # ... + in + { + packages = ...; + nixosModules.default = ...; + devShell = ...; + checks.x86_64-linux = ...; + } +} +``` + +I've opened a VS Code session with an "empty" flake configuration. Nix flakes let you create a set of packages, development environments, NixOS modules, and even end to end integration tests. To start, this flake will import nixpkgs: + +```nix +# Nixpkgs / NixOS version to use. +inputs.nixpkgs.url = "nixpkgs/nixos-unstable"; +``` + +And then it declares a devShell for all of the developer dependencies: + +```nix +devShell = forAllSystems (system: + let pkgs = nixpkgsFor.${system}; + in with pkgs; + mkShell { + buildInputs = + [ go_1_21 gotools go-tools gopls nixpkgs-fmt nodejs yarn ]; + }); +``` + +This is relevant because I want you to imagine a world where your compilers aren't in your shell by default. This devShell configuration adds the packages relevant to the project to the development environment. This is a Go project with some CSS managed by Tailwind, so it's got the Go compiler, some Go development tools, npm, and yarn. A pretty normal set of things really. + +To enter the development environment, run `nix develop`. I'm going to run `yarn start:css` in another shell to rebuild when any of the template files change. + +Now that we have that, let's see how we would add a package to the flake. One of the flake output kinds is Nix packages, so we make an output named packages and paste in some boilerplate to get a Go package working: + +```nix +packages = forAllSystems (system: + let + pkgs = nixpkgsFor.${system}; + in + { + default = pkgs.buildGo121Module { + pname = "douglas-adams-quotes"; + inherit version; + src = ./.; + vendorSha256 = null; + }; + }); +``` + +The package would look something like this. This forAllSystems / nixpkgsFor hack is something you can work around with flake-utils, but for right now I'm doing _everything_ manually. This is basically a bunch of predefined copies of nixpkgs for all the supported architectures, much like there's a devshell for every supported architecture. Either way, we get a Go module built into a package, and we define the dependency hash as null because this is only using the standard library. It's called default in the flake because it's best practice to name your package that. + +Just to test it, you can run `nix build` to build the default package in that flake.nix file: + +``` +nix build +``` + +Perfect! It builds! The binary is in ./result/bin/ and we can run it wherever we want. + +``` +$ ./result/bin/douglas-adams-quotes --help +Usage of ./result/bin/douglas-adams-quotes: + -addr string + listen address (default ":8080") + -slog-level string + log level (default "INFO") +``` + +If it didn't work we wouldn't get this far! + +```nix +nixosModules.default = { config, lib, pkgs, ... }: +with lib; +let + cfg = config.xe.services.douglas-adams-quotes; +in +{ + options.xe.services.douglas-adams-quotes = { + enable = mkEnableOption "Enable the Douglas Adams quotes service"; + + logLevel = mkOption { + type = with types; enum [ "DEBUG" "INFO" "ERROR" ]; + example = "DEBUG"; + default = "INFO"; + description = "log level for this application"; + }; + + port = mkOption { + type = types.port; + default = 8080; + description = "port to listen on"; + }; + + package = mkOption { + type = types.package; + default = self.packages.${pkgs.system}.default; + description = "package to use for this service (defaults to the one in the flake)"; + }; + }; + + config = mkIf cfg.enable { + systemd.services.douglas-adams-quotes = { + description = "Douglas Adams quotes"; + wantedBy = [ "multi-user.target" ]; + + serviceConfig = { + DynamicUser = "yes"; + ExecStart = "${cfg.package}/bin/douglas-adams-quotes --slog-level=${cfg.logLevel} --addr=:${toString cfg.port}"; + Restart = "on-failure"; + RestartSec = "5s"; + }; + }; + }; +}; +``` + +We defined the devShell to build the program development. We defined the package to build the software, and now we'll define the module to tell NixOS how to run the software. This is a basic NixOS module. It's defined inline to the flake for now, moving it to its own file is an exercise for the reader. + +Like I said before, a NixOS module is a function that takes in the state of the world and returns new additions to the state of the world. This NixOS module provides some options under `xe.services.douglas-adams-quotes` and then if the module is enabled, it creates a new systemd service to run it in. We're in the future, so we can use fancy things like DynamicUser to avoid having to run this service as root. + +```nix +options.xe.services.douglas-adams-quotes = { + enable = mkEnableOption "Enable the Douglas Adams quotes service"; + logLevel = ...; + port = ...; + package = ...; +}; +``` + +The real fun part comes when you define options for the service. Every one of these options correlates to CLI flags so you can change various options on the fly. It's good practice to map any non-secret configuration settings to options so that users can have easy escape hatches for changing things like the HTTP bind port or log level to get debug output. Secrets are a more complicated thing due to how Nix works, so we're not going to talk about those today. + +So we have everything we need now. We have development environment configuration, a package build, and finally a NixOS module to get the service running. The last step is to push it into prod. I have a NixOS virtual machine set up for this on my MacBook, but you may want to run this somewhere else, such as in Hyper-V on your gaming tower. Or maybe the cloud, I won't judge! + +Now we get to the fun part, enabling the NixOS module. I'm going to use the VS Code Tailscale extension to SSH in and open up the files in my VM, so lemme do that real quick. + +Let's peer into my VM's deployment flake and see what we can do to deploy it. This is a brand new, never opened VM, the only thing I did was set up a flake in /etc/nixos/flake.nix that imports the autogenerated configuration from the installer. This allows us to import things like the Douglas Adams Quotes service into the VM. + +```nix +{ + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + + outputs = { self, nixpkgs, ... }: { + nixosConfigurations.douglas-adams = nixpkgs.lib.nixosSystem { + system = "aarch64-linux"; + modules = [ + ./configuration.nix + ]; + }; + }; +} +``` + +Here's what the file looks like. I import nixos unstable, and then I create a nixos configuration for a machine named douglas-adams. This "modules" block has a list of NixOS module filenames or literal expressions. This lets you import NixOS modules from other flakes and define your own NixOS modules on the fly. + +```nix +{ + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + inputs.daquotes = { + url = "github:Xe/douglas-adams-quotes"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + + outputs = { self, nixpkgs, daquotes, ... }: { + nixosConfigurations.douglas-adams = nixpkgs.lib.nixosSystem { + system = "aarch64-linux"; + modules = [ + daquotes.nixosModules.default + ./configuration.nix + ({ pkgs, ... }: { + xe.services.douglas-adams-quotes = { + enable = true; + }; + }) + ]; + }; + }; +} +``` + +To make it import this, first we add a new input that points to the Douglas Adams quotes flake. This then gets threaded into the outputs function, we import the module, and finally enable it on the system. + +(Pause) + +That's it. That's how you enable the service on the VM. Now all that's left is SSHing in and running a nixos-rebuild to enable it. + +I just right-click on the douglas-adams node, ssh in with Tailscale handling the auth, and then run the magic nixos-rebuild command: `nixos-rebuild switch --flake /etc/nixos` and hit enter. + +And now the source code gets pulled, the package gets built, the service gets created and then we can see that the process is running. Let's prove that it's working: + +``` +[root@douglas-adams:/]# curl http://localhost:8080/quote.json | jq +{ + "quote":"The story so far: In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move.", + "person":"Douglas Adams", + "source":"The Restaurant at the End of the Universe", + "id":2 +} +``` + +Et voila. But, we aren't stopping there. I also enabled serving it with Tailscale Funnel so that you can see it on your phones: + +``` +tailscale serve https / http://localhost:8080 +tailscale funnel 443 on +``` + +Get your phones out, I'm gonna be showing a QR Code: + +<XeblogConv name="Mara" mood="hacker" standalone>For those playing the Xe Iaso blog extended universe home game, visit [m85-kongir.shark-harmonic.ts.net](https://m85-kongir.shark-harmonic.ts.net) instead.</XeblogConv> + +Scan this QR code. You can trust me, right? It's not gonna be a Rick Roll. I'm not that mean. When you do, you'll connect to my VM on my laptop on the conference wifi, yet still exposed to the public internet. + +(Pause for audience to discover that it does in fact work, applause) + +<XeblogPicture path="talks/2023/asg-nixos/031"></XeblogPicture> + +In conclusion, NixOS modules aren't hard. It's just options to configure systemd or nginx or even Tailscale. It's a function that takes in the state of the world and returns new parts to add to the whole. This gives you a nearly infinite level of composition and logistical freedom to implement whatever you want. Every systemd option is exposed as a NixOS flag. Your programs can become services trivially. It's just that easy. I promise. + +But, now you know how to make your own NixOS modules for fun and (hopefully) profit! + +<XeblogPicture path="talks/2023/asg-nixos/033"></XeblogPicture> + +Before we get this talk wrapped up, I just want to thank everyone on this list for helping me make this talk shine. Thanks everyone! + +(Pause for applause) + +<XeblogPicture path="talks/2023/asg-nixos/034"></XeblogPicture> + +And with that, I've been Xe! Thank you so much for coming to this talk. I hope you've had a good conference and I'll be wandering around in case you have any questions. I've posted a summary of the code samples on my blog at xeiaso.net so you can look into my code some more. + +Oh by the way, if you're looking, Tailscale is hiring. I know it's probably rare to see someone like this at a Linux conference but if you're a Windows expert please let me know. That role has been so hard to fill. + +I try to answer every question I can, but if I don't get to you, please email dynamicuser at xeserv dot us and I'll reply to your questions as soon as I can. + +Thanks again to the All Systems Go organizers for having me here and I hope you continue having a good day. Be well! + +--- + +There was a question about encrypted secrets in NixOS. I suggest using [agenix](https://github.com/ryantm/agenix) to have age-encrypted secrets in your NixOS configs. It has you encrypt things against SSH host public keys for your machines. It's a bit of a hack, but it works well enough that it's what I use in prod for my own stuff. This really needs to be solved upstream with proper handling of secret values at the Nix level. + +<XeblogConv name="Aoi" mood="wut">Why do you need to use something like agenix at all?</XeblogConv> +<XeblogConv name="Mara" mood="hacker">Every file in `/nix/store` is world-readable. Depending on your threat model and if your NixOS configs are open source, this can be fine. If your threat model includes public NixOS configs, this becomes less fine; especially when CI is brought into the mix. You wouldn't want someone to figure out what your secrets are in your CI flow and then exfiltrate [Tailscale authkeys](https://tailscale.com/kb/1085/auth-keys/) or something, that could be bad!</XeblogConv> + +--- + +[Script link](https://cdn.xeiaso.net/file/christine-static/talks/2023/asg-nixos/ASG-nixos.html) diff --git a/lume/src/talks/conf42-static-analysis.mdx b/lume/src/talks/conf42-static-analysis.mdx new file mode 100644 index 0000000..1798481 --- /dev/null +++ b/lume/src/talks/conf42-static-analysis.mdx @@ -0,0 +1,422 @@ +--- +title: How Static Code Analysis Prevents You From Waking Up at 3AM With Production on Fire +date: 2022-06-09 +slides_link: https://cdn.xeiaso.net/file/christine-static/talks/Conf42+SRE+2022.pdf +--- + +<xeblog-talk-warning></xeblog-talk-warning> + +<XeblogVideo path="talks/conf42-staticcheck/talk"></XeblogVideo> + +<XeblogConv name="Mara" mood="hacker"> + If that didn't work, try [here](https://youtu.be/cVUrScvthqs) for the YouTube + version! +</XeblogConv> + +<XeblogConv name="Cadey" mood="coffee"> + The talk video will be live at 2022 M06 10 at 13:00 EDT. It will not work if + you are reading this at the exact time of release or before it is released via + Patreon. +</XeblogConv> + + + +Hi, I’m Xe Iaso and today I’m going to talk about static analysis and how it +helps you engineer more reliable systems. This will help you make it harder for +incorrect code to blow up production at 3AM. There are a lot of tools out there +that can do this for a variety of languages, however I’m going to focus on Go +because that is what I am an expert in. In this talk I’ll cover the problem +space, some solutions you can apply today and how you can work with people to +engineer more reliable systems. + + + +As I said, I’m Xe. I’m the Archmage of Infrastructure at Tailscale. I’ve been an +SRE for long enough that I have moved over into developer relations. As a +disclaimer, this talk may contain opinions. None of these opinions are of my +employer. + +I’ll have a recording of this talk, slides, speaker notes, and a transcript of +up in a day or two after the conference. The QR code in the corner of the screen +will take you to my blog. + + + +When starting to think about the problem, I find it helps to start thinking +about the problem space. This usually means thinking about the total problem at +an incredibly high level. + +So let’s think about the problem space of compilers. At the highest possible +level, a compiler can take literally anything as input and maybe produce an +output. + + + +A compiler’s job is to take this anything, see if it matches a set of rules and +then produce an output of some kind. In the case of the Go compiler, this means +that the input needs to match the rules that the Go language has defined in its +specification. + + + +This human-readable specification outlines core rules of the Go language. These +include things like every `.go` file needs to be in a package, the need to +declare variables before using them, what core types are in the language, how to +deal with slices, etc. + +However this specification doesn’t define what _correct_ Go code is. It only +defines what _valid_ Go code is. This is normal for specifications of this kind, +ensuring correctness is an active field of research in computer science that +small scrappy startups like Google, Microsoft and Apple struggle with. + + + +As a result though, you can’t rely on the compiler itself from stopping +incorrect code to be deployed into production. A lot of trivial errors will be +stopped in the process, but it won’t stop more subtle errors. This is an +example of the kind of error that the Go compiler can catch by itself, if you +declare a value as an integer you can’t then put a string in it. They are +different types and the compiler will reject it. + + + +I know one of you out there is probably thinking something like “What about +other languages like Rust and Haskell? Aren’t those compilers known for +correctness?” + + + +That’s a good point, there are other languages that have more strict rules like +linear types and explicitly marking poking the outside world. However the kinds +of errors that are brought up in this talk can still happen in those languages, +even if it’s more difficult to do that by accident. + + + +Static analysis on top of your existing compiler lets you move closer to +correctness without going the maximalist route like when using Rust or Haskell. + + + +It’s a balance between pragmatism and correctness. The pragmatic solution and +the correct solution are always in conflict, so you need to find a way down the +middle. + + + +In general, proving everything is correct with static analysis is impossible. It +takes a theoretically infinite amount of time to tell if absolutely every facet +of the code is correct in every single way. This is a case where the perfect is +the enemy of the good, so here are some patterns for things that can be proven +with static analysis in Go: + + + +- Forgetting to close an HTTP response body +- Making typos in struct tags +- Ensuring that cancellable contexts get cancelled in trivially provable ways +- Writing invalid time formats +- Writing an invalid regular expression that would otherwise blow up at runtime + + + +These kinds of things are easy to prove and are enabled by default in `go vet` +and staticcheck. + +Also for the record, incorrect code won’t explode instantly upon it being run. +The devil is in the details of how it is incorrect and how those things can pile +up to create issues downstream. Incorrect code can also confuse you while trying +to debug it, which can make you waste time you could spend doing anything else. + + + +This is an example of Go code that will compile, will likely work, but is incorrect. + + + +This is incorrect because the HTTP response is read from, but never closed. +Failing to do this in Go will cause you to leak the resources associated with +the HTTP connection. When you close the response, it releases the connection so +that it can be used for other HTTP actions. + +If you don’t do this, you can easily run into a state where your server +application will run out of available sockets at 3AM. So you may be tempted to +fix it like this: + + + +However this is incorrect too. Look at where the `defer` is called. + +Let’s think about how the program flow will work. I’m going to translate this +into a diagram of how this program will be executed. + + + +This flowchart is another way to think about how this program is being executed. +It starts on the left side and flows to the end on the right. + + + +In this case we start with the http dot Get call and then defer closing the +response body. Then we check to see if there was an error or not. + + + +If there wasn’t an error, we can use the response and do something useful, then +the response body closes automatically due to the deferred close. Everything +works as expected. + + + +However if there was an error, something different happens. The error is +returned and then the scheduled Close call runs. The Close call assumes that the +response is valid, but it’s not. This results in the program panicking which is +a crash at 3AM. This is the kind of place that static analysis comes in to save +you. Let’s take a look at what `go vet` says about this code: + + + +It caught that error! To fix this we need to move the `defer` call to after the +error check like this: + + + +The response body is closed after we know it’s usable. This will work as we +expect in production. This is an example of how trivial errors can be fixed with +a little extra tooling without having to use an entirely maximalist approach. + + + +If you use `go test` then a large amount of `go vet` checks are run by default. +This covers a wide variety of common issues with trivial fixes that help move +your code towards the corresponding Go idioms. It’s limited to the subset of |