blog: GopherCon EU 2023 demo details

Signed-off-by: Xe Iaso <me@xeiaso.net>

2 files changed, 338 insertions, 0 deletions

diff --git a/blog/gophercon-eu-demo.markdown b/blog/gophercon-eu-demo.markdown
new file mode 100644
index 0000000..76ffbd8
--- /dev/null
+++ b/blog/gophercon-eu-demo.markdown
@@ -0,0 +1,203 @@
+---
+title: Of course the network can be a filesystem
+date: 2023-06-30
+tags:
+ - wasm
+ - wasi
+ - wazero
+ - golang
+ - rust
+---
+
+<div class="warning"><xeblog-conv name="Cadey"
+mood="enby"><big>Spoiler alert!</big> This references details for my
+GopherCon EU talk Reaching the Unix Philosophy's Logical Conclusion
+with WebAssembly. This may ruin some of the cosmic horror feeling that
+I tried to inspire in the audience by gradually revealing the moving
+parts and then crashing them all together into one "oh, oh god, why,
+no" feeling when the live demo happened. My talk page and the video
+will go up in August.</xeblog-conv></div>
+
+<center><figure><xeblog-picture
+path="blog/2023/berlin/strasse"></xeblog-picture><figcaption>A picture
+of a side street in East Berlin, near the conference venue for
+GopherCon EU 2023.</figcaption></center>
+
+One of the fun parts about doing developer relations work is that you
+get to write and present interesting talks to programming communities.
+Recently I travled to Berlin to give a talk at GopherCon EU. It was my
+first time in Berlin and I've enjoyed my time there (more details
+later). This year at [GopherCon EU](https://gophercon.eu) I gave a
+talk about WebAssembly. Specifically how to use WebAssembly in new and
+creative ways by abusing facts about how Unix works. During that talk
+I covered a lot of the basic ideas of Unix's design (file i/o is
+device IO, the filesystem is for discovering new files, programs
+should be filters) and then put all the parts together into a live
+demo that I don't think was explained as well as I could have done it.
+
+Today I'm going to go into more details about how that live demo
+worked in ways that I couldn't becaise I was on a time limit for my talk.
+
+<xeblog-conv name="Mara" mood="hacker">If you just want to read the
+code, [look here in the /x/
+repo](https://github.com/Xe/x/tree/master/conferences/gceu23).</xeblog-conv>
+
+## Dramatis Personae
+
+I glossed over this diagram in the talk, but here's the overall
+flowchart of all the moving parts in my live demo (for those of you on
+screen readers, skip this image description because I'm going to
+explain things in detail):
+
+![A diagram, explained below](/static/img/gceu23-demo.svg)
+
+There's two main components in this demo: `yuechu` and `aiyou` (extra
+credit if you can be the first person to tell me what the origin of
+those names are). `yuechu` is an echo server, but it takes all lines
+of user input and then feeds them into a WebAssembly program. The
+output of that WebAssembly program is fed back to the user. You can
+change the behavior that `yuechu` does by changing the WebAssembly
+program that it uses as a filter.
+
+`aiyou` is a WebAssembly runtime that exposes the network as a
+filesystem. It doesn't really do anything special and doesn't pass
+through some fundamentally assumed things like command line args and
+other filesystem mounts. It really just is intended to act as an echo
+client for my demo. The most exciting part of it is the `ConnFS` type,
+which exposes the network as a filesystem.
+
+<xeblog-conv name="Aoi" mood="grin" standalone>I get it! If sockets
+really are files, then the network can be a filesystem,
+technically!</xeblog-conv>
+
+Otherwise most of this is really boring
+[Rust](https://github.com/Xe/x/blob/master/conferences/gceu23/wasip1/echoclient.rs)
+and
+[Go](https://github.com/Xe/x/blob/master/conferences/gceu23/cmd/yuechu/main.go)
+code. The real exciting part is that it's embedding Rust code into a
+Go process without having to use the horrors of CGo.
+
+### ConnFS
+
+In [Wazero](https://wazero.io/), you can mount a filesystem to a WASI
+program. You can also use one of the Wazero library types to mount
+multiple filesystems into the same thing, namespaced much like they
+are in the Linux kernel. In Linux these filesystems are usually either
+implemented by kernel drivers, or programs that use
+[FUSE](https://www.kernel.org/doc/html/latest/filesystems/fuse.html)
+to act as a filesystem as far as the kernel cares.
+
+In Go, we have [`io/fs.FS`](https://pkg.go.dev/io/fs#FS) as a
+fundamental building block for making things that quack like
+filesystems do. `io/fs` is fairly limited in most cases, but the ways
+that Wazero uses it can make things fun. One of the main ways that
+`io/fs` falls over in the real world is that files opened from an
+`io/fs` filesystem don't normally have a `.Write` method exposed.
+
+However, an [`io/fs` file](https://pkg.go.dev/io/fs#File) is an
+_interface_. In Go, interfaces are views onto types so that you can
+expose the same API for different backend implementations (writing a
+file to another file, standard out, and connections). The `File`
+interface doesn't immediately look like it has a `.Write` method, but
+there's nothing that says there can't be a `.Write` method under the
+interface wrapper.
+
+In Wazero, if you have your files implement the `.Write` call, they
+will just work. Write calls in WASI will just automagically get fed
+into your filesystem implementation.
+
+In my talk I said that these methods are common to both sockets and
+files:
+
+- `open()`
+- `close()`
+- `read()`
+- `write()`
+
+So you can use this to shim filesystem operations over to network
+operations. I did exactly this with
+[ConnFS](https://github.com/Xe/x/blob/6e8d83bb628cc3fff6b6bfc22cc7f769a02b934f/conferences/gceu23/cmd/aiyou/main.go#L61-L91)
+in my demo program `aiyou`.
+
+<xeblog-conv name="Aoi" mood="wut">Uhhh, isn't this going to fail
+horriffically in the real world when any network hiccups at all
+happen? This doesn't seem like it would be the most stable in the long
+term.</xeblog-conv>
+<xeblog-conv name="Cadey" mood="percussive-maintenance">Well, yes this
+isn't going to work very well in the real world. However, this demo is
+a really unique case because it connects to localhost (so you don't
+have to worry about network stability) and only runs for about 30
+seconds each run (so you don't have to worry about the philosophical
+horror of long-lived network connections). In practice, you can use
+[`/dev/tcp` in
+bash](https://andreafortuna.org/2021/03/06/some-useful-tips-about-dev-tcp/)
+to do most of the same thing as ConnFS.</xeblog-conv>
+<xeblog-conv name="Aoi" mood="coffee">Why am I not surprised that bash
+does something cursed like that.</xeblog-conv>
+
+### The server
+
+All that's left in the stack is the echo server, which really is the
+boring part of this demo. The echo server listens on port 1997 (the
+significance of this number is an exercise for the reader and
+definietly not the result of typing a random four digit number that
+was free on my development box) and every time a connection is
+accepted it tries to read a line of input from the other side. When it
+gets a line of input, it runs that through the WebAssembly program and
+returns the results to the user.
+
+That's about it really.
+
+## Programs are like functions for your shell
+
+This lets you use programs as functions. Stdin and flags become args,
+stdout becomes the result. I go into more detail about this in [this
+talk](https://xeiaso.net/talks/wazero-lightning-2023)
+based on [this article](https://xeiaso.net/blog/carcinization-golang).
+This is something we use at Tailscale for our fediverse bot.
+Specifically for parsing Mastodon HTML.
+
+So realistically, if you can use something as stupid as the network as
+a filesystem, you can use _anything_ as a filesystem. The cloud's the
+limit! But do keep in mind that any complicated abomination of a
+code-switched mess between Go and Rust can and will have a cost and if
+you use this ability irresponsibly I retain the right to take that
+power away from you. Don't ask how I'd do it.
+
+## Some pictures
+
+I've never been to Berlin before and I took some time to take pictures
+with my dslr. I think the results are pretty good. I've attached some
+of my favorites:
+
+<center><figure><xeblog-picture
+path="blog/2023/berlin/bruke"></xeblog-picture><figcaption>This is a
+picture framed through part of a bridge in Berlin. It's worth noting
+that the halation effect of the light bleeding past the bricks is
+really difficult to capture with smartphone cameras, but absolutely
+trivial with a dslr.</figcaption></center>
+
+<center><figure><xeblog-picture
+path="blog/2023/berlin/respite"></xeblog-picture><figcaption>I've been
+trying to refine my technique with the dslr and part of it is framing
+photos to guide interest in the way I want it to flow. This makes the
+bench look like an "accessory" to the main focus of the image, the
+graffiti. I like taking pictures of graffiti when I travel because
+that can tell you a lot about the local culture.</figcaption></center>
+
+<center><figure><xeblog-picture
+path="blog/2023/berlin/sunset"></xeblog-picture><figcaption>I loved
+capturing this sunset from a boat in the middle of the river that
+divided West and East Berlin. This one took a couple tries to avoid
+being photobombed by part of the boat, but I'm very happy with the
+result.</figcaption></center>
+
+It's been great fun. I'd love to come back to Berlin in the future.
+I'm considering getting some of the better photos printed and might
+sign some to send to my patrons. Let me know what you think!
+
+I'm going to upload more of the photos to my blog later, I need to
+invent a new "photo gallery" feature for my blog engine. I could use
+something like Instagram or Google Drive for this, but I really like
+the tactility of having everything on my infrastructure. I'll figure
+something out.
diff --git a/static/img/gceu23-demo.svg b/static/img/gceu23-demo.svg
new file mode 100644
index 0000000..6baf4cd
--- /dev/null
+++ b/static/img/gceu23-demo.svg
@@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by graphviz version 2.40.1 (20161225.0304)
+ -->
+<!-- Title: G Pages: 1 -->
+<svg width="655pt" height="458pt"
+ viewBox="0.00 0.00 655.00 458.40" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 454.4)">
+<title>G</title>
+<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-454.4 651,-454.4 651,4 -4,4"/>
+<g id="clust1" class="cluster">
+<title>cluster_0</title>
+<polygon fill="#d3d3d3" stroke="#d3d3d3" points="8,-121.6 8,-409.6 176,-409.6 176,-121.6 8,-121.6"/>
+<text text-anchor="middle" x="92" y="-393" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">aiyou</text>
+</g>
+<g id="clust2" class="cluster">
+<title>cluster_1</title>
+<polygon fill="#d3d3d3" stroke="#d3d3d3" points="254,-8 254,-442.4 639,-442.4 639,-8 254,-8"/>
+<text text-anchor="middle" x="446.5" y="-425.8" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">yuechu</text>
+</g>
+<g id="clust3" class="cluster">
+<title>cluster_2</title>
+<polygon fill="#c0c0c0" stroke="#c0c0c0" points="532,-16 532,-409.6 631,-409.6 631,-16 532,-16"/>
+<text text-anchor="middle" x="581.5" y="-393" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">wasm program</text>
+</g>
+<!-- echoclient -->
+<g id="node1" class="node">
+<title>echoclient</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="92" cy="-358.8" rx="75.82" ry="18"/>
+<text text-anchor="middle" x="92" y="-354.6" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">echoclient.wasm</text>
+</g>
+<!-- ConnFS -->
+<g id="node2" class="node">
+<title>ConnFS</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="61" cy="-253.2" rx="42.8984" ry="18"/>
+<text text-anchor="middle" x="61" y="-249" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">ConnFS</text>
+</g>
+<!-- echoclient&#45;&gt;ConnFS -->
+<g id="edge1" class="edge">
+<title>echoclient&#45;&gt;ConnFS</title>
+<path fill="none" stroke="#000000" d="M75.2889,-341.0016C71.1181,-335.5511 67.1629,-329.2741 64.7938,-322.8 60.0178,-309.7489 58.7635,-294.4005 58.8322,-281.4425"/>
+<polygon fill="#000000" stroke="#000000" points="62.3374,-281.3278 59.1529,-271.2229 55.3409,-281.1082 62.3374,-281.3278"/>
+<text text-anchor="middle" x="117.1031" y="-301.8" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000"> &#160;mounted at /dev &#160;</text>
+</g>
+<!-- connFile -->
+<g id="node3" class="node">
+<title>connFile</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="92" cy="-147.6" rx="53.3129" ry="18"/>
+<text text-anchor="middle" x="92" y="-143.4" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">connection</text>
+</g>
+<!-- ConnFS&#45;&gt;connFile -->
+<g id="edge2" class="edge">
+<title>ConnFS&#45;&gt;connFile</title>
+<path fill="none" stroke="#000000" d="M55.5304,-235.3385C52.0749,-220.8453 49.4204,-200.1724 56.251,-183.6 57.9974,-179.3628 60.5107,-175.3609 63.407,-171.6737"/>
+<polygon fill="#000000" stroke="#000000" points="66.1088,-173.9039 70.2101,-164.1351 60.912,-169.2141 66.1088,-173.9039"/>
+<text text-anchor="middle" x="113.3745" y="-196.2" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000"> &#160;opened to yuechu &#160;</text>
+</g>
+<!-- lis -->
+<g id="node4" class="node">
+<title>lis</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="319" cy="-358.8" rx="56.7775" ry="18"/>
+<text text-anchor="middle" x="319" y="-354.6" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">net.Listener</text>
+</g>
+<!-- connFile&#45;&gt;lis -->
+<g id="edge3" class="edge">
+<title>connFile&#45;&gt;lis</title>
+<path fill="none" stroke="#000000" d="M131.0631,-160.0366C144.2667,-165.6619 158.3465,-173.3887 169,-183.6 200.6837,-213.9683 187.1068,-236.9715 214.575,-271.2 235.0622,-296.7294 263.8983,-320.2462 286.079,-336.4716"/>
+<polygon fill="#000000" stroke="#000000" points="284.0593,-339.3301 294.2229,-342.3221 288.1434,-333.645 284.0593,-339.3301"/>
+<text text-anchor="middle" x="244.7125" y="-257.4" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">TCP</text>
+<text text-anchor="middle" x="244.7125" y="-240.6" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">connection</text>
+</g>
+<!-- conn -->
+<g id="node5" class="node">
+<title>conn</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="478" cy="-147.6" rx="46.3511" ry="18"/>
+<text text-anchor="middle" x="478" y="-143.4" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">net.Conn</text>
+</g>
+<!-- lis&#45;&gt;conn -->
+<g id="edge4" class="edge">
+<title>lis&#45;&gt;conn</title>
+<path fill="none" stroke="#000000" d="M306.7424,-341.0203C290.9304,-315.6648 267.8438,-268.5065 289.1114,-235.2 318.773,-188.7478 380.1048,-166.5545 424.861,-156.2012"/>
+<polygon fill="#000000" stroke="#000000" points="425.6876,-159.6033 434.7104,-154.05 424.1939,-152.7645 425.6876,-159.6033"/>
+<text text-anchor="middle" x="352.9443" y="-249" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000"> &#160;accepted connection &#160;</text>
+</g>
+<!-- stdin -->
+<g id="node7" class="node">
+<title>stdin</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="570" cy="-42" rx="29.6339" ry="18"/>
+<text text-anchor="middle" x="570" y="-37.8" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">stdin</text>
+</g>
+<!-- conn&#45;&gt;stdin -->
+<g id="edge5" class="edge">
+<title>conn&#45;&gt;stdin</title>
+<path fill="none" stroke="#000000" d="M493.1919,-130.1623C508.5327,-112.5537 532.334,-85.234 549.3743,-65.6747"/>
+<polygon fill="#000000" stroke="#000000" points="552.0629,-67.9169 555.9928,-58.0778 546.7849,-63.3187 552.0629,-67.9169"/>
+<text text-anchor="middle" x="566.7465" y="-99" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">input from</text>
+<text text-anchor="middle" x="566.7465" y="-82.2" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">user</text>
+</g>
+<!-- logger -->
+<g id="node6" class="node">
+<title>logger</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="475" cy="-253.2" rx="48.6989" ry="18"/>
+<text text-anchor="middle" x="475" y="-249" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">ln.Logger</text>
+</g>
+<!-- stdout -->
+<g id="node8" class="node">
+<title>stdout</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="576" cy="-253.2" rx="34.7903" ry="18"/>
+<text text-anchor="middle" x="576" y="-249" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">stdout</text>
+</g>
+<!-- stdout&#45;&gt;conn -->
+<g id="edge6" class="edge">
+<title>stdout&#45;&gt;conn</title>
+<path fill="none" stroke="#000000" d="M560.6875,-236.7C544.6242,-219.391 519.1994,-191.9944 500.7806,-172.1473"/>
+<polygon fill="#000000" stroke="#000000" points="503.2222,-169.6329 493.8543,-164.6838 498.0912,-174.3946 503.2222,-169.6329"/>
+<text text-anchor="middle" x="577.7465" y="-204.6" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000"> &#160;output from</text>
+<text text-anchor="middle" x="577.7465" y="-187.8" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">program</text>
+</g>
+<!-- stderr -->
+<g id="node9" class="node">
+<title>stderr</title>
+<ellipse fill="#ffffff" stroke="#ffffff" cx="573" cy="-358.8" rx="33.0411" ry="18"/>
+<text text-anchor="middle" x="573" y="-354.6" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">stderr</text>
+</g>
+<!-- stderr&#45;&gt;logger -->
+<g id="edge7" class="edge">
+<title>stderr&#45;&gt;logger</title>
+<path fill="none" stroke="#000000" d="M557.6875,-342.3C541.6242,-324.991 516.1994,-297.5944 497.7806,-277.7473"/>
+<polygon fill="#000000" stroke="#000000" points="500.2222,-275.2329 490.8543,-270.2838 495.0912,-279.9946 500.2222,-275.2329"/>
+<text text-anchor="middle" x="564.4341" y="-310.2" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">error</text>
+<text text-anchor="middle" x="564.4341" y="-293.4" font-family="Iosevka Aile Iaso" font-size="14.00" fill="#000000">messages</text>
+</g>
+</g>
+</svg>