1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
---
title: "[talk] The carcinization of Go programs"
date: 2023-03-24
basename: ../wazero-lightning-2023
slides_link: https://drive.google.com/file/d/1ANxRJPzNeKbLogZz0wCH_o9E9jNLypja/view?usp=sharing
tags:
- wasm
- wazero
- rust
- golang
skip_ads: true
---
<xeblog-talk-warning></xeblog-talk-warning>
<xeblog-video path="talks/2023/wazero-lightning"></xeblog-video>
## Transcript
<XeblogConv standalone name="Cadey" mood="enby">
This is a lightning talk version of [this
post](https://xeiaso.net/blog/carcinization-golang).
</XeblogConv>
<XeblogSlide name="2023/wazero-lightning/01" essential></XeblogSlide>
Computers are complicated. Programming computers is even more
complicated. Sometimes you have a square hole, but the square peg you
need is written in Rust and the rest of your program is written in Go.
Linking these two worlds together like this is a painful process
fraught with peril, fear, terror, utter torment, and lemon-scented
moist towelettes.
<XeblogSlide name="2023/wazero-lightning/02"></XeblogSlide>
Normally if you want to call a Rust function from Go, you need to
expose a C interface in Rust, and then bind to that C interface with
cgo. This does work. It's a thing you can do, but it only works the
most reliably at very small scales. The main problem with doing this
is that it breaks people's semantic expectations of how tools should
work.
<XeblogSlide name="2023/wazero-lightning/03"></XeblogSlide>
When Rust compiles `.so` files that link against your program, they
also link against system libraries. This isn't normally a problem,
except if you have people like me around that use a hipster distro
where everything is different. You also have to maintain a separate
dot s o file for each OS and CPU architecture combination you support.
Your build step is no longer `go build`, it's in Makefile land.
<XeblogSlide name="2023/wazero-lightning/04"></XeblogSlide>
So, imagine a world where you could just put one binary blob into your
git repo, and have that Just Work everywhere. Without making your
build more complicated than `go build`. Without configuration when
people are building the code. Imagine how much easier that would be.
<XeblogSlide name="2023/wazero-lightning/05"></XeblogSlide>
I bet you can guess where I'm going with this, I'm talking about the
carcinization of Go programs via WebAssembly. This is how I snuck Rust
into a Go shop.
<XeblogConv name="Mara" mood="hacker">
The "carcinization" refers to the evolutionary tendency of programs becoming
either crabs or trees when time stretches to infinity. If you can imagine
library use as evolution, then this joke makes more sense.
</XeblogConv>
<XeblogSlide name="2023/wazero-lightning/06"></XeblogSlide>
Why?
<XeblogSlide name="2023/wazero-lightning/07"></XeblogSlide>
Mastodon. Mastodon uses a protocol called ActivityPub to federate
posts. ActivityPub posts are formatted in HTML. Reading Mastodon posts
from the API returns HTML. We want to show these posts off in a Slack
channel, and Slack doesn't support using your own HTML. It has its own
bespoke markup format called Slackdown because of course it does.
<XeblogSlide name="2023/wazero-lightning/08"></XeblogSlide>
There was nothing off of the shelf to handle this in Go. I assume that
this problem is fairly novel. Anything that was close to this just
made atrocities out of the text in ways that I couldn't customize.
<XeblogConv name="Aoi" mood="wut">
I guess some solutions could be out there, but just locked in closed-source
repos.
</XeblogConv>
<XeblogSlide name="2023/wazero-lightning/09"></XeblogSlide>
Then I remembered a Rust library that I use on my blog:
[lol_html](https://docs.rs/lol_html/latest/lol_html/). lol_html is a
streaming HTML parser/rewriter. I use it on my blog for all my HTML
shortcodes and I know it lets you mangle HTML into other formats with
element handlers. Mastodon has a list of HTML tags it will send out
over the API, so I used that to assemble a little test program in
Rust.
<XeblogSlide name="2023/wazero-lightning/10"></XeblogSlide>
When I wrote this program, I made it in a fairly naiive way. I took
HTML over standard input and had it spit out slackdown on standard
output.
This idea just so happens to be one of the core parts of the Unix
philosophy: programs should be filters that take input in one form and
then transform it to output in another form. This made it trivial to
test against arbitrary HTML from the Mastodon API and get things down
to the output format I wanted.
<XeblogSlide name="2023/wazero-lightning/11"></XeblogSlide>
Then comes integration. I already knew I didn't want to deal with the
surreal horror that is dynamically linking Rust code into Go (but I
could figure it out if I needed to), so on a lark I decided to just
try compiling it to WebAssembly with [WASI](https://wasi.dev) and see
if that works.
<XeblogSlide name="2023/wazero-lightning/12"></XeblogSlide>
It did. I then applied some optimizations and got it down to a 200
kilobyte ball of mud that did exactly what I needed. Then all I needed
to do was glue it into the Go program with Wazero.
<XeblogSlide name="2023/wazero-lightning/13"></XeblogSlide>
Wazero made this trivial. I overrode the input and output buffers,
then had it kick things off as needed. At first my code was very lazy
and slow, but it worked. The only build step was `go build`, just like
I wanted. I committed the WASM blob to git and shipped the hack. It
worked perfectly.
<XeblogSlide name="2023/wazero-lightning/14"></XeblogSlide>
The last part was making it faster. With some guidance from #wazero on
Slack, I managed to get each call of this function down to two hundred
microseconds. That's faster than the equivalent code in Go using its
[HTML parsing library](https://pkg.go.dev/golang.org/x/net/html) that
I only found out existed about a week ago.
<XeblogSlide name="2023/wazero-lightning/15"></XeblogSlide>
And now this atrocity is shipped to production and holds together the
Mastodon post announcement service. Most people aren't aware that it's
a thing, and it runs fast enough that nobody really cares that it's a
programming turducken of Go, Rust and WebAssembly. It works perfectly
and it wouldn't be possible without the efforts of the Wazero team.
<XeblogSlide name="2023/wazero-lightning/16" essential></XeblogSlide>
I'm Xe Iaso, and I hoped you enjoyed my talk about programming crimes.
I do developer relations at Tailscale as the Archmage of
Infrastructure. If you have any questions, please don't hesitate to
reach out at [wazero2023@xeserv.us](mailto:wazero2023@xeserv.us). I'm
more than happy to answer them.
Congrats to the Wazero team for the 1.0 release! Be well, all!
## With apologies to
- Renee French
- The Rust community
- The Wazero team
- Hbomberguy
|
