Technomancy

in which more paths are charted towards code independence

2026-04-25T17:08:11Z

As the ninth anniversary approaches of the last project repository I created on Github, it's been so encouraging to see more and more projects migrate away from Microsoft Github. I love seeing the rise of Codeberg, not the least because it's a democratically-run non-profit which isn't subject to the whims of the extractive arm of an unaccountable exploitative mega-corporation.

If you're currently hosting a project on Github and you recognize the harm of Microsoft's monopoly, it's undeniable that Codeberg is your quickest and easiest offramp to an empowering and pro-user place. But at the same time, I don't see Codeberg as the be-all and end-all of Github replacements, for a couple reasons. Most obviously, putting all your eggs in one basket isn't ideal, even if it's a much better basket than the one we used to have. We shouldn't replace Github with one site at all; we need the strength and resilience that only comes with diversity¹.

But beyond that, I'm reminded me of the way that some dominant ideas can become so dominant that it becomes difficult to even imagine alternatives to them. Forgejo (the software behind Codeberg) is unapologetically a Github clone. For most people in software development, Github has been synonymous with development workflows for so long that other ways of thinking have not only languished, but have started to feel nearly inconceivable. Even for people who want very much to get away from Microsoft, the design decisions made for Github follow you around from Gitlab to Bitbucket to Codeberg/Forgejo².

This is a pattern you see over and over with open source alternatives to dominant systems; for a long time GNOME and KDE were just trying really hard to build the exact same thing as Windows, but without Microsoft. Mastodon is trying really hard to be Twitter without Twitter. When a company dominates the market, they capture not only their own users but even the imaginations of people actively resisting them. It usually takes a great deal of time and effort to carve out new patterns of thinking.

If you just want to get off Github, that's fine; you should do that first, and you don't need to tackle the job of learning a new flow right away. But I want to make space for people to think about alternatives. What does a post-github world look like? What other flows could we use to support project development and contribution? When people talk about moving off Github, the first objection is always "but everyone has an account there already!" And this is a fair point; no one wants to create Yet Another Account. I'm only really willing to consider alternatives that don't require creating another account, especially not in a distributed system where each project might be on its own server.

Let's start by going back to the basics. Simon Tatham has written up a guide describing how he uses Git without any forge-like software involved. He keeps his repositories on his own server with nothing but a read-only gitweb rendering their contents as HTML. If you want to contribute a patch, you can publish your own clone on your host of choice and let him know (via email, chat, social media, or whatever) which branch to pull from. This method works best for projects with a single maintainer and a small community. It's a good starting place for most projects. Then when you have more going on, you might want to add a little more structure to help keep track of things.

And then there's projects like Radicle offer a flow that manages to work in a peer-to-peer way with no server involved. Being fully peer-to-peer, your machine gets all the data synced locally, so it works great when offline too, and you don't have to create yet another account on another system. This is a bold vision of how the future of development might look.

But when I was searching for an alternative for Fennel I went looking to the past instead. Back when I was fresh out of university and an enthusiastic Subversion user, I set up an instance of Trac, a web-based version control, ticket, and wiki system. I used it for all my projects, but it was mostly just a personal organization tool as at the time I did not have any projects that got more than one or two patches from other contributors. Then in 2007 I switched from Subversion to Git, and had to drop Trac. A few months later in a moment of weakness, I started putting my projects on Github.

Over the years I lost track of Trac, largely due to the fact that since it was made by fans of Subversion, it took them several years to add support for Git, by which time most people had switched away from it. But Trac offered something really special. The individual features around ticket reporting and wiki pages are impressive, but most of all, I see Trac as the pinnacle of web app design from the golden age of progressive enhancement.

What does that mean exactly? Well, a few years after Trac dropped off my radar, we saw the rise of React, GraphQL, and the Single-Page App (SPA). While there are certain type of sites for which the SPA interaction style makes sense, these technologies saw mass adoption across every single kind of web app, regardless of whether they were a good fit. As the software industry is largely driven by trends and fashion, React's "cool factor" led to a mass amnesia regarding how to make normal, well-designed web apps made of HTML with a sprinkling of scripting on top and everything became one big tangled, brittle mass of Javascript that completely keeled over when scripting wasn't available. But Trac got it right: the server generates HTML (quickly³!) and sends it to you, and in a few cases uses some optional scripting to get things like live previews. You can use it with Netsurf or lynx!

Anyway, when I set up Fennel's Trac, I was mostly impressed with how easy it was to get going, but there were a handful of things that were weirdly off. The most glaring is that even in 2026, the default setup is still to use Subversion instead of Git. Support for Git is now built-in at least, but for some reason you still have to enable it in the config. An easy change, but a strange one.

Secondly, the format used for the wiki pages is ... somewhat dated. It's not bad or anything, but in 2026 using anything other than Markdown just feels pointlessly contrarian⁴. There does exist a plugin that lets you write in Markdown, but it only lets you create Markdown sections of pages that otherwise use the standard formatting. There's no way to use Markdown as the default formatter for wiki pages, issues, and comments. At least, there wasn't until I wrote my own plugin to do it based on the existing plugin.

The third problem is maybe the weirdest: the default setup has no way to create user accounts! It delegates login completely to Apache. In order to add a user, the admin has to SSH into the machine and edit an htpasswd file in a text editor. Needless to say, if you're using this for any kind of community project, (or if you don't use Apache) this is borderline useless. There are some plugins that add workable authentication systems, but all the existing ones either require all users to sign up for (you guessed it) Yet Another Account or require you to tie your logins to a huge company like Google or Github. Neither of those paths were acceptable to me.

When I set this up, I had just recently been working on Fedibot club, an unrelated project. For that, I had implemented a Fediverse-based OAuth system that allows users to specify their server where they already have an account, and use that existing account⁵ to log in. Rather than having a fixed set of servers that you're allowed to authenticate with, it dynamically registers an OAuth client for any new server it's never seen before⁶. So I took this exact flow and ported it to Trac. Despite somehow not really ever having written any Python before in my life, I found it surprisingly easy to implement, using an existing hard-coded OAuth plugin as a guide. It clocks in at 124 lines of code.

When I'm trying out a new system, sometimes I run into some issues and think, "OK, I can put some elbow grease into this and get something working" but I usually expect it'll be a little slipshod and require putting up with things getting janky or ongoing tweaks. I encountered a twofold surprise here: one being simply that the fixes were so easy for me to make as someone who knows nothing about either Trac or Python, and the second being that once those few fixes landed, that pretty much took care of everything, and the remaining things worked basically flawlessly. It's a little disappointing that these things couldn't be addressed upstream in the project, but hey, I'll take the win.

Anyway, whether you end up using Trac or not, I hope you can find something that works for your project. Good luck!

[1] I've seen a lot of my friends stand up their own little Forgejo servers, and I love this. However, I believe there's a big missed opportunity here too. If you don't have an account on Alex's Random Forgejo, you can use OAuth to log in with an account on another server, which is great.

However, users are limited just to servers where the site's admins have manually registered OAuth clients! Alex might allow users to log in using their Github account, and Alex might allow users to log in using their Codeberg account, but it's very unlikely Alex's Random Forgejo will allow users to log in with their account on Mel's Random Forgejo, so there is still some centralizing gravity at play. Fortunately this is very easy to fix! My implementation of OAuth login using dynamic client registration clocks in at around a hundred lines of code. If it weren't for Forgejo being implemented in Golang, I'd be inclined to give a shot at implementing it. Perhaps some enterprising reader would be up for it!

[2] But not Sourcehut! And I do like Sourcehut overall; they have the best CI of any system I've used, and page load times are unparalleled. Its email-based contribution flow can be kind of awkward, but I have to applaud them for trying something different..

[3] I run my Trac instance on my home server, which is a Core 2 Duo Thinkpad from 2010. Trac uses about 160MB of RAM, and loads pages in consistently less than 200 milliseconds; dramatically faster than Github. (Your experience might be slower than this due to more network hops, but outside extreme cases it should still be a good deal better than Github.) When I first got it set up, it felt slow due to an attack of scraper bots from LLM companies, but once I installed Iocaine that took care of it.

[4] Don't get me wrong; I'm not saying Markdown is good. It's a lot like English: I'm not using it because it's good; I'm using it because it's both tolerable and ubiquitous.

[5] As a grudging compromise, I've installed a Github OAuth plugin for users who don't have a Fediverse account, but like ... come on, people.

[6] If you're interested in reading more about how this works, I've got a fairly thorough write-up that should be useful to anyone hoping to build a similar system. It's easier than it sounds! I'd love to see more people using this strategy to break away from corporate monopolies.

in which web pages are served but also fetched

2025-04-08T15:33:17Z

I'm not much of a "web programmer". But when I do make web sites, I mostly like to write programs that spit out HTML files, and then put them on my web server. It's simple, fast, and gets the job done for most of the web sites I've needed. Every so often I come across something that needs a little more functionality than static pages can offer, and in those cases I'd put up a 15-line CGI script to save off some data for later analysis or whatever. Works great, and I highly recommend it when it fits the problem at hand.

But what about sites that are mostly dynamic? Using CGI in that case might not be a great fit. Or what if you need to use a server that doesn't support CGI at all? If you're already using nginx for your site, then OpenResty looks like a compelling choice. But I haven't used nginx in ages. I was curious about more standalone alternatives. That's when I discovered moonmint.

Update: since writing this post, I've switched over to using lua-http instead of moonmint as it is receiving ongoing maintenance. I gave a talk on this at FennelConf 2025.

The moonmint web framework was created by Calvin Rose, who also created the first version of Fennel¹. It is written in Lua, but it's very easy to use from Fennel. Now of course Lua does not have any functionality built-in from which you can make a web server. You can't open a socket without some third-party code. Often people use luasocket for this, but moonmint takes a different approach and uses luv instead, which is a library that provides bindings to libuv, a multi-platform asynchronous I/O framework.

My goal was to build a hyper-personalized search engine that indexed just the links I've posted. This felt like a good fit for moonmint. I found that SQLite contains a very capable full-text search engine called fts5 and decided to use that to store the pages. For the final piece of the puzzle, I used pandoc to convert HTML into plain text suitable for indexing.

Once I had all the pieces put together, the first working version of the site came together very quickly. I fed it a list of URLs from my bookmarks file and was able to index and query them with ease in around 160 lines of code. But the problem with this first version is that I was feeding the URLs directly to pandoc, which was very convenient when the pages loaded, but pandoc would index them regardless of the response code, so I got a lot of 404 pages in the index. So I needed to start making these HTTP requests myself in order to ensure that only successful requests got their pages included.

And this is where things started to get tricky, because making HTTPS requests on the Lua runtime is ... unfortunately a bit of an achilles' heel at this point. There are a lot of options, but none of them are without significant problems.

The moonmint framework actually came with a module for making HTTPS requests, but its relied on a luarocks dependency for its TLS functionality, and that dependency has since introduced incompatible changes. Due to luarocks version philosophy of "just give me whatever version you feel like", it has stopped working with moonmint. These kinds of problems are predictably somewhat common with luarocks, so I try to use other ways to get libraries when possible².
Installing via apt-get instead of luarocks will tend to get you more stable results. There are two Lua HTTP clients available in apt-get on debian stable, which is what I run on my machines: lua-http and lua-curl. Neither of these are packaged for use with Lua 5.4.
I've used luasocket for projects in the past; mostly for IRC stuff. It's pretty good for that! It doesn't do TLS on its own tho. There is a luasec library that offers a similar API but with added TLS capabilities. It also wraps luasocket's HTTP client in an HTTPS variant. However, luasec does not perform any certificate verification by default! This makes its out-of-the-box behavior very unsafe, and I have to recommend against using it for anything. While it is possible to enable certificate verification, leaving such an unsafe default shows very poor decision-making, and I would not trust this library to do other things right if they get such a basic thing wrong.
The latest one I've seen is lua-https, created by the folks behind LÖVE, the 2D game framework. My project has nothing to do with games, but the LÖVE folks tend to do a good job with their APIs, and I figured it would be worth giving it a shot. It's not available in apt-get yet, but unlike most Lua libraries that have C code included, it's pretty easy to build yourself, so I added it as a git submodule and threw a handful of lines into my Makefile to compile it³. The only downside I've seen so far with this library is that it does not support streaming the responses. For my indexer, it would be nice if we could look at the HTTP headers before fetching the whole request, so I could skip known-unusable mime types like videos. However, given the alternatives, this seemed like the best choice for me.

My situation is a bit unusual. I have indexing runs which happen in batch jobs completely disconnected from the web app. They populate the SQLite database, which is only ever queried from the web app. The web app doesn't make any of its own HTTPS requests. If it did, using lua-https would mean that those requests would block until they completed, during which the server could not process any other requests. The site I'm building is a hyper-personalized search engine. You're welcome to use it too, of course, but it's really most useful to me. It has a target audience of one. So I'm not at all concerned about problems like blocking the main thread. If I were, I would need to replace it with a non-blocking equivalent, ideally tied into the event loop of luv.⁴

My search engine is also unusual because its indexer does not really "crawl" at all; that is, it fetches a list of pages but does not recursively branch out to the pages that those pages link to. That means the index remains pretty small. For a general-purpose search engine, this would render it more or less useless, but for a hyper-personalized search engine, it's honestly not that bad, and it makes it much simpler to code. I've gone into more detail on the site about why I built my own search engine in this unusual way so I won't get too much more into that here, but all my URLs so far come either from my bookmarks file, or from links I have posted on my social media accounts. My next planned step is to start indexing links that are posted by accounts that I follow.

Requests right now are very fast; between 20 and 80 milliseconds⁵ depending on how many results there are. I would expect that hundreds of concurrent users would be supported easily without much noticeable slowdown. But if I was interested in scaling this up beyond that, I would put several different lua processes behind a load balancer. I put the server behind Caddy in order to do the TLS termination and to let me have multiple sites on the same server, but I only run one application server for the search engine. If you needed to handle more traffic, balancing across many different server processes across a few different ports is easy to do with a small tweak to the Caddy config.

Anyway, if you are interested in how to build a simple web app in Fennel, take a look at the code! At just under 350 lines, it's pretty straightforward and readable. I'm not sure I would necessarily recommend Moonmint over Openresty for web applications in general, but if you're doing something with simple I/O requirements and want a solution with fewer moving parts, give it a try.

[1] When I first looked at moonmint, I had an eerie sense of deja-vu. It was a project written in Lua by Calvin Rose in 2016 which had not seen any further development since 2016. Those exact same circumstances could also describe Fennel when I first found it! It's about 1500 lines of code. My own fork of moonmint removes the non-functioning OpenSSL stuff.

[2] There's a newer project called deps.fnl which tries to work around the problems inherent in the luarocks model. I did not try it for this because I needed a few dependencies which it couldn't handle like pandoc and a lua library for robots.txt that was kept in svn. (yes, really!) But if you need a library that's hard to use without luarocks but want an actual reliable build, you should give it a look.

[3] The advantage of this approach is it makes it pretty easy to build the dependency with static linking. When you're writing a CLI tool to distribute, this can be really handy. I used this technique for another Fennel project of mine in the past, but it wasn't relevant for a web application.

[4] This is one place where Openresty might be a better choice; it's already got non-blocking functionality for HTTPS requests and database queries and whatever you'd want to do. And the non-blocking calls can be nicely abstracted away using coroutines so that the code reads linearly and doesn't fall prey to the callback-soup style that is common among async programs.

[5] These response times are coming from my home server, which is a 4-core Thinkpad from 2010 on my home DSL. This machine also hosts my two social media servers. The search engine process consumes around 20 MB of resident RAM.

in which things once suspended are resumed

2024-12-31T19:28:41Z

One common question I get about coding in Fennel is about how to effectively use coroutines. Fennel runs on the Lua runtime, and Lua is famous for having very few features, but having carefully selected just the right features to include, and coroutines are one of those features that give a remarkable bang for their buck. But because most languages don't have them, they're often seen as an advanced or confusing feature, but they're not difficult, just different. Once you get a grasp on them, they provide a great deal of flexibility and in many cases mitigate the problem of not having OS threads on the Lua runtime.

What even is a coroutine?

In order to create a coroutine, you need a function to put inside it. Here's an example in Fennel¹:

(local c (coroutine.create (fn [] 5)))
c ; -> #<thread: 0x56546f8fba88>

Lua calls them "threads" which is confusing, but they're not OS threads. Coroutines are a form of cooperative multitasking while OS threads are a form of preemptive multitasking. That means that there's a scheduler which decides when to run or pause an OS thread. With coroutines, it's up to you to schedule and pause them. Despite some similarity to OS threads, you really don't use them in the same way, so I'd hesitate to lean on that analogy too hard.

In order to run a coroutine, you use coroutine.resume:

(coroutine.resume c) ; -> true 5

In the trivial case, it behaves like a function. However, if it were a function, we could call it again, but:

(coroutine.resume c) ; -> false "cannot resume dead coroutine"

That's because each coroutine represents a single run of a function. So far it just sounds worse; a function you can only call once. Great. In order for it to be interesting, we need a function that yields:

(fn f []
  (coroutine.yield 1)
  (coroutine.yield 2)
  :done)

(local c (coroutine.create f))

(coroutine.resume c) ; -> true 1
(coroutine.resume c) ; -> true 2
(coroutine.resume c) ; -> true "done"
(coroutine.resume c) ; -> false "cannot resume dead coroutine"

Yielding is like returning, but resuming starts you back where you were. That's it! That's the whole thing. You now understand coroutines; thanks for reading!

Just kidding; there is one more thing. You can pass values into the coroutine when resuming, and they will be returned by the call to coroutine.yield. To show this, let's make a recursive function that uses that return value:

(fn f [x] (f (+ x (coroutine.yield x))))
(local c (coroutine.create f))

(coroutine.resume c 1) ; -> true    1
(coroutine.resume c 1) ; -> true    2
(coroutine.resume c 6) ; -> true    8

You'll notice the function never actually needs to return. It can loop forever and that's fine as long as it yields.

OK, that's it now for real this time; that's the whole thing. Deceptively simple, but what can you do with it?

Workers working together

Coroutines allow you to write non-blocking code that still looks linear. Without coroutines, writing non-blocking code means leaning on callbacks. Here's a simplified example from a game I made with Matthew Lyon called Tower Institute of Linguistics. You can see the whole example here, but let's start by looking at one function:

(fn move [self target]
  (if (= self.y target.y)
      (set self.x (+ self.x (if (< self.x target.x) 1 -1)))
      (= :stairs (state.map:tile-type self.x self.y))
      (set self.y (+ self.y (if (< self.y target.y) 1 -1)))
      (move self (state.map:find-on-level self.x self.y :stairs)))
  (coroutine.yield)
  (when (not (self:at? target))
    (move self target)))

Workers move with this move function which A) figures out which direction to go, B) takes a single step in that direction, C) yields, and then D) recurses if it's not done. This looks like a pretty normal recursive loop; the only difference is it's got a yield in the middle of it.

So move on its own could be done not too badly with callbacks. Our approach really shines as part of a more complex flow. For context, this is from a game you give orders that get picked up by the workers in the game. Orders require two workers to work together to accomplish them. Without blocking the main loop, workers need to decide what to do next and coordinate on their work. Let's look down at the start function that kicks off a worker's logic:

(fn start [self]
  ;; first check the queue of incoming orders to take
  (case (table.remove state.order-queue 1)
    order (take-order self order))
  ;; next see if any other workers need help
  (case (table.remove state.help-queue 1)
    other-worker (give-help self other-worker))
  ;; if you're too frustrated, leave
  (when (< state.threshold self.frustration)
    (self:leave ["no one understands me.\nI'm tired of this."]))
  (coroutine.yield)
  (start self))

What does it mean to take an order?

(fn take-order [self order]
  ;; add yourself to the help queue and set yourself to waiting
  (table.insert state.help-queue self)
  (set (self.order self.waiting?) (values order true))
  (move self order) ; remember, move yields every step of the way
  (while self.waiting?
    (set self.frustration (+ self.frustration 1))
    (coroutine.yield))
  ;; once we have someone helping us, we can build
  (build-order self order false))

(fn build-order [self order helper?]
  (each [tx ty (order.tiles helper?)]
    (move self {:x tx :y ty})
    (self:build (. order.blueprint ty tx))))

And how about helping?

(fn give-help [self other-worker]
  ;; go to the worker you're helping (yielding the whole way)
  (move self other-worker)
  ;; unblock them
  (set other-worker.waiting? false)
  ;; get to building!
  (build-order self other-worker.order true))

There's a little hand-waving here, like what self.build does. But the point is the overall flow is very clear. Now try to imagine what this would look like using callbacks. You would have to turn the whole thing inside out. No recursion, no loops allowed. It would not read smoothly because you'd constantly be distracted by trivialities.

The Importance of Being Stackful

Other languages have similar constructs: Python and JavaScript have "generator functions" which can also yield. However, generator functions can only yield directly from their body. Other normal functions that they call normally can't yield². They can call other generator functions to let them do a nested yield.

Generators are a bit of a lower-level construct from coroutines. However, they can serve as building blocks to implement "stackless coroutines". Using macro-like tricks, you can implement stackless coroutines by splitting functions into generators at their yield boundaries. The advantage of this approach is that it can be done at compile-time without needing any support from the underlying virtual machine.

But going stackless has one critical limitation: you can only yield from a generator function. This would be fine for the worker example above; we wrote all code that yielded, so we could change it to use generators. If we wanted to yield from existing code that was not yield-aware, on the other hand...

You may now resume REPLing

Imagine you are creating a program with a GUI toolkit and the GUI library provides you with an event loop. You can't block the event loop, but you want to integrate a REPL so the user can run code from within the program. But when you call fennel.repl it makes a blocking call to read from standard in. That's no good—you're writing a GUI program! You can't do a blocking read, and you're not using console input anyway.

You can indicate for Fennel's repl to get its input from another source without too much trouble, since it accepts an options table. Let's put some canned input lines into a table and give it a function that pops those lines. Once the table is empty, we'll return nil which will get treated like an EOF, ending the REPL session.

(local lines ["(+ 54 " " 50 " ")"])

(fn next-line [] (table.remove lines 1))

(fennel.repl {:readChunk next-line}) ; -> 104

But how would we wire this into a GUI? Here's an example that uses LÖVE, a game framework. It starts out by defining one table each for input and output, then declaring a drawing function, which you can just skim because isn't that important for our purposes.

(local [lines input] [[] []])

(fn love.draw []
  (let [(w h) (love.window.getMode)
        fh (+ (: (love.graphics.getFont) :getHeight) 12)]
    (each [i line (ipairs lines)]
      (love.graphics.print line 2 (* i (+ fh 2))))
    (love.graphics.line 0 (- h fh 4) w (- h fh 4)) ; draw input
    (love.graphics.print (table.concat input) 9 (- h fh))))

Output puts things into the lines table; input gets handled by tossing text into input, plus we've got a little helper function to clear out input once we're ready to use it:

(fn out [xs] (icollect [_ x (ipairs xs) :into lines] x))

(fn love.textinput [text] (table.insert input text))

(fn pop-input []
  (let [text (table.concat input)]
    (while (table.remove input) nil) ; clear input
    (.. text "\n")))

Finally we wire it all together with a coroutine that runs our repl function; we override the output function to put things in the table, but instead of creating an input function to read from our GUI elements, we just pass in coroutine.resume directly! Finally the love.keypressed handler responds to the user pressing enter by resuming the repl coroutine with the string from the input table! The repl coroutine runs until it needs input, at which point it yields until the user indicates the input is ready by pressing enter, resuming the coroutine with the input.

(local options {:onValues out :readChunk coroutine.yield})
(local repl (coroutine.create (partial fennel.repl options)))

(fn love.keypressed [key]
  (case key
    :return (coroutine.resume repl (pop-input))
    :backspace (table.remove input)))

So what makes this different from the previous example? Well, in some senses it's simpler; it's just one loop instead of a whole series of actions. But the important thing is: fennel.repl doesn't know it's running in a coroutine! All it has to do is accept whatever function we want as the "read input" function³ replacing reading from standard in. For the rest of the repl code, it's completely irrelevant that it's going to yield waiting for input.

This technique is not possible with stackless coroutines. So it wouldn't work in Python or JavaScript, but it works great in Lua and Fennel! You can turn any code into non-blocking code regardless of whether it was originally written with that in mind or not. It's a really powerful technique⁴ whenever you need some flow control that goes a little outside the usual. Give it a try some time!

[1] I'm using Fennel for my examples because... well, it's my favorite programming language! If you prefer Lua, you can take any little snippet of Fennel code and put it into See Fennel to find out what the equivalent Lua looks like. All the code in this post is very straightforward to translate; mostly you just move the parens. Remember that Lua and Fennel have tail-call optimization, which our code here will use heavily.

[2] If you've heard of "function coloring"; this is the same problem. See the article What Color is your Function?.

[3] For this to work, it's important to note that coroutine.yield is not special syntax. It's a first-class function that can be passed around as a function argument, placed in a table, etc. That's why we can pass it right in with the options table; no wrapper needed.

[4] If you've ever used the site itch.io, you've benefited from this! Because itch.io is written in MoonScript, which compiles to Lua. They have a great write-up of how their I/O uses coroutines to make linear-looking code run in a non-blocking way, so check that out.

in which social media can be put in your own hands

2024-09-13T01:48:12Z

I've been on Mastodon since early 2017 and have really enjoyed it. It's been great to see the Fediverse grow as a user-owned network that can function without a corporate overlord calling the shots, exploiting the user base, and ultimately squeezing it to death for monetization. There are plenty of problems that remain, but one of the biggest ones is that new users have to find an instance to sign up on, and this can be tricky. Installing and administering a Mastodon instance is a lot of work. The liberatory function of the network is only as good as peoples' ability to make use of it, which requires running servers. If running a server is hard, fewer people are going to do it, and the power is going to be concentrated in the hands of a technical elite. A healthy network must make it easy to avoid this kind of centralization; to do that we have to look beyond Mastodon.

In 2019 I ran my own fediverse server out of my home on Pleroma for about a year; eventually shutting it down for a few different reasons¹. Then I started hearing more and more about this new GotoSocial server whose goal was to make it easy to run your own instance, and, well, I liked what I saw! I've been running my own instance since August of 2022 and have made some documentation contributions to the project as well as funding it on OpenCollective.

So what's it like to set up your own GotoSocial server? Well, I can walk you thru what I used for my setup. Yours might be different; that's OK! I made these up-front choices to simplify the operational overhead because I didn't want this to be a hassle and I don't need the extra engineering that comes from trying for nine nines of uptime:

Running from my home DSL: I have an account on a server in a data center that this blog uses, but it's just for static files, email, and simple CGI scripts. It's not a full VPS because I don't want to be on the hook for all the updates and other sysadmin chores, but the downside is that it can't run programs like GotoSocial. But I have an Internet connection at home that I can use. The site will go down when the power or Internet at my home cuts out, but I don't mind. It's fine.
Using dynamic DNS on a subdomain: Similarly to the above, I've had hagelb.org registered forever, so it's easy to add hey.hagelb.org as a subdomain without doing additional registration. In fact this is a CNAME for technomancy.mooo.com which is set up to accept dynamic DNS updates on freedns.afraid.org which I like a lot. It does mean that when my ISP changes my IP address, there is a window waiting for the old cached DNS records to expire, and message delivery will be delayed. It's fine.
Running on an old Thinkpad: Yep, I put it on an X260 that I got back in 2016. It uses about 300MB of memory, and the CPU usage (for my single-user instance with ~750 followers and following ~100) has so far been imperceptible. Sure that machine is used for other stuff, but it's so efficient that you can't even tell. It's fine.
No containers: You can run GotoSocial in Docker. Lots of people do this. However, from watching the GotoSocial help channel, I've observed that people who do this tend to end up running into weird problems that don't happen if you just ... run the program on the host directly, like people have been doing for thousands of years. I can understand the appeal of Docker if you have a very complex, error-prone installation process like Mastodon, but for GotoSocial it's just one standalone binary and zero dependencies. None of the problems Docker is meant to solve are present here, so there's no need to complicate things. I started running my server inside a tmux session assuming that I would switch it over to being a proper systemd-managed service once I got to the point where the limitations of tmux started causing problems. That ... never actually happened. It's fine.
SQLite for the database: For some reason the default example config file in GotoSocial is set to use PostgreSQL. I like PostgreSQL for a lot of things. We use it at work and it's great. But it's absolutely overkill for this; none of those fancy features are going to be needed so just stick with SQLite which doesn't require a separate install and server process to be running. It's fine.
No proxy: GotoSocial can handle serving, registering, and renewing TLS certificates itself, so it can listen directly on port 443. If your machine is going to host multiple web sites, things need to get a little more complicated. Instead of putting your GotoSocial server directly on port 443, you put a different server like nginx or Caddy there to handle TLS termination, and GotoSocial listens on 8080 or something, with nginx dispatching requests to different downstream ports based on the contents of the Host header. I'm not doing this, because I don't have any other sites here. If I wanted to add another site to this machine, it'd be a pain, but for now it's a lot simpler. Update: I was wrong; it's actually incredibly easy to put a proxy up after the fact. Install Caddy and put in 2 lines of configuration about which domain gets proxied to which port, and everything just works. It's fine.
No privileged ports: Unix was designed as a multi-user operating system. By default it will not allow normal users to listen on port numbers below 1024, such as the HTTPS port 443. Nowadays this is pretty silly; rather than multiple users per computer we have multiple computers per user. I don't want my program to have to be root just to respond to HTTPS requests, so I ran echo 'net.ipv4.ip_unprivileged_port_start=0' > /etc/sysctl.d/50-unprivileged-ports.conf && sysctl --system and now any user can do it. Someone will undoubtedly tell you that's unsafe. It's fine.
~~Makefile~~ apt for updates: Until about six months ago, every time a new GotoSocial version came out I'd back up my stuff, curl the tarball², extract it, shuffle some files around, and press Ctrl-C ↑ Return to restart the server. Then I wrote makefile to handle the updates and backups for me. Longer-term I want to get an unofficial Debian package for it that will handle service restarts for me, but that's fairly complicated, and I'm not there yet. It's fine. I have created unofficial debian packaging to make it easier to install on apt-based systems.
Pinafore and Tusky for clients: Unlike Mastodon and Pleroma, the GotoSocial team is focused on creating a backend and not a client. When you want to sign in, you'll need to use a 3rd-party client for that. On my laptop I use Pinafore, which is the best SPA-type web app I've ever used by a pretty large margin. It ran equally smoothly on my newer hardware as on my old Core 2 Duo laptop from 2008 and has excellent keyboard controls. On mobile I use Tusky, which by a bizarre coincidence is also the best app I've ever used on its platform (Android) by a significant margin. You can use GotoSocial with other clients as well; I've tried Phanpy and I hear there are good iOS clients too. So yeah, there's no way to use it without third-party apps³, but the apps are brilliant, so it's fine.
Update: Imported blocklist: I forgot to mention this when I first published this, but the Fediverse is an open network, and there are plenty of bad actors on it. You really don't want to run an instance without having a robust blocklist in place. Before my current instance, I used icosahedron.website as my home, and I found that the admins there had done an excellent job at keeping the bad guys out. So I took their list as my starting point. There's no way to automate subscriptions to blocklists yet, but it's on the roadmap. It's fine.

One huge selling point for GotoSocial over Mastodon for me is that posts on GotoSocial have web permalinks! You can just link to a post and people can open it in their web browser or curl or whatever, provided it is a public post. (The default post visibility is Unlisted, so if you want things to be visible in the browser, you might consider changing this to Public.) In Mastodon, when you send someone a link to a post, they can't read it without loading the entire Mastodon frontend app into their browser, which will is a monster React codebase that will break if they have scripting disabled and also can take up to 30 seconds on older hardware.

There's a search feature built-in with indexing, but it is limited to posts you've written and their replies. But it turns out that after running my server for a little over 2 years, the entire DB has only grown to 2.1GB. When it's that small, you don't need an index; the entire statuses table can be scanned in half a second:

  $ sqlite3 sqlite.db "select uri, content from statuses where content like '%sqlite%'"

So is it all sunshine and roses? Well... yeah, kinda! When I first set it up I was expecting it to at least require a bit of regular admin work, but in fact the only thing I have to do is restart it for updates. Early on there were more missing features⁴, but nowadays overall it's quite competitive and is even landing new stuff no one else has like a fully sandboxed, wasm-based thumbnail generation process and interaction policies (aka Reply Guy Defense System) that allow you to describe in more detail who is allowed to see, reply, or boost a post.

I know running your own server is not for everyone^{5 6}. We still have work to do to make this kind of thing even more accessible and to tear down the gatekeeping that prevents "non-nerds" from learning how to do this stuff. But we've come a long way, and I'm excited to see what the future brings as more people are able to participate in user-owned social media.

Update January 2025: I have a second GotoSocial server running on the same host. It's called Yotosocial and it only lets you post "yo".

[1] This was partly because I was running the server on a Raspberry Pi, at first on an SD card, and then later on an external USB hard drive. It turns out PostgreSQL gets really mad if you use it heavily from an SD card; disk errors caused frequent DB segfaults. But also the direction the project took caused me to rethink my involvement in it; some unpleasant bigots joined the team, and I wanted to distance myself from that.

[2] Unfortunately, the release tarball is a tarbomb. Watch out! But the linked makefile does handle this problem for you.

[3] There is a GotoSocial web UI for settings, but you only use it to configure your account and instance, it's not something you open on a regular basis.

[4] When I started, it didn't have support for mp4 media, which is especially a pain because Mastodon doesn't support gifs, so when people upload gifs it converts them to mp4, so I was missing a lot of those. Hashtags, polls, mutes, and account migration have all landed in the past 18 months. One thing that's still missing is the edit button; for the time being you have to delete and redraft instead, but it's slated for the next release. Domain blocks are also currently all-or-nothing, so you can't set it to drop activity from an instance for all users that you don't currently follow, at least not yet.

[5] If you want to pay someone else to run a server for you, hosted Mastodon has been around for a while, but I've recently discovered that hosted GotoSocial is now offered as well, currently for US$3.75 a month. This is an observation, not an endorsement, as I haven't used the service. This post discusses the process of setting up a hosted server.

[6] In this post I've focused on the technical hurdles, but the social aspect of moderating an instance brings a completely different set of challenges when you have other people on your server. Lots of people just don't; they set it up just for themselves, and that's fine too.

in which an escape route is discovered

2023-12-31T22:07:31Z

I was initially something of a skeptic when I first read about the Language Server Protocol. As a huge fan of REPL-based development, I thought it was a step back from the nREPL protocol, which had several years head start on the "editor-agnostic, language-agnostic tooling" approach¹. But after using it a while on Clojure and Fennel, I've come around; the features are compelling and worth using.

The biggest difference between nREPL and LSP is their approach of what they do with the code in your project. nREPL is built around the assumption that the main thing you want to do is run your code; interact with it back and forth to see how it works. LSP is built around static analysis: "look but don't touch". For the most part they're complementary; LSP is never going to be able to show you something like test results or tracing, and because of greater overall investment of effort, LSP tends to do a better job of surfacing errors directly in your editor, provided they're compile-time errors.

The difference between running code and just looking at it is pretty big, but some languages blur the distinction. Languages with metaprogramming tend to put up fundamental hurdles to static analysis, because once you allow code to be rewritten by macros, how can you even know what it is you should be analyzing without actually running the macro? And once you've run the macro, you're not doing static analysis any more—a macro could run any arbitrary code, including stealing your SSH keys or wiping your disk.

Here's an example: this code uses a macro which introduces a new local, but without running the macro, the Clojure LSP server can't actually know that, and mistakenly flags it as an error:

(deftest test-all
  (utils/with-system [{:keys [database] &as system} {:start true}]
    (let [user (db/insert-user database (test-data/generate-user))
          request {:user-id (:id user) :endpoint :get-user}]
      (is (= (:username user)
             (:username (grpc/handle system request)))))))

This is pretty annoying when you're coding. There are workarounds to some cases; you can spoon-feed the LSP server to just treat a macro as if for analysis purposes it works just like a built-in; in this case pretending to be let would fix it. But it can't work gracefully out of the box because of this inherent incompatibility.

However, most macros don't actually do anything other than pure transformation of their arguments. It's almost always safe to run them during static analysis, except you don't know for sure². That's why Fennel's macros can't access the outside world³. By introducing this one limitation, we supercharged our static analysis capabilities to where they're outperforming much more mature and better-funded languages like Clojure.

Unfortunately, up until the 1.4.0 release, there was an issue with the macro sandbox, in particular around its interaction with the metadata system. Fennel allows you to attach metadata to functions and macros, so you can look up things like argument lists and docstrings in the REPL or other dynamic tooling. These get stored in a metadata table in the compiler.

Now of course, the metadata itself is quite safe; it's just strings and lists of symbol names. There's nothing wrong with exposing this inside the compiler sandbox. You want to allow macro definitions to set metadata of macros. The problem is that the data is keyed not on the name of the function⁴ but on the function itself! So any function which had already been compiled could be run from the macro sandbox, making it easy to escape it.

Anyway, once this was discovered, we quickly released Fennel 1.4.0 with a fix that exposes a write-only proxy table for metadata to the sandbox. Thanks to XeroOl, author of the wonderful fennel-ls language server for discovering this vulnerability and disclosing it.

[1] Actually I still do think that as a protocol it's a big step back from nREPL, but it's not a hill I'm going to die on, because LSP has tons of momentum coming from very deep pockets, and nREPL is almost unheard of outside lisp communities. It's the tale as old as time of the technically superior alternative losing out to the clumsy behemoth. But as a user, you mostly don't care how the protocol is implemented, and the nastiest of the design shortcomings in the protocol seem to have been addressed.

[2] I should note that while I fully agree with the approach of Clojure's LSP server to limit itself to static analysis for safety reasons, this is not a universally-held position. Other language servers don't make the same tradeoffs around safety. In particular I was surprised to learn that rust-analyzer actually runs all macros and specifically documents that it is not safe to run on untrusted codebases. Unfortunately it's unclear whether most of its users are aware of this fact or not. Stay safe out there, folks!

[3] That's a slight oversimplification; it's possible to disable the macro sandbox when it's really necessary, but this is something you wouldn't do unless you knew ahead of time it was safe, and it's certainly not something that the language server would do for you! That said, we have plans to allow you to be more granular and grant specific macros limited access to filesystem capabilities.

[4] Functions in Fennel and Lua don't really have names in a conventional sense, other than the name of the local or table field they're stored in. Since there can be a multitude of these, none of them can be considered canonical or inherent. In a subtle but very real sense, every Lua function is an anonymous function. Stack traces in Lua don't tell you the name of the function as it was defined, but the name used to reference the function when it was called.

in which legibility comes at a price

2022-11-22 19:16:41

It's been a weird few weeks for decentralized social media. The collapse of Twitter at the hands of an incompetent billionaire has led to a massive wave of users on the Fediverse¹, from under a million to over seven million now. And at the time of this writing Twitter hasn't even finished collapsing yet; the site is still hobbling along, albeit with gaping holes. Though by the time I've finished writing this post, who knows. A lot can happen.

As you might expect, the huge influx has caused some technical problems to the Fediverse. The mega-instances like mastodon.social saw slow page load times and hours of lag between something posted and it propagating out to other servers. More reasonably-sized servers have still seen some lesser propagation lag, but the main effect of this influx has been nontechnical. New users have had a hard time finding well-run servers that are accepting new members, and there has been a lot of complaining that the system is hard to understand or navigate due to its distributed nature.

whenever twitter makes a bunch of its users angry we get a new wave of activity on the [Fediverse] and all the new mastodon users engage in the primary hobby of a new mastodon user, which is listing all of the reasons why mastodon doesn't work. it's real fun

- @rootsworks@mastodon.social

Some of this is simply a necessary difference between a single unified corporate system to a distributed network. In the 1990s we saw a similar type of confusion as people left silos like America OnLine (AOL) and CompuServ to explore the early days of the World Wide Web. It wasn't as polished, and there was no central view of the entire network. You had to do more exploring to find independent communities that ran their own sites. People who remember making this shift know now that the effort was worth it, but back then you couldn't fault people for wondering why things couldn't just be easy like it used to be.

But I believe there's more to the situation with the Fediverse today than simply the differences that arise out of technical necessity.

I recently finished reading the book Seeing Like a State by James Scott. There's a lot to unpack in this book², but I want to talk about the way it defines the term legibility. Legibility describes the degree to which a place is understandable and navigable to an outsider.

My favorite example of legibility is street addresses. When I lived in Thailand I was initially very surprised to learn that even long-time residents had a very rough grasp of the names of most of the streets in the city. People navigated by landmarks instead. As a newcomer to the city it was very disorienting; I had to set aside my ideas I brought in about how to navigate and learn to read the city like a local. In the end after living there two years I knew the names of four or five of the major streets, but when someone wanted to know where my house was, I didn't give them a street address; I told them it was a block east of the bus terminal.

This attitude towards navigating looks different depending on whether you're an outsider or a resident. To a resident it's no big deal, but to someone who's new to the city it can be frustrating and bewildering. In some cases (for instance, cities that rely on tourism) there are incentives for the locals to increase legibility, but other factors can incentivize people to reduce their legibility.

Legibility is particularly interesting when it lies at the crux of a power imbalance, and the book describes many such situations. As pre-modern states in Europe consolidated their power, they looked for ways to collect taxes and conscript soldiers more effectively. Customary naming made this very difficult.

Only wealthy aristocrats tended to have fixed surnames…Imagine the dilemma of a tithe or capitation-tax collector [in England] faced with a male population, 90% of whom bore just six Christian names (John, William, Thomas, Robert, Richard, and Henry).

Seeing Like a State by James Scott

In order to consolidate their power over their citizens, states rolled out policies to increase the legibility of the population they ruled over; they started to require that everyone take a surname in order to uniquely identify them to the state in censuses and tax documents. As you can imagine, this didn't go over well; the historical record documents intense resistance, in some cases even open rebellion. The people themselves had no need for surnames; they got along fine with a given name supplemented when necessary with contextual specifiers. "John" might be "John the Baker" in some contexts and "Short John" in others and "John Underhill" to people in the next town. This system worked fine for everyone involved, except those wishing to exert their authority from the outside.³

If this sounds familiar to you, it might be because you were paying attention to social media controversies of the last decade! Prominent advertising companies like Facebook and Google attempted to roll out tremendously unpopular policies requiring users to identify themselves using their legally-documented name⁴. Though these policies were always accompanied by some flimsy justification about "user safety", the real goal was to make their users more legible, primarily for advertising purposes, but also to law enforcement and other authorities.

Well, it turns out just like villagers in premodern France, lots of users don't want to make themselves more legible! Even setting aside the impossibility of a company like Facebook accurately identifying the difference between a pseudonym and a legal name, there are a lot of great reasons to not want to be easy to find. Avoiding targeted advertising is just the tip of the iceberg; ask any member of a marginalized group and they'll tell you that being easy to find can have a high cost when it exposes you to targeted abuse.

Twitter features full-text search of everything posted. This is fantastic for legibility; if you're curious about a topic, you can always see who's talking about it. One of the ways this gets used on Twitter is to pick fights. Trolls find a hot topic and barge uninvited into conversations to yell at people who disagree with their position. On the Fediverse, search is opt-in instead of opt-out. Normal text doesn't get indexed, but hashtags do. If you want your post to become searchable under certain terms, a # is all it takes, but the randos won't show up unless you go out of your way to invite them.

In some ways, people today coming to the Fediverse from corporate social media remind me of how I felt when I first moved to Thailand—lost and bewildered by the lack of street signs. They've spent a lot of time on the highways of platforms that prioritize "engagement" at all cost, and it's taking a while for it to sink in that not all the Internet works that way. They've never considered that illegibility can be a defense mechanism..

Become illegible.⁵

[1] The Fediverse is a network of interconnected social media servers that exchange posts. Most of its usage is based on a similar model to Twitter, but there are some systems on it that focus on photos or videos. Because the most common Fediverse server is called Mastodon, many people use the word Mastodon to talk about the network, similar to how in the 1990s people said "Internet Explorer" when they meant the web.

[2] While the book is insightful, it is pretty long, and the two case studies in the middle drag on a bit. I strongly recommend reading the first three chapters and the last two chapters, but you can still get 90% of the insight by skipping the two very detailed case studies in the middle.

[3] The scary thing about this is how absolute the normalization of surnames became over time despite the intense resistance. In the end, people accepted the greater control, and even tho surnames were rare in many societies only a few hundred years ago, nowadays they are thought of by most people as "natural".

[4] These policies often misleadingly referred to the legal name as the "real name", as if legal process had some magical access to reality itself which could not be achieved using the customary naming patterns people have been using for millennia.

[5] I have to admit to stretching this a bit for a punchy closing line, but of course it's not as simple as "legibility bad"; obviously there are many cases in which you want legibility. I'm trying to convey the fact that legibility does have downsides, and if you've never even considered this, then it's impossible for you to have a productive conversation about social software in today's context.

in which the word column has several meanings, none of them architectural

2022-07-25T17:44:10Z

Sometimes adding what appears to be the tiniest little feature to a program can lead you down a rabbit hole of edge cases and bugs. This time around it was an innocent-looking question about column numbers in the Fennel compiler error messages. The Fennel compiler has for a while now had a feature where errors show the line containing the error and pinpoint exactly where in the line the error occurred.

Compile error in test/bad/expected-even-bindings.fnl:1
  expected even number of name/value bindings

(let [x 1 y] x)
     ^^^^^^^
* Try finding where the identifier or value is missing.

You can see here that the ^^^^^ indicator highlights the entire binding vector which caused the error. We didn't invent this feature in Fennel; we surveyed existing compilers and stole ideas from the ones that people really seemed to love. In particular, Elm and Rust kept coming up as role models whose error reporting was worth emulating, and both had this error pinpointing as one of the things that people reported liking.

Now the bar is pretty low in general when it comes to compiler error messages; many compilers make only a token effort at providing error messages that are clear and understandable. On the bright side, this means you can put a small amount of effort into this and quickly place in a surprisingly high percentile of quality! Adding this error pinpointing to Fennel only took a couple weekends, and the payoff has been fantastic.

However, it's not all smooth sailing. In order to display this information the Fennel compiler originally stored byte offsets in the AST and read the source file up to the line in question. However, using byte offsets for this led to some issues. Take a look at this code:

(let [无为] nil)

In this case, we want to highlight the [无为] section of the code because it tries to introduce a new local without giving it a value. But its byte offsets are 5-13. If we use those byte offsets to draw the pinpoint indicator, we get this:

(let [无为] nil)
     ^^^^^^^^

Just because the identifier 无为 is 6 bytes doesn't mean it is 6 characters wide; in fact it is only two. Now we don't have a foolproof solution for this problem in Fennel, but we can manage OK. Fennel's compiler runs on the Lua VM, and in versions of Lua from 5.3 onward, we have the utf8.len function which can correctly identify "无为" as a string containing two characters. We have to fall back to the byte offsets when running on older versions of Lua, (unless the utf8 library has been backported) but when we can do better, we try our best.

Unfortunately that's not the only wrinkle we have to deal with. What about this code?

(print "วัด" unknown-var)
             ^^^^^^^^^^^

This code doesn't even do anything "fancy" like unicode identifiers, but it still throws off the error pinpoint indicator. That's because the word "วัด" consists of three characters, ว, ั, and ด. The second is attached to the first, because Thai is an abugida, meaning the vowels are often attached above or below their consonants rather than being written separately. So วัด is nine bytes and three characters, but it only takes up two columns, throwing off our indicators. At this point we have run up against an inherent limitation in this approach. As long as we assume our output is displayed on a terminal, there is no way to determine the width of the code that we're trying to draw attention to!

Even if we can determine the "width" of a character in columns, using that to produce an underline relies on the assumption that every column of characters is an equal width in pixels. The idea that a monospace font can reliably be monospace across all characters simply isn't true. In the best case, a larger character will be displayed in some width which is an integer multiple of the "normal" characters, but there's no guarantee that this will happen, and different fonts will make different decisions about how to render a given character. The only way to know how wide something will be is to attempt to render it in a specific font and measure it.

And Fennel is not the only compiler to run into this bug. Rust and Elm both are easily confused by strings and identifiers which use characters outside the ASCII range, as are OCaml, Clang, and every other compiler we could find which tries to help identify errors this way.

In my opinion the root cause of this bug is that when a white american developer sees a terminal they immediately interpret it as this idealized cartesian plane where they can lay out any writing they want in neatly-spaced characters that behave in "predictable" ways (in other words, behave exactly like English). But the reality is a lot more complex than that! You can't assume that everything people write will fit into your assumptions and your neat abstract boxes. Programming, of course, is forgetting, but we need to at least try to be aware of the costs of the abstractions we choose and consider who it is that ends up being forgotten.

We did find a way¹ around the problem—putting the indicator inline with the code itself allows you to avoid having to know how wide it is. So the next version of Fennel will highlight errors using terminal escape codes, which will even allow the functionality to work with right-to-left languages, assuming your terminal supports them:

[1] I want to thank Reed Mullanix for his post about Unicode and source spans; I had come to the conclusion on my own about the old pinpointing approach being inherently impossible to do correctly from Fennel but his article helped me see that the inline highlighting alternative as a workaround for the problem which could still convey the same information reliably.

in which five different paths lead to methods

2022-02-08 08:52:53

I recently made a change in a codebase I've been working on which illustrated an interesting trade-off around modeling in software. The project was Taverner, an IRC server written in Fennel.

In particular it had to do with the way that channels were modeled. A channel is basically a "chat room"; it's just something that users can join which lets you send messages to anyone else who's also in the channel. In many languages you would define a Channel class which has a bunch of methods like join, part, send, etc. Fennel doesn't have classes, but there are a few different alternatives available¹.

Take 1: Module-based methods

The most obvious approach is to have a channel module which just exports the functions that would have been methods along with a constructor function:

(fn send [{: buffer} nick ...]
  (table.insert buffer [nick (table.concat [...] " ")]))

(fn join [{: members : name &as ch} nick conn]
  (tset members nick conn)
  (send ch "" (.. ":" nick) :JOIN name))

(fn part [{: members : name &as ch} nick ?cmd]
  (tset members nick nil)
  (send ch nil (.. ":" nick) (or ?cmd :PART) name)
  (when (empty? ch)
    (ch.remove)))

(fn flush [{: members : buffer}]
  (each [nick conn (pairs members)]
    (each [_ [sender msg] (ipairs buffer)]
      (when (not= nick sender)
        (conn:send (.. msg "\r\n")))))
  (while (next buffer)
    (table.remove buffer)))

(fn empty? [{: members}] (= nil (next members)))
(fn member-names [{: members}] (icollect [k (pairs members)] k))
(fn member? [{: members} nick] (not= nil (. members nick)))

(fn make-channel [name state]
  {: name :members {} :buffer []
   :remove #(tset state.channels name nil)})

{: send : join : part : flush
 : empty? : member-names : member?
 : make-channel}

There's nothing particularly clever going on here, which I believe is a big strength. Everything is obvious. The make-channel function acts as a constructor, while every other function in the module takes a channel as its first argument, so you can write code like (channel.join ch client.nick client.conn) where ch is a channel table you got from calling the constructor.

The biggest downside here is that it lacks encapsulation. All the data for a channel is exposed in the table that gets passed around to other modules, and it isn't clear which fields are safe to use and which are implementation details which might change later on. In a small codebase maybe this is no problem, but as it grows and changes over time, it will make it more difficult to know what effect a given change might have in a different part of the codebase.

Take 2: Closure-based methods

There's an old saying that "closures are a poor man's objects" and "objects are a poor man's closures". Keeping internal data private by exporting functions whose scope closes over the internal data is one of the oldest tricks in the book:

(fn make-channel [name server-state]
  (let [members {}
        buffer []]

    (fn flush []
      (each [nick conn (pairs members)]
        (each [_ [sender msg] (ipairs buffer)]
          (when (not= nick sender)
            (conn:send (.. msg "\r\n")))))
      (while (next buffer)
        (table.remove buffer)))

    (fn send [nick ...]
      (table.insert buffer [nick (table.concat [...] " ")]))

    (fn join [nick conn]
      (tset members nick conn)
      (send "" (.. ":" nick) :JOIN name))

    (fn part [nick ?cmd]
      (tset members nick nil)
      (send nick nick (or ?cmd :PART) name)
      (when (= nil (next members)) ; last one out off the lights
        (tset server-state.channels name nil)))

    {: name : send : join : part : flush
     :empty? #(= nil (next members))
     :member-names #(icollect [k (pairs members)] k)
     :member? #(not= nil (. members $))}))

{: make-channel}

Now the module only exports one thing: make-channel function, which returns a table that you can think of as if it were an instance of a Channel class. It has functions inside the table which act like methods would. This makes the interface of the channel very clear and well-defined. If you want to do anything with a channel, you have to use one of the functions in the channel table. You can change anything about the internals, and as long as you update everything in the make-channel function, you know you won't break something elsewhere. In a word, it's encapsulated.

But there is one very serious downside to this style²: reloading the code would only affect new channels; existing ones would keep the same old code as before since only the module gets the new functions³. In a normal program I might put up with this, even though I really love reloading. But in a long-running IRC server, it's really not a good idea! Getting everyone to leave a channel so you can recreate a version of it which has the new version of the code is extremely disruptive. I absolutely need the ability to fix bugs and add features while the server is running without disrupting the users, and that meant as nice as this code feels, it's not going to cut it. How can we get both encapsulation and reloadability?

Take 3: Metatable-based methods

Lua tables have one feature which gives them an extraordinary amount of flexibility: metatables. There's a lot you can do with metatables, but for the purposes of this code the most important thing is that you can set an __index method on them which will let you set a fallback for when you try to look up a field which does not exist in the table. This lets us keep doing method lookups using the module table directly (allowing reloads) but also keeping the data itself out of the table which is exposed as the public interface:

(fn send [{: buffer} nick ...]
  (table.insert buffer [nick (table.concat [...] " ")]))

(fn join [{: members : name &as ch} nick conn]
  (tset members nick conn)
  (send ch "" (.. ":" nick) :JOIN name))

;; ... all the methods are the same as the first version

(fn make-channel [name state]
  (let [public {: name}
        channel-state {:members {} :buffer []
                       :remove #(tset state.channels name nil)}]
    (setmetatable public {:__index channel-state})
    (setmetatable channel-state {:__index (require :channel)})
    public))

{: send : join : part : flush
 : empty? : member-names : member?
 : make-channel}

This looks nice! It's very clear what the public fields are (only the channel's name) and the private fields are attached using the first metatable. But if we put the method functions directly into the channel-state table during the constructor we would have the same reload problem as the previous version where the module containing the methods would change after we already pulled the functions out of it, and we wouldn't see the new values. Because of that, we use the module itself as the metatable of the metatable.

There's one big downside to this compared to the previous version: it lacks transparency:

>> (local channel (require :make-channel))
>> (local ch (make-channel "#mychannel" state []))
>> ch
{:name "#mychannel"} ; wait, where are the methods?
>> ch.join
#<function: 0x55c7d468f0> ; but it's found if you ask for it directly
>> (ch:join client.nick client.conn) ; and this works fine!

The functions are found (via __index) when you go look them up, but they do not show up otherwise. This is a common problem with using metatables; they can lead to surprising, unpredictable behavior. While there is a workaround to this (the pairs metamethod) it's error-prone and does not work on all versions of the Lua runtime. Personally I try to avoid metatables unless the downsides of the alternatives are too great. But what other options are there?

Take 4: Class-based methods

Just because Lua and Fennel don't have classes as part of the language doesn't mean you can't use classes; metatables give you the flexibility to construct your own class system if that's what you really want. The middleclass library is one of the most popular implementations of this for Lua, which means of course that we can use it from Fennel too:

(local class (require :middleclass))

(local Channel (class :Channel))

(fn Channel.send [{: buffer} nick ...]
  (table.insert buffer [nick (table.concat [...] " ")]))

(fn Channel.join [{: members : name &as ch} nick conn]
  (tset members nick conn)
  (ch:send "" (.. ":" nick) :JOIN name))

;; the methods are the same as before

(fn Channel.initialize [self name state]
  (set self.name name)
  (set self.members {})
  (set self.buffer [])
  (set self.remove #(tset state.channels name nil)))

Channel

If you're used to Java or Ruby or another class-based language, this may look comfortingly familiar to you. You define a class, and you give it methods. You invoke them using (ch:join client.nick client.conn) notation. But how does it fare on the encapsulation and reloadability fronts?

>> (Channel:new "mychannel" {})
{:buffer {}
 :class {:__declaredMethods {:__tostring #<function: 0x55c7c28450>
                             :empty? #<function: 0x55c7b56660>
                             :flush #<function: 0x55c7bd4aa0>
                             :initialize #<function: 0x55c7b711c0>
                             :isInstanceOf #<function: 0x55c7d45000>
                             :join #<function: 0x55c7d36da0>
                             :member-names #<function: 0x55c7c06720>
                             :member? #<function: 0x55c7f06a80>
                             :part #<function: 0x55c7b710b0>
                             :send #<function: 0x55c7f03e40>}
         :__instanceDict @3{:__index @3{...}
                            :__tostring #<function: 0x55c7c28450>
                            :empty? #<function: 0x55c7b56660>
                            :flush #<function: 0x55c7bd4aa0>
                            :initialize #<function: 0x55c7b711c0>
                            :isInstanceOf #<function: 0x55c7d45000>
                            :join #<function: 0x55c7d36da0>
                            :member-names #<function: 0x55c7c06720>
                            :member? #<function: 0x55c7f06a80>
                            :part #<function: 0x55c7b710b0>
                            :send #<function: 0x55c7f03e40>}
         :name "Channel"
         :static {:allocate #<function: 0x55c7c0a930>
                  :include #<function: 0x55c7b8a5d0>
                  :isSubclassOf #<function: 0x55c7d44f70>
                  :new #<function: 0x55c7d44b40>
                  :subclass #<function: 0x55c7b8a590>
                  :subclassed #<function: 0x55c7af5600>}
         :subclasses {}}
 :members {}
 :name "mychannel"
 :remove #<function: 0x55c7d2e570>}

Yikes! That's a lot of ... stuff. The methods are just dumped straight into a nested table inside the instance itself (twice, for some reason?) and the fields are not encapsulated away at all. The middleclass wiki has some suggestions for how to keep data private but they are quite inconvenient compared to simply using closures. On top of that, the printed representation of the instance is very cluttered and messy. Overall it's not clear that we gain much from this approach beyond a sense of familiarity for people who come from certain other languages.

Take 5: Reloadable, encapsulated methods

So far the closure version from take 2 has appealed to me the most; the tight encapsulation there just feels so tidy. What if we could go back to that but do something about the reloading? Well, there is actually one other concern we haven't touched on with reloading yet, and it leads us to our solution.

When you reload, you're bringing a new version of a module into play in a system that's already running. When your program is a server, that means that you've got "in-flight" connections with active users of your program. What happens when you add a function that expects some new fields that didn't exist when your users initially connected? For example, let's say we add a ban list to the channels. This data wasn't included in the existing channels, but now you need to check it when you join:

(fn join [{: members : banned &as ch} nick conn]
  (assert (not (lume.find (or banned []) nick)) "Cannot join channel; banned.")
  (tset members nick conn)
  (send ch "" (.. ":" nick) :JOIN name))

You could code defensively and make sure that every single reference to the field is wrapped in an or, but that's a drag. You're sure to miss one. And do you really want that check sticking around in your codebase forever? What we really want here is something like Erlang's upgrade process⁴ for when it hot loads a new module. Here we provide an upgrade function which takes the existing table and replaces its contents with the closures from the new version:

(fn make-channel [name server-state ?members ?buffer ?banned]
  (let [members (or ?members {})
        banned (or ?banned [])
        buffer (or ?buffer [])]

    (fn send [nick ...]
      (table.insert buffer [nick (table.concat [...] " ")]))

    (fn join [nick conn]
      (assert (not (lume.find banned nick))
              "Cannot join channel; banned.")
      (tset members nick conn)
      (send "" (.. ":" nick) :JOIN name))

    ;; ... the methods are all the same as the closure-based version

    (fn upgrade [self new-make]
      (each [k v (pairs (new-make name server-state members buffer banned))]
        (tset self k v)))

    {: name : send : join : part : flush
     : empty? : member-names : member?
     : upgrade}))

{: make-channel}

We've extended the constructor to accept all the state fields as optional arguments, (the ones beginning with a question mark) allowing you to build a new version of an existing channel by passing the existing state on in. The upgrade function does exactly this with the private data it's closed over. We'll need to modify the server's reload command to call upgrade on every one of the channels with the new constructor as its second argument. The upgrade function calls the new constructor to get an updated version of the channel, then it takes all these new functions from it and drops them into the existing channel, seamlessly upgrading it in-place without dropping any connections. Any currently-running code which had access to the old channel now can see all the new methods from the new constructor. It's the best of both worlds, and it didn't require sacrificing encapsulation. Best of all only took a few lines of code to accomplish.

But I do want to stress that each of these five approaches are all just trade-offs, and none of them are universally wrong. If you're not writing a server that keeps live connections open, it might not make sense to care about hot-loading upgrades. If you're writing a program that launches, prints its output, and immediately exits, you might not care about reloading, and the second approach is probably fine. If you've got a high tolerance for weird/unexpected behavior, maybe metatables are fine. If serialization is important to you, the first one might come out ahead. Even though the class-based approach is my least favorite, it could suit some projects if the people working on the codebase have a background in object-oriented languages and aren't comfortable changing their style. Context is everything.

All in all I have to say that writing an IRC server has been a lot of fun and not as difficult as I expected it to be. At this point my code is only 366 lines but it supports channels, private messages, channel operators, bans, kicks, listing, and more. Writing an IRC bot is of course easier (a simple one is under a hundred lines) but this could be good if you're looking for a little more of a challenge when picking up a new language.

[1] If you don't know Fennel, you can probably still follow along if you understand scope and closures; the main things to know are that fn declares a function, the curly brackets in the argument list are used to pull fields out of a table argument, curly brackets outside a the argument list are used to make tables, :colon-style is string shorthand, and #(+ 2 $) is shorthand for a function that adds 2 to its argument.

[2] Another smaller problem with this approach is that closures cannot be serialized, so if you had to save off a channel, you can't just take the channel table and write it out to disk. This isn't an issue in Taverner, but it could be for other things which could be modeled this way.

[3] This is because in the Lua runtime used by Fennel, modules are the unit of reloading. Reloading a module involves taking the module table, emptying it out, re-executing the module's file, and pouring the resulting fields back into the original table, meaning that any existing code which had access to the module table can see the new fields. I wrote about this in more detail in a previous blog post.

[4] Of course, Erlang's version is much more sophisticated; it allows the old and new versions of the module to both exist simultaneously, moving process over when it detects an opportune time to call the upgrade function. Since we don't have to worry about concurrency in Lua, it's much simpler.

in which not everything is static, but most things are

2021-09-12 13:51:37

The Fennel programming language recently celebrated its fifth birthday, and we ran a survey to learn more about the community and what has been working well and what hasn't. Fennel's approach has always been one of simplicity; not just in the conceptual footprint of the language, but in reducing dependencies and moving parts, and using on a runtime that fits in under 200kb. In order to reflect this, the Fennel web site is hosted as static files on the same Apache-backed shared hosting account I've been using for this blog since 2005.

Of course, generating HTML from lisp code is one of the oldest tricks in the book[1], so I won't bore anyone with the details there. But what happens when you want to mix in something that isn't completely static, like this survey? Well, that's where it gets interesting.

I put the shotgun in an Adidas bag and padded it out with four pairs of tennis socks, not my style at all, but that was what I was aiming for: If they think you're crude, go technical; if they think you're technical, go crude. I'm a very technical boy. So I decided to get as crude as possible. These days, though, you have to be pretty technical before you can even aspire to crudeness.[2]

When I was in school, I learned how to write and deploy Ruby web programs. The easiest way to get that set up was using CGI. A CGI script is just a process which is launched by the web server in such a way that the request comes in on stdin and environment variables and the response is sent over stdout. But larger Ruby programs tended to have very slow boot times, which didn't fit very well with CGI's model of launching a process afresh for every request that came in, and eventually other models replaced CGI. Most people regard CGI as somewhat outmoded and obsolete, but it fits Fennel's ethos nicely and complements a mostly-static-files approach.

So the survey generates an HTML form in a static file which points to a CGI script as its action. The CGI script looks like this, but it gets compiled to Lua as part of the deploy process to keep the server-side dependencies light.

(let [contents (io.read "*all")
      date (io.popen "date --rfc-3339=ns")
      id (: (date:read "*a") :sub 1 -2)]
  (with-open [raw (io.open (.. "responses/" id ".raw") :w)]
    (raw:write contents))
  (print "status: 301 redirect")
  (print "Location: /survey/thanks.html\n"))

As you can see, all this does is read the request body using io.read, create a file with the current timestamp as the filename (we shell out to date because the built-in os.time function lacks subsecond resolution) and prints out a canned response redirecting the browser to another static HTML page. We could have printed HTML for the response body, but why complicate things?

At this point we're all set as far as gathering data goes. But what do we do with these responses? Well, a typical approach would be to write them to a database rather than the filesystem, and to create another script which reads from the database whenever it gets an HTTP request and emits HTML which summarizes the results. You could certainly do this in Fennel using nginx and its postgres module, but it didn't feel like a good fit for this. A database has a lot of moving parts and complex features around consistency during concurrent writes which are simply astronomically unlikely[3] to happen in this case.

At this point I think it's time to take a look at the Makefile:

upload: index.html save.cgi thanks.html 2021.html
    rsync -rAv $^ fennel-lang.org:fennel-lang.org/survey/

index.html: survey.fnl questions.fnl
    ../fennel/fennel --add-fennel-path "../?.fnl" $< > $@

save.cgi: save.fnl
    echo "#!/usr/bin/env lua" > $@
    ../fennel/fennel --compile $< >> $@
    chmod 755 $@

pull:
    rsync -rA fennel-lang.org:fennel-lang.org/survey/responses/ responses/

2021.html: summary.fnl chart.fnl questions.fnl responses/* commentary/2021/*
    ../fennel/fennel --add-fennel-path "../?.fnl" $< > $@

So the pull target takes all the raw response files from the server and brings them into my local checkout of the web site on my laptop. The 2021.html target runs the summary.fnl script locally to read thru all the responses, parse them, aggregate them, and emit static HTML containing inline SVG charts. Then the upload task puts the output back on the server. Here's the code which takes that raw form data from the CGI script and turns it into a data structure[4]:

(fn parse [contents] ; for form-encoded data
  (fn decode [str] (str:gsub "%%(%x%x)" (fn [v] (string.char (tonumber v 16)))))
  (let [out {}]
    (each [k v (contents:gmatch "([^&=]+)=([^&=]+)")]
      (let [key (decode (k:gsub "+" " "))]
        (when (not (. out key))
          (tset out key []))
        (table.insert (. out key) (pick-values 1 (decode (v:gsub "+" " "))))))
    out))

The final piece I want to mention is the charts in the survey results. I wasn't sure how I'd visualize the results, but I had some experience writing SVG from my programmatically generated keyboard cases I had constructed on my laser cutter. If you've never looked closely at SVG before, it's a lot more straightforward than you might expect. This code takes the data from the previous function after it's been aggregated by response count and emits a bar chart with counts for each response. Here's an example of one of the charts; inspect the source to see how it looks if you're curious:

I had never tried putting SVG directly into HTML before, but I found you can just embed an <svg> element like any other. The <desc> elements even allow it to be read by a screen reader.

(fn bar-rect [answer count i]
  (let [width (* count 10)
        y (* 21 (- i 1))]
    [:g {:class :bar}
     [:rect {: width :height 20 : y}]
     [:text {:x (+ 5 width) :y (+ y 12) :dy "0.35em"}
      (.. answer " (" count ")")]]))

(fn bar [i data ?sorter]
  ;; by default, sort in descending order of count of responses, but
  ;; allow sorting to be overridden, for example with the age question
  ;; the answers should be ordered by the age, not response count.
  (fn count-sorter [k1 k2]
    (let [v1 (. data k1) v2 (. data k2)]
      (if (= v1 v2) (< k1 k2) (< v2 v1))))
  (let [sorter (or ?sorter count-sorter)
        answers (doto (icollect [k (pairs data)] k) (table.sort sorter))
        svg [:svg {:class :chart :role :img
                   :aria-label "bar graph" :aria-describedby (.. "desc-" i)
                   :width 900 :height (* 21 (+ 1 (length answers)))}]
        descs []]
    (each [i answer (ipairs answers)]
      (table.insert svg (bar-rect answer (. data answer) i))
      (table.insert descs (.. answer ": " (. data answer))))
    (table.insert svg [:desc {:id (.. "desc-" i)} (table.concat descs ", ")])
    svg))

{: bar}

In the end, other than the actual questions of the survey, all the code clocked in at just over 200 lines. If you're curious to read thru the whole thing you can find it in the survey/ subdirectory of the fennel-lang.org repository.

As you can see from reading the results, one of the things people wanted to see more of with Fennel was some detailed example code. So hopefully this helps with that, and people can learn both about how the code is put together and the unusual approach to building it out.

[1] In fact, the HTML generator code which is used for Fennel's web site was written in 2018 at the first FennelConf.

[2] from Johnny Mnemonic by William Gibson.

[3] If we had used os.time with its second-level granularity instead of date with nanosecond precision then concurrent conflicting writes would have moved from astronomically unlikely to merely very, very unlikely, with the remote possibility of two responses overwriting each other if they arrived within the same second. We had fifty responses over a period of 12 days, so this never came close to happening, but in other contexts it could have, so choose your data storage mechanism to fit the problem at hand.

[4] This code is actually taken from the code I wrote a couple years ago to handle signups for FennelConf 2019. If I wrote it today I would have made it use the accumulate or collect macros.

in which a laptop defies the trends

2021-09-06 11:18:46

Over the years I've been a consistent[1] user of Thinkpads; early on because I liked the keyboards, but then later just because I wanted hardware that would last a long time rather than a machine with a soldered-in battery that's designed to be more disposable.

My most recent device was a Thinkpad X301 built in 2008 which I started using in 2016. While it's no speed demon, using Firefox with uBlock Origin configured to block 3rd-party scripts by default left it feeling quite usable for my purposes, and the physical design of the device was perfect. They used a rubberized coating for the chassis in the X301 that I haven't seen in any other model that feels really nice on the palm rest. Unfortunately while nearly every component of that machine has withstood the test of time, the battery has not. The original battery's charge is down to around 90 minutes, and while it's swappable, working new batteries in this form factor simply cannot be purchased for any amount of money. I bought from two separate vendors claiming to have original batteries, but both of them sold me a battery which ballooned up and became unusable after a month or two.

When I started to look for replacements I was dismayed. So many of the newer models had fallen into the Appleization trap—everything must be made as thin and as glossy as possible at the expense of every other concern. I don't want a thin laptop! I want a laptop where I can look at it and see what's displayed on the screen instead of my own face staring back at me. It seemed it was still possible to find a model with a replaceable battery, but even this basic feature was becoming increasingly rare.

A couple years ago I became aware of the MNT Reform laptop, and it seems like the perfect antidote to the mistakes the entire industry seems dead-set on repeating. It's a laptop that's focused on open design with schematics freely available and all parts easily serviceable by the end user. Finding this was like a breath of fresh air; it's like someone was finally listening to my frustrations.

The MNT Reform has been described as "the anti-macbook" which I think is fitting, but ironically I prefer to think of it as the Apple ][ of laptops (in a good way). If you're like me and you're fed up with thin laptops, you will be pleased to see that this machine is chonky. It has to be in order to have room for its three most unique features: a mechanical keyboard, a trackball, and a standardized 18650-cell battery bay. Originally the batteries were what caught my attention after all the trouble I'd had buying replacements for my Thinkpad, but when I saw the mechanical keyboard I knew I had to have one. (But also: can we talk for a second about the audacity of producing a laptop with a trackball? Much respect.)

Part of having an open design is having everything documented. While you can get the schematics for everything from the motherboard PCB to the 3D printed trackball buttons, the part that nearly everyone will benefit from is the excellent Operator Handbook which describes the usage of the system in detail.

Other than the thick size, perhaps the most eye-catching feature of the Reform is its transparent bottom plate, which is laser cut from acrylic. Similar to the open lid of the Apple ][, it invites you to take a look inside and reminds you that this machine isn't magic: it's wires and capacitors and screws and connectors. It's physical parts you can understand and control.

This machine isn't perfect though; there are trade-offs. The four ARM Cortex A53 cores in the CPU do not perform any out-of-order or speculative execution, which means they are not vulnerable to attacks like Spectre and Meltdown, but at the cost of speed. (I'm using it mostly for chat, email, and developing the Fennel compiler, and it's plenty fast for that.) The lid closes with a satisfying magnetic snap, but it doesn't have a lid sensor, so you'll have to turn off the screen yourself. The stock wifi antenna's range is quite limited. (But you can easily replace it!) Suspend is currently not super reliable, but there are ongoing efforts to improve that.

The keyboard is ... well, it's head-and-shoulders above any other laptop keyboard I've tried. Instead of a comically huge space bar, the bottom row is broken up into a reasonably-sized space bar plus several other useful keys. But it's still frustrating in a few ways. (Note that I'm a major keyboard nerd who has spent a lot of time getting my keyboard setup just right and I am far more picky about this kind of thing than most people!) While you can reprogram the keybord firmware to reassign keys with ease, the physical layout is very awkward. It has a conventional row-stagger which is not great but also not unusual. The problem is that in most row-staggered boards each row is offset from the one above it by 1.25 key widths or so, and on the Reform it's 1.5. Even 1.25 is too much (zero would be ideal), but 1.5 makes it so you have to contort your hand even more to hit keys on the "ZXCV" row.

Of course, it's a hackable laptop! Reprogramming the firmware to rearrange the keys can't fix problems with the physical arrangement, but I've built hundreds of keyboards by hand, so I planned to do design and construct one from scratch for my Reform when I got it. Unfortunately it's a little more complicated than I anticipated; the stock keyboard is integrated with the system controller which is involved with powering on the entire system and controls the OLED display containing the battery indicator, etc. I couldn't just adapt my existing design for a new form factor.

Luckily the folks at OLKB announced they were developing a kit for an improved keyboard with no row-staggering. I'd prefer an ergonomic design, but this is still a big improvement over the stock board, which is itself light years beyond anything I've ever used in a laptop before. I'm looking forward to building one out.

Overall I'm thrilled with this laptop. It's available both as a DIY set which needs some assembly (just screwing things together and plugging connectors; no soldering) and as a prebuilt laptop, but honestly if you're anywhere near the target market for the Reform, you're probably going to enjoy the assembly process and are best off skipping the pre-assembled option. In the end the Reform is a powerful antidote to the user-hostile trends which have prevailed in computing over the past decade or so, and if you're anything like me and you don't mind a little tinkering, I can't recommend it enough.

Update (2023): While the Reform is a great machine overall, my use cases are extremely idiosyncratic in that I need to use EXWM as my window manager for accessibility reasons. The Reform's GPU supports Wayland well, but not Xorg. So I found it to be very frustrating to use. In the end I replaced it with a modified X210 Thinkpad from 51nb that can run EXWM which I'm much happier with.

[1] Starting with a T60p in 2007 followed by an X61, then an X200s, and finally a X301. I took a brief detour with a Samsung ultrabook but the keyboard was so unpleasant that it didn't last long before I sold it.

in which there is no such thing as a functional programming language

2021-03-22 08:35:03

There is no such thing as a functional programming language.

Ahem. Is this thing on? Let me try again.

There is no such thing as a functional programming language.

All right, now that I've got your attention, let me explain. Functional programming does not have a single definition that's easy to agree upon. Some people use it to mean any kind of programming that centers around first-class functions passed around as arguments to other functions. Other people use it in a way that centers on the mathematical definition of a function such as ƒ(x) = x + 2; that is, a pure transformation of argument values to return values. I believe it's more helpful to think of it as a "spectrum of functionalness" rather than criteria for making a binary "functional or not" judgment.

So functional programming is an action; it describes something you do, or maybe you could say that it describes a way that you can program. Functional programming results in functional programs. Any given program exists somewhere on the spectrum between "not functional at all" to "purely functional". So the quality of "functionalness" is a property that you apply to programs.

Obviously "functional programming language" is a term in widespread use that people do use to describe a certain kind of language. But what does it really mean? I would argue that a language cannot be functional; only a program can be more or less functional. When people say "functional programming language" what they mean is a language that encourages or allows programs to be written in a functional way.

Except for very rare cases, the language itself does not force the programs written in it to be more or less functional. All the language can do is make it more or less difficult/awkward to write functional programs in. Ruby is rarely called a functional programming language. But it's possible (and often wise) to write functional programs in Ruby. Haskell is basically the textbook example of a functional programming language, but imperative Haskell programs exist. So calling a programming language functional (when taken literally) is a bit of a category error. But "a language that encourages programming in a functional way" is an awkward phrase, so it gets shortened to "functional programming language".

Incidentally the exact same argument about "functional programming language" can be applied to the term "fast programming language". There is no such thing as a language that is fast. Only programs can be fast[1]. The language affects speed by determining the effort/speed trade-off, and by setting an upper bound to the speed it's possible to achieve while preserving the semantics of the language[2]. But it does not on its own determine the speed.

Please don't misunderstand me—I don't say this in order to be pedantic and shout down people who use the term "functional programming language". I think it's actually pretty clear what people mean when they use the term, and it doesn't really bother me when people use it. I just want to offer an alternate way of thinking about it; a new perspective that makes you re-evaluate some of your assumptions to see things in a different light.

[1] If you want to be even more pedantic, only individual executions of a program can be fast or slow. There is no inherent speed to the program that exists in a meaningful way without tying it to specific measurable runs of the program.

[2] For instance, the Scheme programming language has scores of different implementations. The same program run with the Chez Scheme compiler will often run several times faster than when it's run with TinyScheme. So saying "Scheme is fast" is a category error; Scheme is not fast or slow. The same is true of Lua; you will usually get much faster measurements when you run a Lua program with LuaJIT vs the reference implementation.