Queue theory & post offices

Hello again, and welcome to your weekly edition of software engineering meets quantum entanglement. Also known as the Fediverse.

Last week we showed off a big upgrade to ActivityPub replies, alongside a steady stream of beta bug fixes, superseded by new and different bug reports. The first few weeks of beta testing have been pretty stable, and we're encouraged to see that no servers have melted (yet) — so, in the next few days, we're going to be sending out the next batch of beta invites to get a few more people in the door.

If you haven't yet registered your interest in joining the private beta, you can do that over here.

What's new with ActivityPub?

Quite a few things! We're still working our way through bugs and performance issues as they come up in beta testing, and we're making steady progress. In the past couple of weeks we've also upgraded our logging and monitoring, which makes it much easier to find and fix issues when they happen.

On the user interface front, we've improved how articles render inside Ghost. When you click on a post, it opens in a drawer. For 'note' objects (like Mastodon posts) that's a small drawer, appropriate to the short content length.

For 'article' objects (like Ghost posts) we now show a more expansive drawer with extra breathing room for longer posts with embedded rich media.

We also now show 'like' and comment counts on posts, where available, to give more context about how many people have engaged.

And, as mentioned last week, Ghost is now starting to show up in FediDB. This means we'll be able to watch the growth of ActivityPub-enabled Ghost sites over time, and see how it's being used out in the wild.

The big task for the past week, though, was getting started with queues.

Scaling ActivityPub with a queue

When you start to scale ActivityPub you quickly realise the need for a piece of technology called a queue. Fortunately for us, all of the engineers working on ActivityPub at Ghost are British, and, in case you were unaware, the British are in fact the best in the world at queuing. It's a deeply ingrained part of the culture that provides sweet relief of something else to complain about other than the weather.

In fact, if two British people stand close to each other and accidentally form a queue, usually a third will join it, just in case it's important.

So, our team is fully qualified for the task at hand.

In software terms, a queue is a tool that comes in handy when you need to improve the performance and stability of a large number of sequential operations. Say, for instance, sending a post to thousands of subscribers.

The simple way to do this is to have a function that loops through each one of your subscribers and sends out the post to their inbox. This is exactly how Ghost currently sends new posts to the Fediverse when you hit publish.

This mostly works, but it has some major drawbacks. First, it's slow and uses a lot of resources. Second, if anything goes wrong then the whole thing fails in a way that can't be recovered.

In our current setup, you can think of Ghost like a mailman, going door to door on a round-trip to deliver your new post to waiting subscribers.

Ghost spends a lot of effort looking up each address, visiting it, and putting your post into the inbox. If one address turns out to be hard to find, it takes even longer to deliver to everyone else. Worse still, if something unexpected goes wrong with the delivery (like the address no longer exists) then the mailbag spontaneously explodes and no further deliveries are completed.

When Ghost is only delivering a handful of posts these issues are pretty rare, but when it's trying to deliver to thousands (or millions) of addresses, it becomes a serious problem.

A queue makes this whole system more robust. Instead of trying to deliver posts to each subscriber, Ghost sticks all the posts onto a queue, and separate workers go through the queue handling the actual delivery.

To extend our metaphor, rather than delivering each post by hand, Ghost goes to the Post Office and drops off all the letters that need to be sent in one place. From there, a different group of agents take care of individual deliveries. This all happens faster because they work in parallel — and if one delivery fails, then it can simply be retried later without inhibiting others.

So, TLDR, we've started building a Post Office.

Fedify, the library that Ghost uses for ActivityPub functionality, has support for queues built-in natively — and for Ghost(Pro), we're integrating it with Google Pub/Sub to handle everything for us.

This architecture should eventually allow us to reliably scale ActivityPub deliveries to practically any size, while keeping Ghost performant and lightweight in the processing it has to do by itself.

And, of course, as more publishers join the beta with more subscribers and more posts to be delivered, the importance of this increases exponentially!

Bet you didn't think you were going to spend your Sunday reading about pugs going to post offices. But here we are.