After a short tenure as the generator for this website, I’ve retired Gatsby and replaced it with Stasis, a small and simple static site generator written by me in Python. At the risk of spending more time discussing how I generate my blog rather than actually contributing real content, in this post I will describe the frustrations I encountered with Gatsby, the motivations I had for writing my own generator, and some of the things I learned along the way. I will also try to make the case that you too should write your own static site generator. It’s not a hard problem, but it will help exercise your engineering skills in a few ways.

Stasis may not be of use to anyone else, and given my track record I may eventually abandon it myself and move to yet another generator. But for now at least, it fits my needs exactly, since I wrote it to do exactly that. And now I’ve got nobody left to blame when I inevitably become frustrated when it won’t do what I want.

It’s not just about Gatsby

Gatsby turned out to be an immense frustration and not a solution in which I wish to continue investing time. I do not regret it, however, because it did serve me well as a learning exercise. It gave me space to practice thinking in React in a different domain, so I got to see and play with multiple different approaches to problem solving in React. If I am never willing to break out of my comfort zone, then I will eventually stop learning and that’s not a place I want to land. Diving into Gatsby was an excellent foray into an interesting avenue of front-end development and I’m much better for the experience. That said, however, I’ve concluded that what remains for me to learn from this is not worth the fighting I will have to endure to continue maintaining my blog with Gatsby.

A brief summary of my frustrations:

• Extensive amounts of code are needed to build a simple blog
• Rapid development means my site is no longer reproducible
• Far too much pain working with GraphQL
• Key functionality is relegated to plugins that are barely capable
• Customization for things like excerpt handling and post url formatting requires hacky solutions
• No guarantee that the output will be durable over the long term

When I started working with Gatsby, my goal was to simply reproduce what my site looked like with Jekyll. As you can see in the repository, I had to write a lot of code to make that happen. Features that were simple configuration options in Jekyll had to be written from scratch to make them work with Gatsby. I didn’t want to compromise and settle for one of the starter blog templates because it wouldn’t really say much for the tool if I couldn’t even make it into a replacement for what I already had working. So this led to me writing a lot of brittle Javascript, which is now largely broken.

Six months after I built it, my code was no longer compatible with the latest version and I couldn’t reproduce my own site. I knew what I was getting into when I went down this road so I should have seen this coming. Whether it’s a problem with my code, a change in the Gatsby API, or an issue in my environment I am still not entirely sure and to be honest, I am not curious to explore why it’s broken.

The amount of code I had to write to produce a simple two-column blog with paginated indexes of posts, some static pages, and categories was not inspiring. Gatsby, out of the box, is really not well suited for a basic blog. It clearly has promise as a platform with which someone could build a generator that would abstract away all the necessary cruft to support a decent blogging workflow, but having to code it all directly requires far too much scaffolding. A lot of the pain is in working with GraphQL—an extremely overengineered solution to the non-problem of attaching metadata to text files. I initially embraced it because “new shiny”, and in the right use case it could make sense. But that use case involves having to work with multiple disparate data sources and unify them together in the front end. My blog reads markdown files into html. There’s just no reason to bring GraphQL into this equation.

The consequence of Gatsby’s threadbare support for basic blogging functions is that you either have to write the code yourself or try to find a plugin that does the job. The issue with delegating key functionality to plugins is that many of them appear to have been hastily written, haven’t been well tested, and can barely do what they advertise. The S3 plugin in particular is worth calling out here. You can configure it to either delete everything or delete nothing, and then wholesale upload your content directory. The fact that it will gladly upload .DS_Store to your public website without complaining is a little short of infuriating. I just don’t want to talk about it anymore.

The jist of my frustration is that I doubt I can maintain this over the years.  I will run into the same maintenance and upgrade headaches and it’s just not worth it when I’ve had to write custom code to get the features I wanted in the first place.  The output is a mess of json files that are stitched together by the runtime. Do I have any hope that this will be readable by browsers in ten years time? A static site should be, um, static.

I wanted to try Gatsby because I had been doing a lot of work in Javascript and working on it was a good way of practicing with React in a different domain. I was foolishly confident I could keep up with its rapidly changing development. I also overlooked multiple red flags that bothered me, but weren’t enough at the time to get me to reconsider my rash decision. Scrutinizing the move to Gatsby I discovered a host of additional minor quibbles that alone were not deal-breakers, but in total left me very displeased. Many of my internal links were broken, parts of the site were redirecting to insecure http, and strange things happened to my images.

But enough about Gatsby, it’s time to move on.

Why did I write my own static site generator?

Anytime you’re using a framework, if a feature you want is not exposed as a configuration option, then you’ll have to dive into the internals of how the generator works, and this might lead you to writing brittle code. Jekyll largely worked for me because I could do almost everything by editing the config and tweaking the templates. But I had no interest in returning to Jekyll for all the reasons I moved to Gatsby in the first place. Before I embarked on writing Stasis, I evaluated multiple alternative generators, replicating my site in each one—Blogdown, Pelican, and Eleventy were the top contenders. I learned a lot about each one, but in the end what I really learned is that if I ever wanted to be fully satisfied with the code I use to generate my site, I was simply going to have to write the thing myself.

Apart from the unplanned obsolescence aspect causing maintenance nightmares, another reason I have not been able to fully embrace existing generators is that I have many layout quirks that I insist I must incorporate into my site. To name a few:

• I want my post url format to be /YY/slug.html. It’s never the default in any generator and usually requires hacking to make it work.
• Topics (aka “categories”) should be sorted in order of first appearance, not alphabetically or based on size.
• Excerpts: what appears above the excerpt separator should appear on the index page but not in the actual blog post.

I won’t provide a full eval of the alternative generators I tried, but just a few notes. Blogdown is attractive because I work a lot in the R ecosystem. It uses Pandoc and Hugo under the hood with the primary value add being the ability to write posts in RMarkdown and run live code examples during rendering. However, I don’t have a burning need to do that. A better workflow for me when I have posts with some data analysis in R is to simply incorporate the output directly into a plain markdown file. This ensures that I know what the output is going to be and doesn’t break sometime years later.

Pelican is the closest to a drop-in Jekyll replacement, and it’s written in Python so I have at least a chance at customizing it. It’s also extremely stable, which has both good and bad aspects to it. What’s good is that it supports a wide range of common and not-so-common blogging features, but what’s bad is that the code is is highly obtuse, as it’s been progressively abstracted and solidified over many years among dozens of contributors. With all the mature generators like jekyll and pelican, understanding how it works requires a lot of effort even though static site generation is not really a complicated engineering domain.

Eleventy is a Javascript static site generator, and far more purpose built for blogging than Gatsby. It’s definitely easier to work with out of the box, but still leaves me writing an awful lot of brittle code to do what I want.

After mucking around and not finding exactly what I wanted, I decided to write down what it is that I did want. That might help me find what I’m looking for, or at least clarify my expectations about what I static site generator should offer:

Requirements

• Conversion of Markdown posts to html, preferably using a modern enhanced dialect of Markdown.
• Workflow similar to Jekyll: init, clean, build, serve, deploy.
• An easy to use templating system.
• A basic development server for previewing the site locally.
• Index pages with pagination.
• Static or “flat” pages, also written in Markdown and rendered to a specified url (eg. about, contact, 404).
• A sidebar with links, recent posts, and topics.
• Topic pages.
• RSS feed generation.
• Deployment to S3 with dry-run, ignore on server, exclude from upload, and Cloudfront invalidation.
• Static file pass-through for CSS, Javascript, media, and legacy content.
• Purely static html output.

Of all the Markdown conversion tools I’ve encountered, Pandoc is the one I like the most. It has a bunch of nifty enchancements and is useful for far more than simple Markdown to html conversion. And, among all the templating engines out there, Jinja2 has usually worked out very well. So, I really wanted some way to leverage both of these. There’s really not much involved in generating a website from markdown posts, especially when the real heavy lifting—parsing markdown and template rendering—is already taken care of for you. There are endless tutorials on how to write one. I’m sure some tech companies would use it as an example project for a take-home interview test! Python is also highly suitable for this domain. For a clear crisp example of how to spin up your own static site generator, see this tutorial from Thea Flowers.

Writing the generator yourself allows your code to get to the point quickly.  It doesn’t need to satisfy everyone’s use case, it doesn’t need fancy abstractions, it doesn’t need to support themes or plugins, it doesn’t need an extensive test suite (or in my case any tests at all!), and it can fail ungracefully.  But you’ll understand it, so once it does what you want, you’re basically done. In stasis, I’ve made some effort at generalizing, on the off chance that I may want to use it for something besides my own personal blog, and that potentially someone else who has come to the same unsatisfaction with existing site generators may want to use it too.  But not enough to make it difficult to follow.  All I’ve done in stasis is provided a few convenient classes in which to stitch together my web pages along with a sprinkling of metadata.

The most involved piece in writing Stasis has been deployment to S3.  This is also notably the piece at which every major generator leaves you to your own devices.  Part of this is practical, since there are dozens of options for hosting a static site.  So it’s understandable that an SSG would resort to plugins rather than try to design a generic hosting-agnostic deployment procedure that would work everywhere and for everyone.

I have decided many times that it’s not worth finding an alternative to S3. The S3 deployment tool that I really liked when I was using Jekyll is no longer maintained, and none of the alternatives match its features. Because I have a lot of legacy content on my site with which I need my blog to gracefully coexist, your basic s3 sync is not going to cut it for me. There are two options I needed to have:

1. “Ignore on server”: S3 objects matching these patterns would not be modified or deleted.
2. “Exclude from upload”: Local files that should never be uploaded to the server.

And of course I also wanted to enable gzip compression. Having to do all this manually is not really a good option. You might think that a blog post simply updates one new file, but a static site doesn’t work that way. Almost all the files need to be touched, because you have index and topic pages that reference the new post, and a sidebar with a “latest posts” which all have to be updated. This is the drawback in not having a database generate the site dynamically. But at my current writing pace, it still won’t make sense to go back to database-backed dynamic content for many more years.

Things I learned writing Stasis

Now on to the fun part. I made my decision and started developing. The first pass was a single 300 line python module that did almost everything except for the S3 deployment. Once I had this proof of concept working, I then started refactoring, doing a little bit of generalization and abstraction, moving some hard-coded variables into config options, and turning the module into a proper package. Again, this is not a terribly difficult engineering problem, so I didn’t expect to make any earth-shatteringly beautiful discoveries here. But I approached this as a serious exercise to make the most of it. In the end I’m glad I did. Even though Stasis is clearly still very much alpha, I am already quite satisfied with the results and it is definitely good enough to power my blog and share with others.

I think the real lesson for me is that Python just keeps getting better.

Python + Pandoc = Slow. I punted on a live reload or file system watch feature, even though it doesn’t look like it would be that hard to do.  Hey, I’m a slacker, and this feature didn’t seem all that necessary until I discovered how convenient it really is and why almost every generator does it.  When editing a post and wanting to get instant feedback, I have to exit the development server and rebuild the site.  For a small number of posts, it doesn’t sound like such a big deal.  However, a full build of my site with 47 posts takes about 15 seconds.  So I had to choose between being lazy or being impatient.

Live reload can be tricky. Not only do you need to implement a file system watcher, you also have to inject some javascript into the pages, remembering to strip it out later.  I just didn’t want to bother, so I looked into how I could optimize another way.  Some quick profiling showed that by far the biggest time sink was generating each page’s html.  I still don’t know if it’s because Pandoc itself is slow, or that in calling Pandoc I use Python’s subprocess library to call out to it. But it doesn’t really matter who the real culprit is because the end result is some serious slowness. However, since there isn’t much point in regenerating a page’s html if the source hasn’t changed, it seemed like I could easily benefit from some sort of caching system. My first instinct was to do some pickling, which made me cringe because past history revealed this to always be a pain. Then I discovered the shelve library which provides a nice clean layer on top of pickle with almost zero effort. After building a post the first time, I shelve it to a post store. On subsequent builds, we first check if the post “is current”, meaning the source document hasn’t been modified since the output was created. If it’s still current, we can just load it from the post store and skip needlessly calling out to Pandoc to re-render the same result.

I was so happy until I tried it out and found it actually took longer than a fresh build! Something else was not right: my post store was 90MB. Um, my site is about 5MB in total, so how is a cached version almost 20x as large? Each post needs prev_post and next_post variables to enable navigation links in the footer. Naively, I was writing the entire post object into the post store and since Python loves to let you think in terms of references when it’s doing things by value all along, when the post was shelved, prev_post and next_post turned into recursive copies of their respective objects. I essentially had my entire site’s content embedded in each and every post! To solve this I had to take prev_post and next_post out of the post store and just build them dynamically instead of storing them. In the end, it works quite well. Not fully as convenient as a live reload, but close enough.

S3 synchronization with Gzip and S3 ETags. I’ve seen people using (and struggling with) S3 ETags as a way of detecting if a local file differs from its respective S3 object. In the absence of encryption, ETags are usually MD5 hashes of a file, unless the file was uploaded in multi-part, which happens at around the 5MB boundary. Then, it’s an MD5 hash of a concatenated string of MD5 hashes of each upload part. Yuck. I didn’t want to mess with it, so I tried to cheap out by simply comparing S3’s LastModified with the local path.stat().st_mtime. Except yeah that doesn’t work too well because every time you rebuild the site suddenly every single one of your local files has been modified! So originally my S3 deployment wanted to insist on copying every single file on every deploy.

So I had to do the MD5 hashing after all, but then another wrinkle popped up. Since I want to keep transfers into S3 to a minimum, I’m gzipping most of my files before uploading. Note that you can just let Cloudfront do the compression for you when serving to clients, but that won’t help you keep your ingress bandwidth down. No matter what I tried, my MD5 hashes of freshly gzipped files would always be different from what was on S3, even though the content of the files did not change. A fortuitous stumble upon a Stack Overflow post from 2008 and I finally understood why. By decree from God, every gzip header needs to have a timestamp. So by default, Python’s gzip library throws in the current time. So it doesn’t matter whether your file content has changed or not, because Python is going to ensure that the file is different and that subsequent MD5 hashes of it will never be the same! Fortunately, the constructor for GzipFile lets you pass in your own timestamp. So at least now I can ensure that I can read the ETag from S3 compare it with the hash of a newly gzipped local file, and only see a difference if the content has actually changed. I haven’t run into the multipart upload yet, but at the moment I am not planning on writing a blog post containing 5MB of text.

Pathlib is awesome. Shelve isn’t the only new shiny Python library that has made my life so much easier. Pathlib also deserves a serious shout out. Since Stasis needs to do a lot of file system manipulation, I was actually dreading this part because of all the verbose os.path code I suspected I would need to write. Enter Pathlib! This library has finally made it easy to work with the file system in Python.

Developing packages with setuptools and console scripts. This was perhaps the most unexpected (to me) discovery of this project. My bias has always been to avoid the formalism of packages and simply hack together a set of modules and throw a main() function in a run.py. It’s been a long time since I created a package and it was not a fun experience. I learned that it is actually far easier to develop if you have a package as the end goal in mind. Setuptools gives you more flexibility than requirements.txt so you can handle dependencies very easily. In particular, being able to do pip install -e . and have access to your programs defined in console_scripts is so much better than the old way I was doing it, which meant a lot of hard-coding of constants (eg. directories to use) at the top of my main module. From now on, I will try the packaging approach, even for small one-off utilities. The boilerplate is minimal and the gains far outweigh the few extra steps you need to declare your setup.py properly.

So now I have spent all this time writing a generator for a blog that I barely have time to write in, because I’m working on the generator to write more blog posts about writing generators….

And I still haven’t done anything with the awful CSS on my website! Maybe next year.