“Once you have divided labour between man and machine, all that is left for man to do is to shepherd knowledge.” - Mark Burgess

A few weeks back I was catching up with my fellow Clojurian friend Kevin and we got talking, once again, about my 'visions' for knowledge management. It wasn't the first time we'd delved into the topic and this time he really pushed me: "If you believed in this enough you'd be working on it every day! [...] Is this even a problem you actually have?"

Rather than giving a succinct response, I undoubtedly waffled on too long about all the world's problems which I feel are deeply interrelated and could be solved, surely, if only our software systems presented us with more profoundly effective media.

I firmly believe that the end goal of this networked computer age should be to help us (all of us) think together and feel together. Maybe between emails and spreadsheets and Wikipedia and YouTube we just about have that vision covered satisfactorily already, but my hunch is that with the right new ingredients we can go a lot further and a lot faster.

But what are those ingredients, what forms might they take, and am I even the right person to start cooking things up?

Does the world even need new software to be written or do we simply need to better wield and 'sell' what already exists? And what's my role to be? Should I be trying to build a web app? A database?

Should I be focused on assembling a real dream team under the umbrella of some corporate structure? Perhaps it's sufficient to contribute in some small yet significant way within a network of broadly autonomous individuals who already collaborate around an existing common vision? Or do I need to start a new religion?

Or maybe LLMs have already solved most of the foundational problems I perceive and I haven't quite realised it yet.

Nothing is certain. But, as things stand, I feel the focused mission of XTDB remains my strongest bet - a moonshot chance to align a multitude of economic motivations with societal needs and to disrupt a somewhat stagnant software industry with simplicity. That's why I will continue to push it forward as best I can each day.

But in the interests of "building in public" and of not putting all my eggs in a single basket, the least I should do is write down my thoughts here and offer my up tokens to the winds of chaos (LLMs, the wider public etc.) to do their thing with these thoughts as they please, without me getting in the way.

XTDB for Knowledge Management

"You can not see the future without seeing the past" [...] "The good hunter always climbs the highest dune before his hunt. He needs to see. As far as he can see. You need to SEE!", Dune: Part Two (2024)

My hot take on knowledge management is that most electronically-mediated human communication should be oriented around "plans for the future". Even if just very vague hopes about how we want the future to look.

However, to make effective plans we first have to be able to talk about time and reflect on the past. XTDB is designed, for the most part, to accelerate our ability to capture and refer to the past. My hope is that it should therefore also be able to serve as an essential component for all kinds of novel takes on knowledge management software, and not just obscure financial systems.

The opening quote by Mark Burgess hints that I'm far from the first person to care so deeply about this knowledge management stuff (especially in recent years - see also: Notion), but really I included the quote because Mark's latest work is highly compelling to me. More thought out, 'real' and immediately usable than anything I've been able to conjure together thus far. I look forward to helping Mark work through his vision because it feels like we're pointed in this same direction. The project uses Postgres, for now.

Here Comes the Son

In more important news, I had the most wonderful birthday present this year:

He's a delightful little chap. Very 'awake' 😅 And it's been a conveniently timed kick up the backside!

F O C U S

Erase Begin ...a new website!

I figured now is finally the time to get https://jdt.me off the ground.

Having recently attended Strange Loop I was contemplating writing up some brief(er) thoughts as a Twitter thread, but hopefully we can all agree that a new blog is a much better investment for the future.

I also wanted to make use of some downtime to experiment with Clojure tooling for static site generation and the much-hyped fly.io. A static site is the expedient choice for now. If all goes well then I'll upgrade to Platypub (i.e. an XTDB-powered server) once I've successfully ramped up my output!

Strange Loop 2022

Huge kudos to everyone involved in making Strange Loop what it is—I feel like it represents the best of our industry—and I had a wonderful time last week meeting old friends and new. It was great to be back after my last visit in 2019.

I was joined at the conference by a wider group from JUXT as we were platinum sponsors of the conference and therefore needed enough people to operate a booth. The main idea behind sponsoring was to get the JUXT/XTDB brand out there and do some face-to-face networking. Mission accomplished.

My primary purpose for attending was to deliver a bitemporal data workshop, which went off without a hitch. There was an interesting mix of industries represented in the room, from groceries to payments to healthcare. It felt great to be able to use Core2 (XTDB's work-in-progress next-gen engine) to run through SQL:2011 exercises alongside the "Bitemporal Visualizer" tool that I've been building recently (built using Scittle...another of @borkdude's magnificent Clojure innovations!).

My biggest takeaway from the hundreds of conversations I had across the 4 days was that the demand for an immutable database is tangible and that bitemporality in particular is a practical and widely applicable model for immutability, regardless of industry.

DOOM

However, my undisputed highlight of the conference was meeting John Romero, chatting with him about Commander Keen, and picking up a signed copy of Masters of Doom. It has been ideal post-conference holiday reading. In particular it is inspiring to understand what John and the rest of the id team achieved from their humble beginnings, and it really feels like reading the backstory for a non-trivial amount of my own childhood and earliest computing memories.

The id team spent huge amounts of effort fighting/embracing the limitations of early PCs to great effect, and there's no doubt that constraints are essential to creativity. For databases though it seems that removing traditional storage constraints through the use of ~infinite cloud object storage opens up entirely new worlds of creative possibility.

Håkan's talk on Light and Adaptive Indexing for Immutable Databases was a great overview on what immutability affords the internals of a database, but from a user perspective I think the most succinct explanation of what immutability affords is "god mode"...

IDDQD is a cheat code used in the 1993 first person shooter game Doom, which enables a game state that prevents the player from being damaged or killed known as "god mode" -https://knowyourmeme.com/memes/iddqd

First-class immutability is what drew me to Datomic/DataScript/Clojure in the first place. It is the ultimate superpower for tackling the nightmare of legacy, mutable software systems.

Same again next year?

I really hope to be back in St. Louis again in 2023, at the best software conference in the world! What would you like to see there?