(This is a long post. In the words of Blaise Pascal, “I have made this longer than usual because I have not had time to make it shorter.”)

I see the Nix project to have passed a critical moment. I think the Nix project has made great strides in solving some very hard problems, but my fear is that due to mismanagement, Nix will not succeed. In this post, I hope to share my perspective on what a better future for declarative, reproducible builds could look like, but sadly, I don’t think it involves Nix as it exists today. It’s hard to know what the future holds, and I am certain I do not hold the solutions to the problems that exist with Nix. However, I feel it would be a shame for the progress to be lost.

As background, on 2024-04-21, a group of concerned contributors and community members published an open letter to the NixOS foundation detailing Eelco Dolstra’s ill effects on the Nix project’s governance. On 2024-04-26, Eelco published a blog post that has caused many package maintainers to leave, including me. While I don’t have direct experience with the problems in the letter, I very much trust the folks who signed the letter. I am disappointed with how Eelco has chosen to handle the situation. I personally think this is a watershed moment for Nix, and not for the better. Such actions set the precedent for how an organization will act in the future, and given the power structure, there’s not much that can be done. In my mind, this endangers the health of Nixpkgs moving forward, and without healthy package maintenance, the Nix project is doomed. Consider this post a eulogy for a dream.

While the solution-that-is-Nix may evaporate, the problem it solves remains, at least for me. The rest of this blog post is my attempt at sketching what a suitable replacement would look like, in the hopes that it inspires someone to succeed where others have not. Perhaps a replacement is a fork, perhaps it is a wholly new project, perhaps it is the Nix project under better governance. I don’t know.

I’m happy to announce a project I’ve been working on for the last month: pql. pql is a pipelined query language inspired by the syntax of Microsoft’s KQL and Splunk’s Search Processing Language that I developed in collaboration with the fine folks at RunReveal. pql takes queries like:

logs
	| where eventName != ''
	| summarize AllEventNames=groupUniqArray(eventName) by sourceType

and translates them into SQL like:

WITH "__subquery0" AS (SELECT * FROM "logs" WHERE coalesce("eventName" <> '', FALSE))
SELECT "sourceType" AS "sourceType", groupUniqArray("eventName") AS "AllEventNames" FROM "__subquery0" GROUP BY "sourceType";

I personally find the pql query easier to read and modify, while still retaining the power of what a relational database has to offer. pql supports all the basics of filtering, sorting, and aggregating, as well as complex joins.

Under the hood, pql builds a full Abstract Syntax Tree (AST) of its input and transforms it into a sequence of chained subqueries (similar to Static Single-Assignment form). While the generated SQL is sometimes more verbose than hand-written SQL, the queries are straightforward for database query planners to optimize out. This means that the generated queries are just as efficient as the equivalent, more condensed queries.

pql can be used with nearly any SQL-based database. If you’re interested in learning more, check out pql.dev and the announcement on RunReveal’s blog. pql is packaged as a Go library and small CLI that’s available on GitHub under an Apache 2.0 license.

If you're looking for someone to help with a complex software problem like this for your business, I'm available for consulting and contract work! Read about how I can help you and contact me at consulting@zombiezen.com.

rsync is a wonderfully useful utility for copying files both locally and between machines. It makes a reasonable effort to copy only files that have changed, which can save a tremendous amount of time in many scenarios. However, I’ve always been a bit daunted by its plethora of command-line options, and would always resort to its man page for every invocation. Thus, it would never stick in my set of tools I would commonly use. After doing some work where rsync really was the only tool for the job, I decided to finally make a little cheatsheet on how to use rsync.

I was on the Cup O’ Go podcast this last week! I talked about my SQLite package and the history behind it. Go check it out!

I write a lot of shell scripts. Many are one-offs or specific to a project, but every so often, I’ll have a script that transcends to become a part of my toolbelt. For example, nix-op-key is a script I wrote to generate new Nix signing keys and place them in 1Password. It’s not a task that requires a dedicated program, it just needs to glue two existing programs together: nix key generate-secret and op (the 1Password CLI). These sorts of scripts are great, but if you want to share them with someone else (or even just use it on a different computer), how do you do it? Scripts like these depend on specific programs (or maybe even specific versions) being installed and Bash does not have a package manager like pip or the go tool.

As it turns out, Nix is such a package manager. And with flakes, there’s built-in support for installing and running scripts with well-specified dependencies in a single command. For example, you can run my nix-op-key script I mentioned earlier (pinned to a specific version) with:

COMMIT=25e9bd52e977cca415df84ea91028efd92d3da92
nix run "github:zombiezen/dotfiles?dir=nix&rev=$COMMIT#nix-op-key" -- --help

Furthermore, you can install the script using the nix profile install command:

nix profile install "github:zombiezen/dotfiles?dir=nix&rev=$COMMIT#nix-op-key"

(If you try this out yourself, you can uninstall the script with nix profile remove '.*.nix-op-key').

In this blog post, I’ll show you how you can package your own shell scripts with Nix to make them more reliable and easier to share. This article assumes familiarity with Unix command line and Bash shell scripting. I’m using Nix 2.17.1. All of the source code in this post is released under the Unlicense and is available on GitHub.