Similar to the Jupyter kernel for Ivy I hacked up a few months ago, I’ve created a SQLite kernel for Jupyter Notebook.

Check out the demo on GitHub! (Installation is a little DIY if you’re not using Nix, but there are instructions in the README on how to build.) Read on if you’re interested in how it works.

I implemented this kernel in Rust because I’m primarily interoperating with C libraries, and I wanted to avoid the sleep/poll loops I had in the Ivy kernel. I ended up writing low-level Rust wrappers for ZeroMQ, SQLite, and RE2 to make this all work. I think Rust was pretty well-suited to this task but I wish I could use the C more directly with the same safety helpers.

There was some prior art in the form of xeus-sqlite, originally written by Mariana Meireles. When I started work on this project in late August/early September, the project hadn’t seen much activity and seemed out-of-date. It seems that there’s been some new activity as of November, but regardless my kernel’s design goals are different. My aim is to have my kernel be mostly compatible with the SQLite command-line shell including its special “dot” commands. This allows scripts written for the SQLite CLI to be reused verbatim in the notebook.

I’ve been happily using this kernel to debug live SQLite databases and do simple data analysis tasks (including JSON manipulation) in a variety of situations. Because this kernel can access the filesystem to read and write data, I find it fitting into a lot more scripting tasks. I also really like the tabular HTML output, which was easy to implement.

Thanks for sticking around! If you find this project useful, consider supporting me on GitHub so I can continue working on open source tools like this.