Similar to the Jupyter kernel for Ivy I hacked up a few months ago, I’ve created a SQLite kernel for Jupyter Notebook.
Screenshot of a SQLite Visual Studio Code notebook.
Check out the demo on GitHub! (Installation is a little DIY if you’re not using Nix, but there are instructions in the README on how to build.) Read on if you’re interested in how it works.
I hacked out a Jupyter kernel for Rob Pike’s ivy language, an APL-like desk calculator. This allows using Ivy expressions inside a Jupyter notebook and inside Visual Studio Code’s notebooks feature. It’s got rough edges, but it’s good enough for me to quickly do calculations without leaving Visual Studio Code.
Screenshot of a Jupyter notebook with the Ivy demo.
The source is up on GitHub.
(Installation is a little DIY if you’re not using Nix,
but it should go build if you have libzmq installed locally.)
Today, I released a small library called postgrestest. It spins up an
ephemeral PostgreSQL database in Go. I’ve found it quite useful for writing
tests that use PostgreSQL while keeping the test hermetic and reasonably
fast. In my benchmarks, starting a server takes roughly 650 milliseconds and
creating a database takes roughly 20 milliseconds — a 70% improvement and
90% improvement, respectively, over a postgres Docker container in the default
configuration.
You can install it with:
go get zombiezen.com/go/postgrestest
Since I’ve started working at Clutter, I’ve grown to enjoy the Apollo React Client library. However, I’ve noticed that the client’s caching behavior can be difficult to understand and cause surprising issues, so I’ve decided to collect my findings into one easily digestible post. This post assumes a basic understanding of GraphQL.
The major hidden component of Apollo Client is the cache, which sits between every query and the backend server.
I’m excited to share my hobby project from the last few months: a GraphQL server library for Go. While it doesn’t implement the whole spec and might not be ready for production, I think it’s in a useful enough state to share more broadly. I’ve tested this library out with some toy projects and the results have been promising: I’m able to quickly publish a Go struct as a GraphQL endpoint.
Check it out now by reading the docs and adding it to your project with:
go get zombiezen.com/go/graphql-server/graphql
It’s always interesting to me to compare different approaches to solving the same problem. Git and Mercurial are two version control systems that came out at similar times, trying to address very similar requirements. Git came from a very low-level systems perspective, whereas Mercurial spent a lot of effort on its user experience. Despite what you might think, their data models are remarkably similar. It’s from this observation I started my side project — gg. I found myself greatly missing the experience of Mercurial, but I’ve resigned myself to the fact that Git is here to stay.
I came across a rather interesting challenge today while working on gg. I am
trying to replicate the behavior of hg pull, and even though I’ve worked on gg
for over a year now, I still haven’t reached a behavior that I’m satisfied with.
I’ve finally realized why, and it boils down to a very subtle difference in the
data models of the two systems.
Frequently when people discuss what is a “good” Go library, they usually use terms like “idiomatic” or “the Go way”. These terms are imprecise and make it difficult to discuss the merits of different API designs. I recently re-read Don Norman’s The Design of Everyday Things and realized that the same principles of design discussed in the book can be used to evaluate the design of APIs.
This report details an experience I had while writing an RPC system in Go. While Go’s standard I/O libraries make a great many things simple, I found cancellation to be more complex than I would have liked. Parts of this situation have improved in the last couple of Go releases (as I have noted below). I hope this positive trend continues in a way that allows the Go ecosystem to easily propagate cancellation, deadlines, and request values. My intent in this report — as well as the proposal I created back in May 2017 — is to give background and feedback to inform future design decisions. Suggestions for solutions welcome!
(Thanks to Ian Lance Taylor, Damien Neil, Cassandra Salisbury, and Andrew Bonventre for reviewing this report for accuracy and clarity.)
I’ve posted a project I’ve been hacking on to my Bitbucket account: cardcpx. This has nothing to do with my day job, it’s just something I did to help out on a film shoot (which also had nothing to do with my day job).
From the README:
cardcpx is a specialized UI for copying files from a camera card to 1+ replicas. The replica copies happen concurrently, so if you are copying N bytes to R replicas, the time is O(N) instead of O(N * R).
The interface also has includes simple scene/take ingestion, which is stored in a CSV transaction log. Selects will be copied first, so you can do a proofing check on a fast disk while your import finishes.
cardcpx supports a flat directory structure as well as the RED camera directory structure. It assumes that your clip names do not overlap. Attempting to copy the same file name will not overwrite data.
It’s still a bit rough around the edges, but I’m happy with how Go allowed me to compose a fairly complex a tool out of easy-to-understand pieces. This was my first excursion into writing a frontend with AngularJS, but I’m also similarly pleased about the design.
Everything’s open source. Feel free to dig in and use. I’d love to know if other people find this program useful.
This is the second part in my series about creating a low-cost programming environment with an HP Chromebook 11. See Part 1 of Chromebook Hacking for the overview. This blog post assumes you have some basic knowledge of using a Linux terminal.
Danger: This will wipe out your entire local storage! Back up everything!
Enabling developer mode on a Chromebook is the blessed way of “rooting” your Chromebook: it gives you root shell access. This comes with the cost of losing parts of ChromeOS that depend on hardware-backed security systems (namely Netflix. Sad days).
Enabling developer mode is reversible (see below), but going in either direction will wipe your Chromebook’s SSD. Please back up your files first.
Instructions are on the Chromium OS website. For the HP Chromebook 11, the short version is: hold Esc+Refresh+Power button and when you see the boot screen, tap Ctrl+D. Follow the prompts and wait for roughly 10 minutes for the operating system to reinstall.
On the boot screen, you need to press Ctrl+D every time you boot to skip the “Danger, you don’t have verified boot” screen, or you can wait for 30 seconds. You can alternatively press the spacebar to perform a factory reset and disable developer mode.
Go through the standard Chrome OS “create an account” process to get to the desktop.
This step isn’t strictly necessary, but it gives you a very slick terminal interface. If you decide not to do this, you can always open crosh with Ctrl+Alt+T.
Install Secure Shell from the Chrome Web Store. Right-click on the app’s icon in the shelf and choose “Open as Window”. This allows you to use shortcuts like Ctrl+W inside the terminal without Chrome intercepting them. Click the shelf icon to open a new terminal. Create a new profile with a random non-empty user name (I picked foo) and a host of >crosh.
Open a bash shell by running shell at the crosh prompt. Inside your bash shell, run this command:
chromeos-setdevpasswd
This sets your Chromebook’s UNIX password for the chronos user — the UNIX user used for any logged in Chrome profile. Now that you’ve enabled developer mode, chronos has the ability to run sudo. Since all Chrome profiles use the same UNIX user account, you will want to open chrome://settings/accounts and whitelist the users that can access this machine. Disable Guest browsing and restrict sign-in to yourself.
Finally, you should prevent your Chromebook from booting anything other than ChromeOS by running the following from your bash prompt:
crossystem dev_boot_usb=0 dev_boot_signed_only=1
The first parameter (dev_boot_usb) disables booting from an external USB stick. The second parameter (dev_boot_signed_only) forbids booting to operating systems that are not signed by Google.
Now you have a minimal Linux environment with bash and vim. After poking around a bit, you will quickly notice that only Downloads and /usr/local are writable. In the next blog post, I will introduce Crouton: a less restrictive Linux environment for ChromeOS.
In my spare time, I’ve been working on Go-SDL2 and Camlistore. I’m usually not sitting in front of my home desktop, so I want to be able to code anywhere. My battle-worn MacBook Pro could do the job, but it’s heavy, bulky, and aging.
To fix this, I bought the HP Chromebook 11. The display is nice, the price is great, and the keyboard is comfortable. It accepts a microUSB charge, so I don’t even have to worry about packing an extra charger. The bummer is development: good luck trying to run anything besides JSFiddle.
For a while, I was using SSH into my desktop to code. Most of the time this works. The problem was me: some days I would forget to turn on my desktop, other days I left it in Windows, and every day I gritted my teeth if I was coding graphics. I fantasized about a machine that booted quickly and had all the programs I needed.
Turns out, the answer was right in front of me: the Chromebook.
Total Cost: $279 + half a day’s worth of work.
Over the next few days, I’ll go into more detail on these steps.
I discovered that some of my code was still living on my old laptop from high school. The laptop hadn’t been booted in about 4 years, so the clock battery had died, causing the filesystem checks to fail (this is Ubuntu 9.10, which cared about these things). I reset the clock manually, boot up the machine, and decide that to be on the safe (but marginally inefficient) side, I’ll copy my entire multi-GB home directory to my more beefy desktop and filter out what I don’t want there.
Sadly, this laptop came before 802.11n and is limited to 54 Mbps. I have an AirPort Express with only one local port, which is connected to my desktop, but I don’t want to disrupt the network setup. I realize I can chain the Ethernet from my old laptop (named metroid) to my new laptop (named roran), which has 802.11n to connect to my desktop (named nasuada).
ross@nasuada$ nc -l 8080 > mybackup.tar.bz2
ross@roran$ nc -l 8080 | nc nasuada.local 8080
ross@metroid$ tar -jcf - $HOME | nc roran.local 8080
I love Unix.
(This developed from a thread with some other Cal Poly students discussing Go.)
One of the frequently asked questions about Go is why are there no generics in the language. It’s a fair point from the perspective of other OOP languages, but idiomatic Go code un-asks the question.
First, it is important to understand how interfaces work in Go. The gist is that interfaces use virtual method lookups, but retain the underlying type, so a conversion back to the original type is still type-safe. An empty interface (interface{}) works as a type-safe equivalent of C’s void*: you can put any type inside of an empty-interface variable, and then you can un-box it later.
The simplest way to define a linked list is through the empty interface: interface{}.
type Node struct {
Value interface{}
Next *Node
}
And indeed, this is how the container/list package package works, give or take some complexity. However, my argument is that this is the wrong way to do this. It requires an explicit boxing/un-boxing hit for every access, and this is where (I think) most of the call for generics comes from. There’s a much more idiomatic way to do this. My two case studies here are the sort package and the container/heap package from the standard library.
We start with sort, which defines an interface, sort.Interface:
package sort
func Sort(data Interface)
type Interface interface {
// Len is the number of elements in the collection.
Len() int
// Less returns whether the element with index i should sort
// before the element with index j.
Less(i, j int) bool
// Swap swaps the elements with indexes i and j.
Swap(i, j int)
}
This may not strike you as a generic container, but that is precisely the point. You can – with a bit of boilerplate – create a type that wraps a typed slice that implements sort.Interface. Thus, the algorithm can work on any type and only works uses indices. This is much cheaper than boxing/unboxing on access.
The second example is container/heap:
package heap
func Init(h Interface)
func Pop(h Interface) interface{}
func Push(h Interface, x interface{})
func Remove(h Interface, i int) interface{}
type Interface interface {
sort.Interface
Push(x interface{}) // add x as element Len()
Pop() interface{} // remove and return element Len() - 1.
}
Admittedly, there is a bit more boxing/un-boxing going on here, but the idea is still similar. We now have a data structure that requires very few box/unbox operations (two per push/pop). It is important to note that this is still type-safe.
You could implement a binary tree structure similarly:
package tree
func Search(root *Node, val interface{}) *Node
type Node struct {
Index int
Left, Right *Node
}
type Interface interface {
Len() int
Less(i, j int) bool
Push(x interface{})
Pop() interface{}
}
The Search function would push the val interface and use it for comparisons against the values stored in the tree, then pop it once finished. Admittedly, this isn’t the prettiest interface, but it works in a pinch. It works out to be only one box/un-box round-trip.
As we’ve seen, by “flipping” around the algorithms and separating them from the data structures using interfaces, we can still reap the benefits of generic containers from other languages.
I’m finishing up my game, “Urge to Merge”, for my Intro to Computer Graphics course. This is the first time I’ve worked with OpenGL shaders; I’m pleased with the results. These are screenshots from the game.
I’ve moved goray from SourceForge to BitBucket yesterday. This also signals a switch from Bazaar to Mercurial. Along with other changes in the goray project, I think this will make goray easier to work with and contribute to. Read on for more details.
Yesterday, the Go programming language announced their second stable release, introducing language changes, better packages, and general speed-ups. I’m proud to say that I helped out in a small part by contributing a patch to the zlib package.
For those of you who haven’t heard me talk about Go, it’s a programming language that brings all of the pleasant features from scripting languages and mixes it with the simplicity of C. Its syntax and feel is unlike any other language, but after a week, it begins to feel like second nature, and it’s now one of the first languages I grab to solve a problem. I can’t say enough good things about it, and if you haven’t already, download it and try the tutorial. (And yes, it will feel weird at first. Give it a week if you’re coming in fresh.)
I’m also excited that the next weekly will include my implementation of the SPDY protocol — Google’s improvement to the HTTP protocol — as one of the standard packages! The Go development team is exceptionally friendly and open to contributions, and I must say, this is the most fun open source project I’ve worked on. If you want a fun open source project to help out with, start hanging out on the go-nuts mailing list.
Congratulations to Russ and everyone involved on a successful release!
It’s been a while since I’ve posted because of some events that required my attention, but I’m back with yet another goray rendered image! A lot has changed in the rendering infrastructure (refactoring and such), and now it supports spot lights and texture mapping! Texture mapping and Blender export have not been merged to the trunk yet, but they’re coming soon.
The other exciting news is, if you’re bold enough, you can play with goray, too! Check out the Sourceforge project!
Currently expanding the FIRST scouting project to generate PDFs with barcode IDs. Data entry will be epic.
After fixing my ray/triangle intersection code, I’ve moved to working on the material system for my raytracer. I’m starting on the simple material first: Lambert shading with recursive raytracing for reflection.
For those who don’t know that much about raytracing, the Wikipedia article is surprisingly informative. The cool part is that the same code that you use to compute the normal scene just gets called recursively to create the reflection. It also makes physics class fun when you can readily see that just by applying the Law of Reflection, you can make cool effects. Next stop, Snell’s law!
When you have a bug in an enterprise program, money is lost. When you have a bug in a graphics program, art is created.
I’ve been working on rewriting parts of the raytracer I wrote in Go in x86-64 assembly. The test model I’ve been using is Suzanne, one of the primitives in Blender. Needless to say, something in the assembly code is not right, but it’s the first actual image the raytracer has spit back since I removed the pure Go implementation of the ray/triangle intersection code.
I’m currently working on a FIRST scouting data management project in my spare time. The requirements I’m working with are fairly well-defined, but the statistics collected change from year to year. I want to be able to use the same software year after year and keep historical data intact. I’m using Django for the front-end, but instead of using a traditional SQL database, I’m experimenting with using MongoDB, one of those new, fancy No-SQL databases.
I used this code in some systems programming work. It actually does something useful.
Stop using autocomplete. People no longer think about what they say, what they type, and how they spell it. Especially for programming, autocomplete is a sin.
I wanted to paste a text file with my homework assignment into Gmail. Unfortunately, said text file was on the school server and I really didn’t want to permanently download the thing to my hard drive. So this is what I did:
This connects to the school server, retrieves graph.txt from my home directory, and copies it to standard out. I then piped the text into pbcopy, which is the Mac utility for manipulating the clipboard. After that, I switched to Gmail and did a paste.
Unix/bash is cool!
I find this rather amusing. If you write Hello, World in Python in the sand, does that make it sand-script? :P
Now I’ve got an 0xE to Valgrind.
Some days, you just have to use grep.
This is a 30-minute documentary about the patent process and how it applies to software. It’s a remarkably complicated and important issue, and it does a good job of explaining it understandable terms. Even if you’re not into computer science, this is an important issue that affects many aspects of technological progress.
I just started using this a couple days ago. It’s essentially a CSS preprocessor that allows you to use variables, arithmetic, functions, and true inclusion. It’s also very good at compressing CSS for you. Zombie Zen is now using it; it shortened my development time significantly.
I’m reading a book about the vi editor right now. For those less UNIX-inclined, vi is a command-line based text editor for UNIX systems like Linux, FreeBSD, and Mac OS X. It’s installed on basically every system (in some flavor or another. I prefer strawberry Vim, but I’ve been told the cinnamon variety deserves a second glance) and is remarkably easy to use.
It’s also more powerful than I realized.
The problem is that you need to find a good guide to vi. Its built-in help is wonderful reference, but it can’t get you up and running. For years, I’ve been bound to only a fraction of the commands. No more. Why would I use anything else?
[Cue angry Emacs users]
If my readers are so inclined, I may start posting some vi tips.
A DOS prompt makes me sad. A bash prompt elicits unbridled joy in my heart.
The above quote came out of a rather interesting discussion in my early morning programming lecture.
And as stated above, this is my 100th post! I’d like to thank my kind readers and followers for making Zombie Zen a wonderful experience. To celebrate the occasion, I thought I’d send along a picture!
HOPE YOU LIKE FINGER CONTORTIONS. :P
After spending the entire day on it, it’s amazing how joyful it is to get a program to accept keyboard input.
After initially posting about Google’s programming language, and working with it for about three weeks, I think I’m ready to give it a review.
Just a quick overview of Go: type names on the right side, language-level support for concurrent processing, and objects are interface-based. If you’re interested, go check out http://golang.org/.
First of all, I applaud Google for coming up with a language that actually brings something new to the table, instead of the usual “You use this syntax to do the exact same thing you could have done more cleanly in Python!” Go has actually helped me make programs that are more multiprocessing-aware, and that’s something that’s pretty amazing.
I think my main problem with it right now is that the language is fluctuating so much that it’s hard to keep up with. I’ve met some unstable code bases before, but this breaks my code every three hours if I stay on the bleeding edge. Heaven forbid if I don’t update for three days, I have to recompile everything. It’s just ridiculous.
Yes, I know that Go is weeks old. I know that there isn’t an infrastructure for formalizing language changes yet. I further know that the poor developers are getting beaned over the head by trolls on the Google group who just want to whine about how different the language is, without even coding in it in some cases. Darn you, Internet trolls. I like Go’s potential, but I think I’m going to wait a month before trying to do any serious work in it yet.
Definitely wait. I havent been keeping up with it, but from your description I imagine they prematurely released any compiler for the language, if this is not some sort of Alpha or Beta at all.
You’re right, it is my own fault. I broke Rule #42 of Code-land: Any code that doesn’t have a version number is trouble. :)
After initially posting about Google’s programming language, and working with it for about three weeks, I think I’m ready to give it a review.
Dude. Google built a programming language.
I find it ironic that my prototype solver (written in Python) is less buggy than my production solver (written in Java).
I’m finally installing Snow Leopard on my family’s Mac Pro. I’ll keep you posted.
This is my trivia game that I implemented. I still think this is going to catch on soon.