Notational Programming

My historical paper grounding this project, "To Write Code: The Cultural Fabrication of Programming Notation and Practice," earned a Best Paper Honorable Mention award at CHI 2020. Download the paper here.

Notational programming is a long-term project to incorporate handwritten inputs into programming practices, which today is done (for professionals) exclusively with the keyboard and mouse.

One can think of it like handwritten programming, but this can be slightly misleading as it might conjure an image of, say, handwriting Python. Programming practice, at the time of its emergence, was informed by the constraint of typewriting interfaces. To understand exactly how early visions constrained later imaginations, I have situated the dominant 'general-purpose' paradigm as socially and culturally constituted, rather than an inevitable, or universal, interface to computation. Moreover, I show how programming practice arose/arises from friction and conflict between cultural practices.

Case Studies:

"Part 2" of this project, "Notational Programming for Notebook Environments," earned a Best Paper Honorable Mention award at UIST 2022. This important paper explores how users interact with a programming system that blends handwritten and typewritten notation. Preprint available here.

Notate is a publicly-available Jupyter notebook extension that allows you to insert drawing canvases into in-line code. Canvases are treated as NumPy images (2d nparrays) in Python code. In the Jupyter interface, canvas data is automatically saved as metadata upon saving the notebook. You can also copy+paste as expected.

One domain that uses diagrams extensively is quantum computing. For instance, in quantum computing the diagram

A simple example of a hand-drawn quantum circuit definition.

defines two quantum gates, A and B, where B is defined in terms of A. Notate and a custom-built recognizer parses this diagram. By embedding these drawings inside the Jupyter programming environment, they can inter-operate with textual code. This works for basic drawings (the below image is from a working prototype and is not a mock-up):

A circuit drawn inside a Jupyter notebook cell and recognized through a function call.

Here there is a basic circuit drawn in an image that is in-line as "code." The image is fed to a quantum circuit recognizer which outputs an IBM Qiskit circuit data structure.