Literate programming in Go | InfoWorld

During a recent company hackathon, my team’s charter was to improve the documentation for the Steampipe plugin SDK. Like other components of the Steampipe system, the plugin SDK is written in Go and published to pkg.go.dev. The version that existed when we started is here. As is typical, the documentation was an autogenerated catalog of functions and types. To explain how to use those functions and types, we provided guidance on the steampipe.io site.

Our hackathon challenge was to weave such guidance into the generated documentation. If you’re writing a Steampipe plugin, your list of tasks looks like this:

1. Define the plugin
2. Create the plugin entry point
3. Define your first table
4. …

These tasks require that you use functions and types, but while comments attached to those functions and types can enhance the generated documentation, they’re too granular for the high-level exposition we aimed for. Searching for inspiration, Steampipe lead developer Kai Daguerre noticed that our top-level page lacked the overview section he saw in the documentation for pgx, a Go driver for Postgres.

That overview comes from https://github.com/jackc/pgx/blob/master/doc.go, which is one long comment (that uses Go comment syntax) followed by a package declaration.

So we added a doc.go at the top level of our repo to produce this overview.

IDG

Writing a wiki in Go comments

We used the name doc.go because that seems to be conventional, but it could have been called foo.go. What’s salient is that it’s a valid Go file that belongs to a package. The package contains no code, only documentation. We wrote headers to create the sections of the overview, and in each section we described a plugin writer’s task using narrative, inline code examples, internal links to functions and types, and external links to examples elsewhere.

The ability to link within the code’s namespace was a revelation. To describe the task called Add hydrate functions, for example, we wrote this:

# Add hydrate functions

A column may be populated by a List or Get call. If a column requires data not provide by List 
or Get, it may define a [plugin.HydrateFunc] that makes an additional API call for each row.
Add a hydrate function for a column by setting [plugin.Column.Hydrate].

The section defined by the Add hydrate functions header is a link target: Add hydrate functions. And the bracketed items render as links to a type, plugin.HydrateFunc, and to a property, plugin.Column.Hydrate. This was starting to feel like wiki writing!

What we still lacked, though, was the ability to create new wiki pages where we could explain higher-level concepts. The overview was one place to do that, but we wanted to reserve that for narration of the plugin writer’s journey. Where could we discuss a concept like dynamic tables, an advanced feature that enables plugins like CSV which have no fixed schema and must define columns on the fly?

Kai realized that not only could we create new documentation-only packages for such topics, we could also import them so that their names were available for the same kind of shorthand linking we could do with, e.g., [plugin.HydrateFunc]. In the top-level doc.go he did this:

package steampipe_plugin_sdk

import (
	"github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables"
)

var forceImportDynamicPlugin dynamic_tables.ForceImport

And in /docs/dynamic_tables/doc.go he did this:

package dynamic_tables

type ForceImport string

Now we could write a sentence in a comment like this:

/*
Use [dynamic_tables] when you cannot know a table's schema in advance.
*/
package steampipe-plugin-sdk

And the bracketed term autolinks just like any other name in the code’s namespace.

If that seems like more trouble than it’s worth, you can skip the import gymnastics and just use an external link:

/*
Use [dynamic_tables] when you cannot know a table's schema in advance.


[dynamic_tables]: https://pkg.go.dev/github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables
*/
package steampipe-plugin-sdk

Either way, the authoring experience now feels more fully wiki-like. You can create new pages as needed, and weave them into the hypertextual documentation that lives within the code base.

It was, admittedly, a bit of a struggle to use the Go system in the ways described here. The comment syntax is Markdown-like but frustratingly not Markdown; it depends on many implicit formatting conventions. If you go this route you’ll need a local previewing tool, and it’s not super-obvious that the one you want is not godoc but rather pkgsite. Had we discovered the command go install golang.org/x/pkgsite/cmd/pkgsite@latest we would have saved ourselves a lot of grief.

How is this literate programming?

It isn’t! In his eponymous paper Knuth wrote:

The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of style. Such an author, with thesaurus in hand, chooses the names of variables carefully and explains what each variable means. He or she strives for a program that is comprehensible because its concepts have been introduced in an order that is best for human understanding, using a mixture of formal and informal methods that reinforce each other.

To support this practice he invented a system called web whose components, tangle and weave, enabled the author of a program to tell its story in a language that mixed code (originally, Pascal) and documentation (originally, TeX) in a narrative-first way. Our modern ways of mixing code and documentation, and generating docs from embedded comments, only superficially resemble Knuth’s practice, as Mark Jason Dominus pointed out in his essay POD is not Literate Programming (2000).

And yet, now that our hackathon exercise has given me a taste of what code-as-wiki can be, it does feel like a step toward the storytelling approach that Knuth advocates. Recall that he named his system web before there was the web we now inhabit, never mind wikis! I can’t claim that we’re now literate programmers in the Knuthian sense, and I’ve always wondered if only a Knuthian mind could implement that original vision. But this way of improving the narrative quality of autogenerated docs does feel like a step in the right direction.