BLURB

James Somers is Jane Street’s writer-in-residence, splitting his time between English and OCaml, and helping to push forward all sorts of efforts around knowledge-sharing at Jane Street. In this episode, James and Ron talk about the role of technical writing in an organization like Jane Street, and how engineering software relates to editing prose.

SUMMARY

James Somers is Jane Street’s writer-in-residence, splitting his time between English and OCaml, and helping to push forward all sorts of efforts around knowledge-sharing at Jane Street. In this episode, James and Ron talk about the role of technical writing in an organization like Jane Street, and how engineering software relates to editing prose.

• mdx, the modified Markdown format that supports executing OCaml code blocks
• More on the 4 types of technical writing that James references
• Donald Knuth’s original book on Literate Programming
• Knuth’s TeX typesetting system
• More on John McPhee’s use of KEDIT
• Peter Seibel’s Coders at Work
• David Goodsell’s The Machinery of Life
• Scott Huler’s Defining the Wind
• Some of James’s writing on our tech blog

TRANSCRIPT

00:00:04

Ron

Welcome to Signals and Threads, in-depth conversations about every layer of the tech stack, from Jane Street. I’m Ron Minsky. A note about audio quality: We tried out some new equipment this episode, and it didn’t work out quite as well as we’d hoped. That’s said, we think the interview came out pretty well, audio issues notwithstanding. So, without further ado.

00:00:26

Ron

Today, we’re going to have a conversation about documentation and writing in the context of software and software engineering. And I am very pleased to have with me James Somers. James has experience both as a professional writer and a professional software engineer. And he’s worked here at Jane Street for about three and a half years. James, maybe to get started, you can just describe what your job here is.

00:00:47

James

Well, I’m making it up as I go, I think. But I would say I spend most of my time writing code actually, probably about 75% of my time. And the rest I sort of am a writer or editor in residence. So just sort of helping people write documentation, edit it, think about it, and doing all of that as it comes in.

00:01:07

Ron

And why does that seem important to you? What is the thing that you’re trying to achieve at Jane Street for the organization?

00:01:14

James

Yeah. It helps to know what it’s actually like to build software here. So I think being a programmer as the first thing that I am, and then being a writer or editor is the second thing, is going to really help all of the documentation that I look at. So for instance, you’ve got to know the language. OCaml is a strange language that I didn’t know beforehand. You have to know our libraries, we produce basically our own standard library for OCaml. We build a lot of our own development tools in-house. We have a testing framework and a build system. We have our own unique way that we deploy applications, monitor them, log them. And I think just having empathy for a regular developer here requires you to be a regular developer here because it’s kind of unusual. There’s not really an experience outside that can prepare you exactly for this ecosystem, which is very much our own.

00:02:05

Ron

So I think what you’re talking about there is ways in which, in order to be an effective writer here, you need to understand the environment. And maybe that’s just a thing you need to be an effective writer in any context is to understand the thing about which you’re writing, right? Writing and thinking are deeply intertwined. And if you don’t understand the context, it’s hard to do an effective job of writing. But I was getting at a different question, which is, what do you think of as the purpose of this role of writer in residence? It’s clear why Jane Street needs software engineers, we have a big and complicated business, we need lots of custom software to execute on that business. But what do you think of as the role of someone who has a kind of special focus on writing in a place like this?

00:02:44

James

That’s a good question. I think if you don’t have somebody who focuses on that, your documentation will likely go the way of all documentation everywhere, which is that it’s a bit of a backwater. It becomes a bit neglected, in part because it’s very hard to do well, in part because it’s not as exciting as building a new thing. And I think people are not necessarily natural writers, and they’re not necessarily natural technical writers. I mean, if you think about when people are writing documentation, it’s usually right after they’ve built some piece of software, right after they’ve written a library, say. It’s almost the worst moment possible to write about the thing because you’re so immersed in it. And in particular, you’re immersed in all the implementation details because that’s what took all of your time. And so you’re not really thinking about it the way that an outsider would think about it, and yet you’re the one writing it. And so I’m sort of a perpetual outsider. I think that’s my job, is just to be an outsider all the time, to be the dumbest person who can understand what I’m writing about. And that is usually going to be the level of comprehension that the reader is at. If they’re not as dumb as me, they’re at least less patient or more pressed for time. And so they need the slightly more empathetic version of the document.

00:04:10

Ron

So it seems like you come in and engage with Jane Street’s culture around writing and teaching and generating documentation in a bunch of different ways. I’m curious what you think of as the good sides and bad sides of the culture that you see around this kind of work. In what ways is Jane Street good at generating documentations? And what do you see as essentially problems, things that people need to learn more about, need to get better at?

00:04:34

James

Yeah. I think people are very conscientious and very aware in general, that documentation is an important problem we’re solving. It’s not really an afterthought here the way that it can be in other places. And I think there’s a great amount of tooling and energy that goes into keeping documentation up-to-date. But I mean, we still have all the same problems that anybody else does. And I think part of the issue with our culture is that we have a culture of heavily associating software with the specific people who build and maintain it. I mean, we’re not a giant company, we’re relatively big but not giant. I remember the weirdest thing about the company when I joined was that I get to my desk and there’s a telephone on it. And I remember thinking, like, “Why on earth would you call a programmer?” But you call a programmer because here, unlike maybe if you’re working in another language, the answer to some question is not on the internet, it’s in the room somewhere or in the office or in one of the other offices.

00:05:40

James

And so you pick up the phone and you talk to the specific person who owns a piece of code. Our code review system even bakes ownership into it. So people own files, they have an obligation which is clearly defined. And I think there’s many, many benefits to that environment, but it does mean that a good document could just be a list of people who are associated with a project. A lot of people would find that document pretty useful, because those are the people you could pick up the phone and talk to.

00:06:10

Ron

And actually, that document, in some sense, exists, you can navigate to any part of the tree and go to the tools and say, “Who are the people who review this piece of code?” Who are typically the people who are most familiar with it. So you can go and discover who is it that owns any particular file in the tree.

00:06:25

James

Right, right. We haven’t yet hooked it up to the phone system, but really it wouldn’t be that hard to do. But, you know, the downside of that is that you’re not leaving a paper trail for other people. You’re not leaving marks of the conversation that you had over the phone. And those searchable trails turn out to be incredibly useful, especially for new people, maybe people who haven’t been around to have those conversations for many years. So the writing is, to say a very obvious thing, writing is humanity’s invention for storing knowledge over time. And I think that fact can sometimes be forgotten in an organization that produces software. But write stuff down and it’s useful forever after.

00:07:10

Ron

Right. And writing is a way of achieving scale, of distributing knowledge at scale. And you’re talking about the culture as it is and how you’ve seen it for the last three years. I’ve been here for almost 20 years at this point, and when I started, there were about 40 people, now there’s about 1500. So the organization has grown massively, but I think objectively you’re obviously right, in the sense that you compare it to a really large tech organization, we’ve got a couple of orders of magnitude to get there. But this thing you point out of it being very human centric, there are people who are deep experts, I think some of that comes from just being a place where we build technology that’s fairly unique, there isn’t as much outside information on it, you can’t go and look up on Stack Overflow to understand lots of the details, sometimes because it’s about obscure finance stuff and sometimes because of the programming language. And also because it’s just an organization with very low turnover, there are just lots of people who’ve been here a really long time and have become deep wells of institutional knowledge.

00:08:06

Ron

But even so, over the time that I’ve been here, I feel like I’ve seen a real shift where it’s become more and more important to write things down. And from my perspective, that’s why at some point I thought, “Oh, we have to hire someone to focus on writing.” Because I felt like it was a place where we were historically pretty weak and needed to get better. And a thing I’ve always been very aware of is, writing’s really hard, and lots of people aren’t comfortable doing it. And so having someone who feels like they can spend the time really helping people to write better seemed really important.

00:08:37

James

Hear, hear.

00:08:38

Ron

In some ways, it actually feels parallel to me to the transformation that the organization went through around testing. Early on, when we built our first trading systems, testing was a lot thinner than now I would care to dwell on. We relied a lot more on the fact that there was a very small code base, very well understood by a small set of people who were thinking about it very deeply, and relying on things like the protections from OCaml’s type system to help you catch lots of kinds of errors. And at a certain scale, I think that worked okay. And then over time, it became more and more awkward that we didn’t have really good testing practices. And then at some point, there was a kind of revolution where we started thinking much more deeply about testing, did a lot more of building out testing tools. And now, it’s just a pervasive thing everywhere that people are constantly thinking about, not just how to write their code, but how to write their code in a way that’s easy to test. And how to build tools that make that testing as lightweight as possible. And I still feel like we are not as far down the line of making that same transformation around documentation as I’d like us to be.

00:09:42

James

Yeah. Well, I think one difference is that in some ways, testing is a single player game and documentation is a multiplayer game. So when you improve your testing story, you make your development life a million times easier as an individual. I was just working on something earlier today, I spent the last two days just building a test harness for this thing, because once I have that test harness, the development becomes infinitely easier. And I’m going to be able to make new decisions about how this should work much more confidently. When you try to adopt something like documentation-driven development, let’s say, you’re mostly making your life worse at first and not necessarily is this ever going to pay off for you personally because you’re not really the audience for the documentation. Of course you might come back to something that you haven’t worked on in a long time. And it’s very helpful to have good documentation there.

00:10:36

James

But really what you’re working on when you’re writing a piece of documentation is kind of other-focused, in a way that a test is giving you a super power for yourself. So I don’t know if that gap can ever be fully bridged because I don’t know if you’re going to have the same “aha” moment where a developer realizes, “Oh my God, I’m going to be 20 times as productive if I write better documentation.” The organization will be 20 times as productive. So it’s sort of incumbent on the organization to think about this and have this “aha” moment, but it’s not going to translate into individuals, I don’t think.

00:11:10

Ron

Yeah. I mean, I think testing is halfway in between the simple task of writing code and the task of writing documentation, in the sense that you definitely get personal benefit from testing. But I feel like, especially in a large, complex project, the really great situation is where you and all of the people around you are pervasively testing the work. And then you can reach into parts of the code base that you haven’t previously touched before and make changes with much greater confidence because there’s a significant base of testing. And that kind of effect does happen in the context of documentation. If you walk into a code base that’s thoroughly and well documented, it’s just much easier to learn and to make changes and to understand what’s going on. It’s I think a smaller effect than testing, but some amount of empathy for other people can really help.

00:11:52

Ron

But for sure, some of the documentation you write is just for users. And there I guess part of what you want is some other kind of feedback, which actually brings me to another thing. So you spend a bunch of time working on writing, but also you said you spend the majority of your time writing code. And that code is all around documentation, but it’s on the tooling side of documentation. Can you talk a little bit about what kind of work you do there? And then, what kind of role you think tools play in the larger task of documenting and writing?

00:12:20

James

Yeah. So one of the things I work on is a tool for literate docs, where basically you have like a markdown file with code blocks in it, but those code blocks compile. And the benefit of having the code blocks compile is that they’re not going to go out of date. If they depend on some library and you change the API to that library, the documentation will fail. It’ll throw an error, just like any other code file. And that helps mitigate one of the biggest, if not the biggest problems with documentation, which is that it tends to rot. So this helps with documentation from going stale. It also means that docs are tightly integrated with our build system. A lot of the work that I do is just a consequence of that fact. But we have a lot of control over our documentation. So one of the things I work on, I’m sort of a web developer in a former life, is publishing our documentation that comes out of our build system to a website where people can read it. It seems fairly simple, but we try to make cool features on this website, like the ability to see how many people are looking at your document. Try to give people incentives to write better documentation, maybe by implementing something like likes, allowing people to comment, just sort of making a social piece of software.

00:13:34

James

The other thing I work on is an internal search engine. So basically, something that searches our wiki, these literate docs, API docs, internal blogs, mailing lists, even code review comments. Some of this is still in the works. And the benefit of doing that is obviously you need people to be able to find documentation. I mean one thing that you find as you do scale as an organization is that people might be producing incredibly good documents and just nobody’s finding them. And that actually happens a lot. It’s sort of like the plight of a tool developer is always that they need to make their tools discoverable. So I worked for a long time on the tools and compilers team at Jane Street, and these people were incredibly productive with the tools that they built, very often more productive than their external users, because the external users didn’t necessarily know about all the great features. So it’s incumbent on the documentation system to make things as discoverable as possible.

00:14:31

Ron

Right. So zooming out for a second, at a high level, search is about discoverability. Some of the features you talked about in the publishing system are about essentially social feedback, almost the gamification of documentation, making it so that when people generate documentation they can see the impact on the world and how useful it is and how often it’s referenced, even if they can’t see how it affects their own workflow, as we were talking about before. And then, the first one you talked about is about giving feedback between the software and the documentation itself, so that when the software changes, you get cues that tell you that things need to be updated. So I guess like three different aspects of how humans interact with the documentation, how you can push that forward to make people get more out of it and do more of the documentation work.

00:15:16

James

Yeah, I think to that last point, it sounds mechanical and boring. Just that, “Oh, the docs are going to break when your code changes.” But actually it’s just an excuse to go back and look at the docs, and to change them. And most people, I think, who encounter a build error in some documentation are going to have a moment where they rethink, “Does this make sense anymore?” I think people are actually pretty good about thinking about their documentation when it’s in sight. It’s just that it is so often out of sight, out of mind. A lot of what the tooling does is just remind you that the docs need to be kept up-to-date, and people are reading your documentation. That’s the social thing, it’s like just being reminded that this page is actually being read by people is a hell of an incentive to make it better.

00:15:56

Ron

And also knowing that pages aren’t being read by people. Knowing that you’ve spent a bunch of time creating some documentation and no one’s using it, that’s time that you can save next time if you figure out what’s worth spending your time on and what isn’t. It’s funny, you talked before about how documentation doesn’t so much help the developers who are doing the work, it helps the people who are using their software, and that’s why it’s sometimes left behind. I’ve in some ways had the opposite experience. So one of the things I spend a fair amount of time doing is reviewing other people’s code. And over the years, I’ve come to spend much more of my time thinking about essentially documentation. When I read a piece of code, the first thing I will look at is the doc comments that explain what the code is for. I’ll look at the high-level description of the change and try and understand that. I’ll look at the names of functions and variables, various identifiers in the code, to try and see if I can understand what the code is supposed to be doing, before I actually read the code that does anything.

00:16:54

Ron

And I think the most productive thing I do usually when code reviewing is sit down and write some documentation, or just ask the person who gave me the feature, either to add a little more explanation themselves or to explain it to me so I can sit down and write some documentation as I’m doing the review. And I feel like a thing that I very often discover when doing this is that this uncovers real conceptual mistakes, real bugs in the system. People I think sometimes think, “Well, no, I should just write my code so it’s really clear and then I don’t so much need to write the documentation.” And I think this is almost never true. In reality, when people sit down and think through documentation, they end up having a deeper and better understanding of the code that they’re writing themselves.

00:17:36

James

I couldn’t agree more. I mean, how many times have I written the change-set message, the commit message, and realized that I’m going on and on trying to explain away some terrible decision I’ve made. It’s that going on and on where I’m sort of starting to picture my reviewer reading this message and having to confront the reality of the monster that I’ve created, that I think, “Okay, actually this whole thing was a first draft.” And I have to go back and write the second draft. I think a thing you learn, obviously in any kind of creative work, maybe in writing it’s especially vivid, is that all writing is rewriting, is what my sophomore English teacher said to me.

00:18:18

Ron

Do you think of the process of writing English prose and the process of writing software as similar to each other? Do you feel like you’re using some of the same mental machinery as you do that? Or do you think of them as very different exercises?

00:18:30

James

I think in the micro they’re pretty different. And I would say in the micro software feels easier to write because there are fewer degrees of freedom. So there’s so many ways that a sentence can go wrong, that a paragraph can go wrong, that a full piece can go wrong. At least in the world of software, you’re sort of constrained by the library functions available to you, the thing exactly that you’re trying to do, the language. The language is a formally constrained language with a lot fewer building blocks than something like English. So I think in the micro it does feel pretty different. And one of the things I really like about this mix is that it’s not like I like writing more than programming or I like programming more than writing, and they’re each a break from the other. Writing is so cognitively, creatively painful in a way, sometimes, and programming is so delightful, it’s like playing a little video game. But you know, by the same token, coding can maybe sometimes feel small in a way that writing can feel big.

00:19:33

James

But the macro thing is very similar, in that you think that you’ve gotten it down, and you have not. And the better you get at writing, I think, is often about just having a tolerance, the way a distance runner has a tolerance for the miles, to the revision process. And that is exactly the same as a programmer. I think I’m a fairly naive programmer compared to some of the people we have around here. And the thing you notice them doing is just having this willingness to dive back in after they’ve supposedly finished something. And that willingness to dive back into the same damn piece of work over and over and over again, I mean, that’s the mark of quality, that’s what produces quality. And that is definitely the same between the two.

00:20:28

Ron

Yeah. In my head, I feel like they’re also similar, in both cases, I often want the end result to tell a story. I don’t want the code to tell a story because I on some artistic level want to tell a story, but in some sense, there’s some architectural, structural story that is inherent in the software. And I want that story to be written out as clearly for the person who’s reading it as is possible. And so when I think about how I organize the code into pieces, how I name things, how I write comments and documentation within there, part of what I want to do is present the structure of the system in a way that’s legible to someone, that they can sort of sit through and understand the structure of the argument in some sense that the software represents. And the way I think about writing and the way I think about coding, at least in that way, has a certain similarity.

00:21:14

James

Yeah. We’ve talked about this, you and I, several times, just the idea that writing is really about the effect that it has on your brain as you’re trying to accomplish something. I mean, it forces you to organize what you’re thinking. And facing down some reader, be it a reader of code or a reader of prose, is largely modeling that person in your head and the effect that that modeling has on the thing that you’re trying to produce, basically what you said, but more confusingly. See, that was the first draft.

00:21:48

Ron

So, before you were talking about how sometimes when you’re trying to write something down, explaining some piece of code and you find yourself writing a lot of documentation, you start getting nervous and realize that things are probably more complicated than they should be. And in some sense, the need to write a lot of documentation is itself the sign of a problem. One of the discussions and disagreements I’ve gotten into with various developers over the years here and in other places is trying to argue to people that they should actually write documentation. And one of the arguments you sometimes hear as well, the code should really be self-documenting. You should make it so it’s clear and concise enough that no comments are really necessary. And that’s really the ideal state for a code to be in. And I find this deeply unbelievable. And I’m curious, what kind of response do you have to that perspective where someone says, “No, no, no, I just make the code so simple that no documentation is required”?

00:22:39

James

Well, maybe two things. One is that this is a big trope in Ruby world where I come from because Ruby’s metaprogramming enables this crazily expressive API where it feels like you’re almost writing English as you write the code. But people produce these beautiful DSLs, domain specific languages, that allow you to write code in that very fluent way. And they’ll say, “To have a comment anywhere near here is a smell. It’s a sign that I’ve done something wrong. That my DSL is not quite expressive enough or clear enough.” But then you go look at the code supporting that API and it’s a total crazy disaster. And that’s one place where comments might be useful is in explaining the undergirding of some clean API that’s self-documenting, so to speak. But there are just things that don’t sit well in the code itself. So like, “What’s the runtime complexity of this function?” That’s not really something that you’re expressing in code.

00:23:37

James

And if you’re looking at a standard library and you’re looking at various different implementations of a data structure, what you want is some guidance as to how to make a decision about which of these data structures to use. I mean just yesterday, I was coming across some problem with concurrent writes to a file. And there was a module for doing this file writing asynchronously, and there was a parameter that just said, “Exclusive: true.” And if you hit the exclusive parameter, you supposedly have a lock on the file that you’re writing to. So that seemed like that solves it. But there was a five paragraph essay above this function saying, “Well, actually it’s more complicated.” And I think that’s a case where if I just read the type signature and I just had this flag, there’s a million things going on that are subtle properties of the operating system of my problem that requires some kind of actual discussion. And I was very pleased to see an actual discussion that felt like somebody trying to guide me through these decisions in the comment.

00:24:41

Ron

Right. The idea that you could mark an API and say, “Oh, you get an exclusive lock,” and that’s the end. It’s like, “No, no, no. You said there’s an exclusive lock.” There is a long and complicated story. Gather around the fire, let’s discuss what locking is about.

00:24:54

James

Exactly.

00:24:55

Ron

Yeah. That’s an excellent point. Like in some sense, I feel like the idea that you can write software that’s so clear, that it doesn’t need any documentation, hinges upon some delusion about how clean you can make the abstractions. I think you can always fantasize about having an extremely fancy type system which would capture more of the properties. People work on type systems that capture performance characteristics so you could know some upper bound on the asymptotic complexity of the operations, just by the type signature. But that really is a fantasy. There are no realistic type systems that do a good job of this. And any type system that people use in a real language that actually is practical has quite severe limitations on its expressiveness, on how much it can tell you about what’s actually going on in the code.

00:25:41

Ron

And I feel like the thing that you described in Ruby land sounds like a different kind of delusion. The complexity you mentioned there was the complexity of the underlying implementation of the macro stuff. So that’s one thing, but also it’s really hard to build macros that do a really excellent job of hiding all of the complexity of the underlying implementation. Just simple things like when there’s an error, does the error come out in a nice clean way or does it somehow reference the desugared representation? When you’re doing this kind of macro stuff, you’re like translating one thing that the person wrote into some other representation, and then somewhere there’s an error that refers to that. That translation can be weird and complicated, and there’s a dozen things that can go wrong with macros. The idea that every time you write a library, you do such a good job of creating a clean and simple abstraction that you don’t really have to tell a story about it, I just haven’t seen very much software that looks like that.

00:26:33

James

Yeah. I think this idea that code should be self-documenting is a good prod, a nudge in the direction of improving, especially in a language like OCaml with a powerful type system, in packing as much into that type system as it can handle. So, don’t have a function that’s, let’s say, about a person’s vaccination status, that goes from a string to a string, and say with a comment that the string input is a Social Security number, and the string output says what kind of vaccine they got. In OCaml, we would never do that. I would often write so-called stringly typed functions early on here, and my reviewer would say, “Well, actually make a structure, like a Social Security number type that has validation, has its own methods on it, and use that as the input to this function.” Then, yeah, when you read that function, you get a much better idea of the universe of possible outcomes, the universe of computations that this thing can take. And yes, we should really be encouraging people to do that. But it becomes a bit of a silly, extreme argument to say that if you see a comment, it’s a smell.

00:27:44

Ron

Right. And your point about how it’s useful to categorize things and give types to various different concepts in your code, I think that’s useful even when there is no special functionality associated with those, it just adds clarity to the system. You get this kind of flow analysis throughout your program, that like, “All these values over this place. That’s a Social Security number.” And you just know more about the semantics because whenever someone constructed it, it was in their face when they were constructing the Social Security number, and whenever it was passed around, it was clear what was being passed around. So it ties together, throughout your program, the handling of this value, so that you now just have a much better grasp on what the value means. And so individual things like functions are clearer, your data structures are easier to understand.

00:28:22

James

Yeah. There’s a really nice benefit of doing that, which is that when you’re in your editor and you’re looking at some local variable whose scope you don’t really understand, you can ask the editor what type is it. And that often just tells you everything you need to know about what’s going on in that area of the code.

00:28:40

Ron

So you’ve spent a bunch of years working in a dynamically typed environment. You’ve now spent a few years working in this somewhat obscure, precisely typed programming language, in OCaml. How do you think that affects the culture of documentation?

00:28:55

James

Yeah, that’s an interesting question. I mean-

00:28:57

Ron

I’ll say, at first I thought you were going to say that in a language like Ruby, people are more incentivized to write documentation, in the same way that in dynamically typed languages, people tend to be more incentivized to write tests because types act as a proxy for both certain kinds of documentation and certain kinds of tests. It’s interesting to hear you suggest that macros in some ways can cut against that. And maybe there’s just unique things about the Ruby culture itself that push in that direction.

00:29:23

James

Yeah. My first job was at a Ruby on Rails – Rails is the Ruby web framework that’s very popular – consultancy and this place prided itself on test-driven development. And I think TDD as a kind of culture, or a cult, is a cult to which I happily belong, is maybe that came out of Ruby, maybe that came out of a dynamic world. But I think it’s kind of too simple to say that one kind of language encourages one kind of culture of documentation, because we have a culture around testing for reasons that are kind of orthogonal to the language. If you’re writing a trading system that could potentially blow up the company if it has an error in it, then you’re going to basically throw the kitchen sink at making sure that it’s correct. And part of that is going to be the type system, and part of that is definitely going to be tests.

00:30:20

James

When you’re working on a web application that’s trying to grow as quickly as possible, and mostly what you’re concerned about is, will people like this new feature, testing is kind of a dumb idea, especially if you’re a small enough company. And so the fact that Twitter had its Fail Whale, it was constantly falling over, I think it maybe has as much to do with the stage that they were at as a company and product, as it did with the language they were using. But of course, switching from Ruby or whatever it was to Scala or whatever it became probably helped too.

00:30:55

Ron

Right. I think you’re totally right that people overestimate how important languages are for all sorts of different aspects. I think of how software systems work and how organizations work. I think culture is a big deal. And I think architecture is actually incredibly important. I think there were a lot of things that changed when Twitter did its rewrite, and that caused the Fail Whale to become a thing of the past. But I think a lot of those changes were not specifically about the language that was used, but were about all sorts of changes that people made to the architecture of the system at the time.

00:31:24

Ron

It actually reminds me of my own experience here at Jane Street early on, our early systems were VBA and some Java and some C#. And then we did this big rewrite of the core systems into OCaml. And it’s easy to think of that as an exercise in just changing the language. But I think that’s not what it was like at all. A lot of things changed. I think we improved lots of things, but we also had completely different sets of people, had new people coming in, grew the team that was working on the software enormously. And there were lots of improvements we made over that time because of that. And I think the language was, actually in the end, a quite important part of that, but by no means the majority of the improvements.

00:32:06

James

Yeah, we have, in my own experience on those tools that I’ve worked on here, it’s been OCaml the whole time, there’ve been terrible design decisions, terrible architectures and fairly good ones. And the differences in the design language is being held constant, the programmers being held constant, so those things can make an enormous difference.

00:32:25

Ron

Here’s a thing I’ve noticed about the culture I’ve seen around documenting OCaml code, both internally and on some external things, which is that the documentation on a lot of OCaml code tends to be very concise and precise and designed for people who already have a pretty good mental model and less discursive, less example-driven than I’ve seen in other places. I’m curious what you think about this. You obviously … Rather, I’ve seen you spend a lot of time building very example-driven forms of documentation, and I kind of wonder what you think about those two styles of how one might document something.

00:33:03

James

Yeah. One is good and one is bad. No. The very first document I worked on when I joined Jane Street was an example-driven guide to this tool we have for querying S-expressions. S-expressions being just lists recursively packed, it’s a way to store data like JSON, let’s say.

00:33:23

Ron

It’s like a version of XML that dates back to the mid-‘50s. Fewer angle brackets, more parentheses.

00:33:30

James

Yeah. So we had this tool and somebody came to me and said, “I think we need a better README.” Okay, I’ll take a look at this README. And not to throw the author of this README under the bus because they produced an incredibly valuable tool and many, many excellent documents, but this was not the best document. This was a formal grammar, basically, that had been cleaned up a little bit for human consumption. And the idea that that would be the README of a tool that was basically like jq, some tool for query in JSON, was incredibly frustrating. It was like, it made me angry. It made me really angry, in fact. It was one of the hardest documents to understand that I could imagine. I almost didn’t have a sense for what the tool did and how good the tool was and what the tool’s practical language was. What are the things that you would actually want to do and in what sequence and why? And that can only be gotten through, or it’s at least better gotten through, an example-driven guide.

00:34:30

James

So that was my very first week at Jane Street was coming across a README that was a formal grammar. And I think it is something about the language. People who self-select into the OCaml programming community are, by their nature, more interested in programming language theory, in formal languages, in pure mathematics, things like that, than people who self-select into the web development community. And there are good things about that. If you want to ask a data structures question, maybe you’d prefer the group who has a stronger theoretical background. But it felt inhumane, frankly, that document. And I think that is something that is pervasive in the OCaml world and maybe in the functional programming world.

00:35:15

Ron

So that’s a really good example, but I’m not sure it’s an entirely fair example. I remember that document and to be entirely fair to the author, the author was fully aware that it was not a great piece of documentation and was happy to have it improved. And that was a case where he was a guy who came with a very strong theoretical PL background, and had built a little language, which was very useful and very nicely arranged, and that echoed certain things he’d seen in other places. And so in some sense, for him, just writing down the bare skeleton of, “Here are the different operators and here’s the definition of what the operators do,” for him, that clicked in his head. So I think that’s how the documentation came to be in that mode. And that’s the classic curse-of-knowledge thing.

00:35:55

Ron

But to argue the other side, I think there is a legitimate trade-off where if you write documentation that’s very full of examples and very detail-driven, it can be a real slog to get through. It can just be long. And there’s something lovely about documentation that just says the absolute minimal thing that an educated, or at least a particular kind of educated consumer wants in order to understand what’s going on. I often feel this way when I read the documentation about a new programming language or a new web framework or whatever. And I feel like I understand this space pretty well. If I talked to the right person, they could give me seven sentences and I would suddenly know exactly in what quadrant of the design space this thing was and how they’re approaching it. And I would immediately get a lot of knowledge very quickly and very efficiently. And it would be great for me, and for other people it would be utterly useless. And so one of the things I struggle with when writing documentation is I think there is legitimate tension. Concision is really good, at the same time meeting the readers where they are is also really good. And one of the problems with writing is you can’t meet all readers where they are with the same piece of writing. So on some level I wonder how you think about managing that trade-off.

00:37:09

James

Yeah. I think you just have many documents that serve all the different people, and that say explicitly what kind of document they are. We saw this recently with one of those classes about a web framework that we’ve built here, kind of like React for OCaml. And hey, there’s a pretty concise way to locate the thing in design space. Three words: React for OCaml. But there’s some people who come to this thing with basically no experience in web development. And so they need a tutorial that explains, where does this thing fit into how you use your browser, and what’s going on under the hood there. And there’s some people who are very familiar with something like React. And so what they need is just, “Here’s a transliteration from the React API to our tool’s API.” And I think, ideally, you would have all of these things. And that’s what good documentation does, is it contemplates the universe of readers upfront and presents a menu and says, kind of, “Choose your own adventure. If you’re this kind of person, start on page 42. If you’re this kind of person, start on page 1.” And yeah, you can’t serve all of those people all at once in a single document, and I don’t think you should try.

00:38:25

James

There’s a very popular thing about documentation, and I wish I remembered who the author was, that basically says there are four kinds of technical documentation. There’s a guide, a tutorial, there are references, and then some other, I forget the fourth quadrant. But anyway, this kind of framework for thinking about, explicitly, what are the kinds of documentation that your group, your library could produce is a very useful discipline for actually just trying to generate these different reader types in your head.

00:38:57

Ron

It also reminds me a little bit of a thing my PhD advisor said to me many years ago, which is when he talked about the writing of a paper, he said you should assume that people are going to drop off in reading at various points of the document. And you want to optimize the presentation of the document so you can get as much information, as much useful information into people’s heads as early as possible. Most people will just read the title. So make the title really clear. And then some people will get all the way through the abstract, and to try and get your core ideas through there. And then a bunch of other people will read the first section or the first couple of pages or whatever. And each of those levels, try and get as much of the conceptual framework across as you can. And I wonder if that’s a good model. And I also wonder if there’s something one could do at the level of the tooling to make things more navigable.

00:39:46

Ron

A thing that I often struggle with is when you look at some piece of code that has a lot of documentation in it, you sometimes feel like you’re missing the forest for the trees. I kind of want to just stand back and be like, “Can you just show me what the types are? Get rid of all these words and just show me the types.” And that will give me a bunch of information and be able to explore and say, “Oh, okay, let me go from this to see some examples using this.” And so some more dynamic set of tools that let you see more of the documentation, depending on what you’re doing, depending on what mode of exploration you’re in, I think could help you construct better trade-offs.

00:40:21

James

Yeah. And in some sense I think this is something that is learned both by the writer and reader in an exchange that takes place over time until you develop a thing called genre. And what we maybe don’t have so much of in technical documentation is genre. So when you open a newspaper, it’s written in an inverted pyramid style where you have the highest level of facts upfront, and then it slowly gets more and more detailed. But once you figure out that’s the style that newspaper articles are written in, you can approach them depending on what kind of day you want to have with that kind of article. So if there’s some topic you know nothing about, and you just want the facts, you might just read the headline. If there’s something you know a lot about, and you want to get to the crux, the detail, then you go find that paragraph which you know is probably going to be about paragraph seven. I think that kind of convention that you’re talking about with academic papers also exists, where people know to read the abstract maybe a few lines from the end of the introduction, let’s say, the figures and the results depending on what field they’re in. Probably they might not read the conclusion because that often just recapitulates the introduction. But I mean, reading a paper is really reading 12 sentences or something like that, at least as a first scan.

00:41:41

James

When you talk about technical documentation, and I think about the documents that people produce here, they don’t fall into any kind of conventional form like that. And I think this idea of the four kinds of documentation, we have tutorials, how-to guides – I just looked this up – reference, and the last is explanation. That’s a beginning of a template for how to produce things that do have a genre so that you know, “Oh, this is a tutorial. This is how you read a tutorial. This is how you write a tutorial.” And tooling could absolutely be kind of baked into that scheme or some other scheme. And yeah, maybe you start having a little more legible ecosystem.

00:42:23

Ron

I almost want to add one other genre of writing to those four, which is evangelism. When I think about the kind of things that I want to do with writing, sometimes it’s about, “Let me explain the problem. Let me explain how this library works,” point out what the pitfalls are. And sometimes it’s just like, “I want to get you excited about an idea, excited about some tool that you could use, come to understand how something could be useful for you.” Which, it’s not exactly a tutorial and it’s not a guide, it’s some other kind of persuasive form of writing, which I also think is really important, and I think is hard for people to do. I think it’s especially hard in Jane Street’s culture to do a good job of evangelism. Because evangelism requires you to be expansive and excited and make big claims about things. And I feel like there’s a lot of cultural push towards being careful and precise and not overstating and having a good understanding of what the holes in your own tools are.

00:43:18

James

Right. And what happens is if nobody is selling their thing with the hard sell, you end up with a bunch of carefully qualified sales pitches. And then you end up with people using a lot of different tools that basically do the same thing. And you wish one of them had just won, even if it wasn’t perfect for everybody.

00:43:37

Ron

So a somewhat different topic, have you ever played around with any of the classic Literate Programming tools?

00:43:44

James

Actually, for a thing that is prizing legibility above all, I found the explanation of what Literate Programming was and what a literate program actually read like to itself be kind of illegible. Possibly, I just came at this fairly early in my life and probably I would pick up The Art of Computer Programming, Volume 1 through N, and find that pretty illegible too, and not any ding on the author. But I think I tried to read some of the TeX source code, which is written by Knuth in this literate style. And-

00:44:22

Ron

It’s maybe worth stopping and saying like Donald Knuth, a great early figure in computer science, continues to work today, did a lot of important algorithmic, theoretical work, wrote a bunch of important books, and, insanely, at some point as he was starting to write The Art of Computer Programming, he looked around and was like, “There are no good-enough layout programs.” And so he stopped his entire life and sat down and wrote TeX, which is this foundational publishing system used in math and physics and computer science academia. He was like, “There aren’t really any great fonts.” So he created fonts and font-design tools. He’s a very interesting, important and I think deeply strange character. And one of the things he invented is this concept of Literate Programming. Can you try and give a summary of what Literate Programming is?

00:45:10

James

And I should say, I basically don’t get it, so how could my summary be any good? But I think it’s basically like really trying to make your software read like a piece of prose. So it’s not that you have documentation that’s separate from the code, it’s that you have code interleaved with documentation. And the order that you read the code in is the order that the document takes you in. So it’s very linear in a way that reading code normally is very non-linear. So don’t imagine jumping around having many buffers open in your editor at once, looking at a bunch of different files. Imagine just being told a story, once upon a time, all the way till happily ever after, where somehow along that story, you end up encountering all of the code. And it’s basically impossible to imagine.

00:46:03

Ron

Right. And weirdly part of what goes on with these tools is you write your documentation interlaced with your code, but the ordering of the documentation is presented and the structure of the files is quite independent. So you might somewhere deep inside, put in some section which is your opening introduction to the code and the full PDF that’s generated, as you said, walks through the entire code base. But in some weird order that’s meant to optimize the presentation of it as a piece of prose, but can be arbitrarily different from the order in which it shows up in the actual code base.

00:46:34

James

My sense with this kind of thing is that in the hands of a master, in the hands of a Knuth, it can be incredibly powerful. And clearly many, many people have read the TeX source and have been able to build upon it. And I think it’s still sort of held up as one of the best pieces of software that we have. So obviously it worked really well. But it just seems like the kind of practice that is maybe best not attempted by us mortals.

00:47:00

Ron

Yeah. I’ve often thought that Knuth is, in some sense, too smart for his own good. He gives advice that is maybe not the best advice for people whose brain is not quite capacious as his. There’s a great book by Peter Seibel called Coders at Work that has a bunch of interviews with famous programmers. And one of them is an interview with Knuth. And the one with Knuth is really wild. One of the things he complains about, he’s like, “Ah, there’s all this kind of abstraction stuff,” but he’s not that interested in abstraction because he’s working on some program and he wants to reach six levels down to adjust the behavior of something to make it work exactly the way that he wants it to work and have exactly the performance that he’s going for. And he just finds these layers of abstraction to be getting in the way of that.

00:47:43

Ron

This is a guy who’s not having trouble understanding how his program works, which I do not sympathize with. I desperately need layers of abstraction to have any sense of what’s going on in a large and complex program. And I think it worked up to a certain scale. I think TeX was an artifact, which is a quite big and complicated piece of software, but it could basically fit in his head so he could get away with it. But it’s hard to see how that advice scales up to larger programs or to less smart people.

00:48:10

James

Yeah. And I think some programs that are the product of a single mind are just very different in nature and architecture than programs that are the product of a team. And often, where you see a lot of problems is in the transition from one to the other. So when an auteur-ed piece of software becomes a maintained piece of software, and there are many, many maintainers, that is the real art of computer programming beyond the algorithm stuff.

00:48:39

Ron

In the end, I think Knuth is a charming writer, but he just made idiosyncratic technical choices that I suspect fit his neural architecture and do not fit others nearly as well.

00:48:49

James

Yeah. And yet we still use TeX all day, every day.

00:48:53

Ron

Yeah. Which makes me ask a different question, which is … So you spend a lot of time writing, all sorts of kinds of writing, purely technical writing, all various kinds of expository writing. How do you approach writing in terms of tools? What do you like to have in front of you when you’re writing?

00:49:09

James

I use, for taking notes, TextMate, which is just a text editor on the Mac. One of the things about TextMate is it’s extensible like any good text editor should be, and you can add your own syntax highlighting for various kinds of markup. And so if I’m writing a piece of prose, for instance, and I have a notes file, I kind of have a structured notes file and a structure I invented where to-do items might be highlighted differently in the editor. One thing I do is block quotes are highlighted differently, just so that I know, “Where is my thought and where is something that I’ve read.” It’s like very distinct. I used to pay more attention to this and had a highlighting for quotes. So anything that was in quotes was a different color. And so you can kind of just like squint and look at a piece of writing and say, “Okay, I’m quoting way too much.” People want to hear from you. They don’t want to hear from your sources directly.

00:50:05

James

I’m reminded of John McPhee, who’s a kind of towering figure in nonfiction writing and journalism. McPhee, I think he’s like 80 years old now. And he’s such a classic figure that maybe you would think he uses a typewriter or something. But actually, he got a friend to build him an extension, basically a mode, into an editor called KEDIT, which allows him to collate notes and recursively organize notes into buckets and outline with those buckets easily available. And it’s got cool things like you never want to use a word that is kind of rare enough to be noticeable more than once in a piece of writing. And he has a mode or a function that runs, and it finds words like that, and it’ll say, “Hey, you’ve used the word ‘impassive’ twice in this article. And you only get one ‘impassive’ per article.” And it’ll highlight it.

00:51:08

James

So I think actually this is probably under done among writers who mostly get word processing software off the shelf, but software unfortunately is this thing that is kind of developed by the developers and sold to everybody else. But yeah, I like being able to kind of hack on my editor. So I have a slightly hacked up TextMate in front of me. And by the time I get to a final draft or a draft that we’re editing with somebody else, it’s usually in Google Docs or something.

00:51:37

Ron

I have to say, I find it absolutely mind-bending that John McPhee uses KEDIT. Not to mention a custom McPhee mode made just for him. That is phenomenal.

00:51:47

James

Yeah. He writes about this in several articles in a book called Draft No. 4, where he really gets into his process. But yeah, it’s very cool and I’m sort of surprised that more people aren’t doing something like that.

00:51:59

Ron

Yeah. I guess it highlights that there just aren’t a lot of people who are in this intersection of people who are technically comfortable enough to modify their editor and people whose primary job is writing.

00:52:10

James

Yeah. I think if it speaks to anything, it’s about how hard it is to modify your editor. Or, these tools should not be so difficult that they require fluency and whatever the hell KEDIT is written in, in order to make something that’s slightly custom for yourself, but that’s a whole other discussion.

00:52:27

Ron

So how much is your own writing process, when you’re doing something like a tutorial around a piece of software, to what degree is it McPhee-like in that way where you kind of prearrange the kind of structural elements then fill in the prose to make it whole? And to what degree is it more like a stream of consciousness, you kind of write it from beginning to end?

00:52:45

James

I think it has to do with how much you are collecting from the beginning. So like something that you don’t know that much about and are trying to kind of wrap your head around, maybe it’s more McPhee-ish because you’re basically acting like a journalist. If I go into a library that I’ve never used before, and my job is to write a tutorial for that library, I’m basically a reporter who doesn’t know anything and is trying to pretend like they do. And step one is you talk to people, you read some stuff, you try some stuff. I think that can be a thing that can be very valuable is actually trying to use this library. And then yeah, you’re sort of collecting notes, putting them in these envelopes, moving the envelopes around.

00:53:27

James

I think there’s very few things that have no structure at all, or sometimes you spill onto the page because you sort of have these pre-cached notions about what you want to say. You’ve done this in your head already, the organizing, and then it comes out in one form, and then maybe you take another crack at it. I mean, that’s the thing about all writing is that even if you write an email, I mean that email can be losslessly compressed by like 70%. I mean, that’s just the lesson that you learn when you have editors compressing stuff constantly, is that the compression algorithm for lossless compression of a piece of writing is very efficient. It’s like gzip level good. And so, if you write an email and you don’t gzip it with your brain before you send it, you’re causing everybody a lot more pain.

00:54:18

Ron

It’s often better than lossless compression because I feel like when I take a piece of writing and make it smaller and tighter, I’m often adding information in the process, making it clearer and simpler for the person to understand what’s going on.

00:54:31

James

Yeah. I just had an example of this the other day that I was sending an email and included in it was an error trace from some exception that I had, and I just pasted in the exception. And there were like 14 lines, and I found myself saying, “Okay, a lot of these lines aren’t important, just focus on this bit.” And I was like, “Why am I saying this? And why am I pasting all 14 lines? Just cut the exception.” There’s no rule about having to show the whole context and it wasn’t doing anything useful. And I think that kind of thing just always occurs to you on the second read. And so if there’s one piece of advice about how to do this better, it’s just never send, never publish, never assume that it’s done, a first draft. A first draft is just that.

00:55:17

Ron

Yeah. I feel like I run into this all the time when sending emails. And in fact, I think a common pattern is I’ll have something I want to send and I’ll compose an email and then I’ll read it over and tighten it up and read it over and tighten it up. And then sometimes the best of all things happen, and I’m like, “I really shouldn’t send this at all.” And then I delete the email.

00:55:33

James

Right, right.

00:55:34

Ron

I realize the world is better without it.

00:55:36

James

We talk about technical writing as if we’re all sitting down and writing some tutorial from scratch or writing a design document or something like that. But actually, the bulk of technical writing that we do is emails to one another. And if I could get up on the pulpit and talk to the writers of the world, it would just be like, “Your emails are the thing that’s consuming everybody’s time all the time. And taking a little extra care with them is going to go a long way.”

00:56:04

Ron

Right. Which I guess highlights the fact that technical writing is actually a very broad thing. With that lens, I’m actually curious whether you have any technical writing or technical writers that you particularly like or are excited about. Who do you like to read in the kind of space of writing about software and technology?

00:56:21

James

That’s a good question. In terms of software, I don’t know if I have any particular author that comes to mind. There are a few things that I really liked. I’ve never used Rust, but I’m really impressed by their API documentation. It uses a million examples, it’s extremely consistent. Every function links back to its place in the source and things like that. And in a way it’s kind of authored by the community. I like things like the TensorFlow documentation. I always point out that their tutorial for just getting started is incredibly well presented. The thing where you basically write a program that can recognize handwritten digits. But when I think about technical writers, I’m often thinking in other domains.

00:57:12

James

So in biology for instance, there’s this person named David Goodsell who writes books that are incredibly well illustrated because in addition to being a molecular biologist, he’s an artist, and he draws proteins. And you can’t see proteins, so you have to kind of imagine them based on data that comes out of various instruments. And he just does an incredibly good job imagining them. And because he’s not a computer, but rather a person, he can emphasize the things that are important. I think of a book called Defining the Wind, which is about the Beaufort scale for wind. So this is incredibly beautiful, written in kind of King James Bible-esque English, scale for saying how windy it is. So as an example, a wind of less than one knot is described as “calm smoke rises vertically.” And then a gentle breeze, which is a wind of seven to 10 knots is described as “leaves and small twigs constantly moving, light flags extended.” There’s just like a kind of poetry to the language.

00:58:23

Ron

Yeah, that’s lovely.

00:58:24

James

Anyway, there’s this book that talks about the scale. And one of the things it kind of points out, that I remember from this book, is that Beaufort was doing land surveying, and I think he would go out in his ship and he would look at coastlines and he would go onto the islands and so on and try to write up what they were all about. And if you took a photograph from the shore and said to somebody, “This is the coastline,” actually, it’d be a lot less legible than if you had them draw a picture because the whole thing the brain is doing is distorting reality and making things that are supposed to stand out or that are unique, more salient, and that’s when illustration does.

00:59:09

James

So I think the best kind of technical writing is doing this warping of reality into a space where the things that you should care about are made more salient. And often shortening things, illustrating them, thinking carefully about… maybe the most important thing you can do is just trust your own judgment about what’s boring and what’s interesting. And if you read a piece of code, if you read around a library and just pay attention to what you’re thinking, what your reactions are, and you write those down, often it’s going to be less stuffy, less formal, maybe less comprehensive, but it’s going to bring out the things that matter most and most readers are maybe going to have a similar reaction to you.

00:59:53

Ron

Right. Which highlights that even technical writing is in many ways more human than it is technical.

00:59:57

James

Yeah. I hate the phrase “technical writing.” The title technical writer sounds about like the most boring possible job you could imagine. And that’s not what I’ll necessarily say that I do at parties, but it’s no different than any other kind of writing. And it’s something that we all do constantly without thinking of ourselves as technical writers.

01:00:20

Ron

All right. Well, thank you so much. This has been really enjoyable. It’s been fun. I feel like documentation is a topic, and writing in general as a topic, that I’m constantly thinking about and constantly engaging in and very rarely get to talk about in this kind of meta way. Thanks for joining me. This was a lot of fun.

01:00:36

James

Thank you so much.

01:00:39

Ron

You’ll find links to some of the things that we discussed, including some of James’s writing, and some of the software that he’s worked on, at signalsandthreads.com. Along with a complete transcript of the episode. Thanks for joining us, and we’ll see you next time.