BLURB

Peter Bogart-Johnson was one of Jane Street’s first program managers, and helped bring the art of PMing—where that “P” variously stands for “project,” “product,” or some blend of the two—to the company at large. He’s also a poet and the former editor of a literary magazine. In this episode, Peter and Ron discuss the challenge of gaining trust as an outsider: how do you teach teams a new way of doing things while preserving what’s already working? The key, Peter says, is you listen; a good PM is an anthropologist. They also discuss how paying down technical debt isn’t something you do instead of serving customers; what Jane Street looks for in PM candidates; and how to help teams coordinate in times of great change.

SUMMARY

Peter Bogart-Johnson was one of Jane Street’s first program managers, and helped bring the art of PMing—where that “P” variously stands for “project,” “product,” or some blend of the two—to the company at large. He’s also a poet and the former editor of a literary magazine. In this episode, Peter and Ron discuss the challenge of gaining trust as an outsider: how do you teach teams a new way of doing things while preserving what’s already working? The key, Peter says, is you listen; a good PM is an anthropologist. They also discuss how paying down technical debt isn’t something you do instead of serving customers; what Jane Street looks for in PM candidates; and how to help teams coordinate in times of great change.

Some links to topics that came up in the discussion:

• LIT Magazine (more recently here)
• How to be a PM that engineers don’t hate and How to be an engineer that PMs don’t hate

TRANSCRIPT

00:00:03

Ron

Welcome to Signals and Threads, in-depth conversations about every layer of the tech stack, from Jane Street. I’m Ron Minsky.

It’s my great pleasure to introduce Peter Bogart-Johnson. Peter is a program manager here at Jane Street, and has been here for about three years, where he’s been involved in a ton of different projects that we’ve done. Lots of different things across many different groups at all sorts of levels of complexity. Thanks for joining me, Peter.

00:00:30

Peter

Thanks for having me. Very glad to be here.

00:00:32

Ron

You have a pretty interesting background. You were not always a program manager. Maybe to start, you could tell us a little bit about what you did before you got to Jane Street. And in fact, before you got to being a program manager at all.

00:00:43

Peter

Yeah, happy to. I’ll preface this by just saying I’ve worked in technology for about 12 years. My route here was a bit circuitous, maybe a bit non-traditional.

So what I studied in school at any rate was theater and English lit. I was a double major in both of those. And then I went on to graduate school for both as well, in California and then here in New York.

Moving to New York to study creative writing, specifically poetry, I went to school at The New School. All their graduate programs are in the evening, so I was able to have a full-time job while I was also in school, which was a very busy time.

But the area that I worked in at the time was the nonprofit field here in New York. So I worked for a couple of charities, specifically charities that were involved in providing services to low-income families in Upper Manhattan and the South Bronx, doing advocacy work, fundraising, program development, things like that.

At the same time since I was involved in writing, I had also started editing a literary journal called LIT that is a New York-based journal. While I was doing that, one of the editors that I worked with, interestingly enough, worked for a company that perhaps many people have heard of called D. E. Shaw & Co. and had joined as a generalist, which was just somebody who did a lot of non-technical work and did so by virtue of the fact that he had this other agenda, being a writer and being interested in other things. And that was something that D. E. Shaw and all the companies associated with D. E. Shaw, including DESRES where I eventually worked, was also very interested in doing in terms of their hiring.

00:02:05

Ron

Where DESRES is short for D. E. Shaw Research?

00:02:06

Peter

D. E. Shaw Research. That’s right. For those of you who don’t know, D. E. Shaw Research mainly focuses on biomolecular simulations for a particular thing called molecular dynamics and building supercomputers to support that.

So through him, I was introduced to D. E. Shaw, and interviewed there, and joined a team that included many other people like me who were in graduate school, or had just finished graduate school, or were writers or painters or musicians or something like that, doing all the strictly non-deep technical work, although we started moving more and more in that direction over time.

00:02:39

Ron

So did Shaw make an explicit point of hiring people with MFAs and poetry to work on managing and organizing projects for them?

00:02:46

Peter

Well, I don’t think they were advertising in MFA journals or anything like that, but they definitely had an approach of, “Well, there’s this class of job that is not specifically technical, maybe it’s technical-adjacent. And we could fill it with smart, talented, creative people who might bring a slightly different perspective to the role, maybe more familiarity and comfort in abstract thinking and holistic thinking.” And the notion was we’d hire these people, they would do this job, then they would go off and become famous writers and musicians, and they wouldn’t be here anymore. But a lot of us sort of got to like it, and stayed in this work over time.

00:03:21

Ron

So this is totally different from any hiring process I’ve been involved in. I feel like every time I think about hiring, I always think, “Well, what’s the subject matter? And let’s try and find people who know the subject matter.” So it’s super different, which makes it really interesting to me. I’m just curious, how do you think it works?

00:03:36

Peter

I think it works well. In a way that you can see a parallel, we can talk about this later probably, in the types of people who come to Jane Street as well to fill roles that are not specifically technical, but might be technically adjacent. But yeah, I think that at D. E. Shaw, in particular at DESRES, it worked very well because it did this thing that I thought was very important, which was instead of engineers doing all the technical work and then also switching context and doing things like technical operations, or project management, or product management, or any of these things that surround the technical work, moving that to a set of people who are just focusing on that and having more of a multidisciplinary approach to your team, it made the technical contributors much more effective and focused on the problems they were trying to solve. It seemed to work very well. A number of us who started in those roles ended up staying in this field.

00:04:25

Ron

So I totally get the idea of focus being important. And as an organization grows and people get more specialized, being able to get people who really take over and own various, kind of, sub-parts of the overall thing that you’re trying to achieve.

I’m curious, when you think about why people with these kinds of essentially creative backgrounds were good fits for the kind of work, to what degree was it about the things that they learned in those other areas applying in a surprising way, to what degree is it from them just being from a different context? Which in some ways, not too tied up with the subject matter, or is it something else?

00:05:01

Peter

I think it’s probably a combination of a lot of those things. I did this exercise once where I actually talked about how a poem and a project have a lot in common in the sense that they both might have restrictions. They both might have a particular audience that they’re trying to reach out to. They have a beginning, middle, and end, you would hope. And there are these things about them that are kind of common.

Part of that was a joke, but part of it was an interesting exercise. And I think there’s a little bit of that in the sense that you are also a person who makes things and is interested in creating things. And I think engineers are incredibly creative, and you just do it in a very different context. So you sort of get the thing that is being done, but you’re looking at it through a different lens.

And I think that is just helpful because it brings an outsider perspective, and I think that’s something that is very valuable. Something that I’ve tried to think about a lot over time is how to do this work and maintain a little bit of that going into it so you’re not so enmeshed in the context that you forget.

00:05:55

Ron

Right. Because part of the problem of doing this work for a long time is, like it or not, you end up learning things.

00:05:59

Peter

You do. Yeah, you do. You learn things. There’s a whole language that you learn. There’s a way of interacting with people, and I think all those things are really valuable. But I do think that there’s some value, a lot of value, in fact, in trying to maintain that separation of views and the things that you care about, and being able to move between them and that there’s a value in that.

00:06:17

Ron

So do you still write?

00:06:19

Peter

Yeah, I do. I try to write every day. Sometimes I fail in writing every day, but I certainly do try. It’s a big part of my life. The thing that I do outside of work is, being in New York, there is a very robust writing community here. And in fact, there are multiple writing communities here. Maybe it’s one of the only places in the world where you can really have that. Over time of living in New York and being a writer, I’ve become pretty enmeshed in a few of those communities. I ran a reading series for a long time in Brooklyn. Obviously, I edited a literary journal for a long time, and I still write and I send my work out for publication, and try to get published, and do readings myself, and read at them, and just maintain contacts with other writers here, and share work, and edit, and critique. Yeah, it’s a big part of my life, and I try to do as much of it as I can.

00:07:01

Ron

Not that we should go into a slam poetry session or anything, but what kind of things do you write?

00:07:05

Peter

Yeah, poetry specifically. The type of poetry would be like… There’s a school of poetry that was particular to New York called The New York School, which was the poetry version of abstract expressionism. So this is a mid 20th century poetry school that originated here in New York, obviously The New York School, and centered around parts of Lower Manhattan like the East Village and Lower East Side, and has grown over time to have multiple generations past that.

In particular, there are institutions here in New York that I’ve tried to stay as close to as possible. One of them being The Poetry Project in the East Village on Second Avenue, St. Mark’s Church. It is a center of poetry culture in New York.

00:07:43

Ron

So there’s a whole different podcast we could do now where we dive all into the writing, but I’m going to resist the temptation, although it’s a pretty serious one. Let’s talk a little bit more about being a PM at Jane Street. And maybe I should provide a little bit of background, and then I want to hear a little bit more from you about what your process coming here was like.

But maybe a thing to say about Jane Street is we’ve spent most of our existence as if this whole discipline of being a program or a product manager just kind of didn’t exist. The place, when I started a little more than 20 years ago, there were about 30, 40 people in the organization. And the process of building technology was very kind of integrated. We kind of thought end to end about the process that people who were writing the software were sitting next to and working closely with the people who were the users of that software. And things weren’t in some sense just big enough and complicated enough to have organization of that be its own separate discipline.

And the organization has grown a lot since then. We were 30 people. We’re now over 2,000 people. And in that time, the complexity of building software projects has grown dramatically. And it’s at the point where when you want to build a complicated thing, you often need to involve lots of different people across lots of different teams. It’s a lot more work to even figure out what’s the right thing to build, because there are so many more people who are on the consuming end of any given project you build.

And I guess one thing that I kind of saw over multiple years, and I think lots of us saw, is that we were in some ways getting worse at it. It was more and more obvious that as we tried to do bigger and more complicated things, that we weren’t that good at getting organized and making sure that we really understood what the problems were, that we shared priorities effectively across different teams. It would be really easy … I think that we ran into a lot, say on the Tools and Compilers team, is, there’d be something that someone was building, say, some new language feature, and you had to deliver that.

But just delivering the language feature wasn’t enough. It turns out you also had to make changes to the editor integration to support that language feature, and to libraries, and refactoring tools, and lots and lots of different places. And we’d find ourselves dropping the ball several times along the path and things would take forever to land.

And that’s a relatively small version of the problem. Even bigger, you do a big effort that’s not just inside of the developer tools world, but that crosses the people who are managing servers, and developing trading software, and people who are building custom software and trading desks, and getting all of that stuff to work together. Just got really, really complicated. And so we started getting more serious about treating the organizational pieces of this as a first class thing.

But I think that left us in a somewhat hard spot in the sense that we started bringing people in who were experts in this area into an organization that just kind of didn’t really know how to engage, right? We’d never had PMs before. And so just getting used to that I think was complicated. Complicated for us and complicated for the people coming in. So all that background said, I’m curious what your experience was like coming here. And actually, how did you find us in the first place?

00:10:40

Peter

So I came to Jane Street in the early days of the process of figuring out how to work with PMs as you described it. I had a colleague who I had worked with at DESRES who we both left DESRES and went off and worked in other places. We both worked at startups, and she came and was one of the first PM hires here at Jane Street.

And then around that time, the startup where I was working was reaching the point where we were trying to get acquired basically. And it was a good time to start thinking about what next steps would be. I’d been with that company since roughly early on in its life cycle. And it seemed like a really interesting opportunity to come to a place that just did not have this kind of role and to really think about it as a new discipline here and try to grow the practice. And it seemed like an interesting, challenging, maybe fun thing to do, to try. Yeah, so that was how I was introduced to Jane Street.

00:11:30

Ron

And to make it all the more challenging, this was in the middle of the coronavirus pandemic?

00:11:33

Peter

Yes. My first call with somebody from Jane Street was in May of 2020. So very early on. I think we were all still trying to get used to even how to have these kinds of conversations not in an office, not in a place where you’re meeting somebody face to face.

And obviously a naive person that I was at the time about anything about Jane Street just assumed, “Well, maybe this role is new, but they’ll know how to talk to me and know how to work with me, because who doesn’t know how to work with somebody like this?”

So when I first joined Jane Street, I worked on our Networking team. So this is a very also interesting time and an interesting part of the firm to join. This is a part of the firm obviously that is fundamental for anything to happen here. But I would say in terms of technology, is a few layers removed from the business operations of the firm, the actual business activity of the firm. And so gaining context to the firm itself is very interesting from that vantage point.

So when I joined that team, there were a couple things that were really needing to happen. One, there was just a set of projects that they were in the midst of trying to push through that all had this level of complexity that you just kind of talked about. These are multidisciplinary projects that touch many, many teams and had deep implications for the growth of the firm. In this case, there were two data center builds that we were in the midst of. And there was a very large compute cluster that our Research team uses that needed to move from one of those data centers that was still being built, to the other one that had just kind of finished being built.

I mean, all this was happening at once. One team that was really pushing on it, in this case, the Networking team with a lot of ancillary teams involved as well, and a lot of moving parts. So that was one thing. There was a very clear mandate. We need somebody who’s going to corral this work, who’s going to care about pushing it forward, making sure the requirements are known, making sure everybody has the same end state in mind when they’re actually doing this work. And at the same time, trying to create a world where projects like this could happen in a more sustainable way. So not just ‘do the projects,’ but create some system whereby projects like this could be executed effectively in the future.

So it was pretty interesting. It was a little bit of sailing on the ship as you’re building it, if you will, or in this case, in a data center as it’s being built. For me, the challenge was doing all this and then trying to gain context on the firm at the same time.

And then also as you said, most of these teams had never worked with a PM before, and there was a bit of educating the teams how to work with me, and the teams educating me how to work at Jane Street. It was a very interesting time.

00:13:57

Ron

What are the things that strike you as things that were missing at the time when you started?

00:14:01

Peter

So a concrete thing is one system where everything is being tracked. No matter where you go, you’re going to find people using different tools to organize themselves and try to figure things out. But when you are working on a very complex project that touches a lot of teams and has a lot of moving parts to it, like a data center build or a huge compute cluster migration, having bits of that live in many, many different places and not having any real connection between them makes that so much harder.

So one of the first things that we really focused on, and this is a fundamental thing that I think most PMs here do when they first join a team, is let’s establish a single point of entry for information. So you don’t need to switch contacts. You don’t need to go to your Google sheet, back to your PM tool, back to an email thread, back to some sort of chat thread of some sort, to get all the pieces of that information. Let’s have them all live in one place.

00:14:53

Ron

So one thing that’s tricky, I think, about driving this kind of change, where you come in and people aren’t quite using the right kind of tools and workflows that you think would make the most sense to help them kind of scale up. I guess two things. One is, convincing them that it’s worth their time to go through and change how they operate. And the other is understanding well enough how they operate to make good suggestions about changes.

People have built lots of workflows here over time, in an ad hoc way. Some of them are good, some of them are not so good. In many cases, it’s reactive to the situation they’ve been in. It’s kind of built up over time in a way that reflects things about the organization and the people who work here.

So how do you think about that process of understanding enough that you can effectively work with people, and really bring them along, and convince them that where there are changes to be made, that they should make those changes?

00:15:40

Peter

I think part of it is building trust takes time, and you need to do a couple things. One, you need to meet a team where they are and not come in with too strong of a point of view. And you can imagine somebody helicoptering into a team and looking at a world that has been created, as you say, reactively, and kind of grown over time. And that person might say, “Okay, we are going to go full agile in every part of our work. We’re going to cultivate a nice slim backlog of things. We’re going to do sprints for everything, and we’re going to do story points.” A lot of these terms that people throw around in product and program management, not taking the time to actually get to know how the team works.

00:16:18

Ron

And it’s maybe worth saying, those words are basically foreign jargon here at Jane Street. I have never thought about a story point for almost anything I’ve ever thought about. I’m curious, what do you think about-

00:16:32

Peter

Probably saved you years on your life.

00:16:35

Ron

(laughs) What do you think about all of this ornately developed set of methodologies that are out there?

00:16:39

Peter

I think what you want to do is you want to try to take from that basket of those frameworks and find the things that work best for your team. And maybe it’s not from those frameworks at all. Maybe there are some things that you bring in. Maybe the thing that you bring in is, “Okay, let’s think about the bare minimum version of the thing that you’re trying to build before you do the fully baked thing, and try to get it out into the world and get feedback on it before you do the fully baked thing.” How about start there?

That’s a very small borrowed ideal from a couple of these frameworks where it’s like, “Cool, in order for me to get a reaction from my user base, I need to get something in their hands quickly so they can react to it, and not build every single feature I think that they want. Because very, very likely they’re not going to use everything.” And I think when you do that and if you do it well, you’re not dropping a bureaucratic thing on top of a team. You are showing them slowly over time and incrementally that things can get better. Those sorts of changes are ultimately what builds trust and shows your value as a PM to the team.

00:17:37

Ron

Yeah, I guess I’m much more comfortable with the idea of narratives rather than methodologies. Learning how people approach things, figuring out what ideas apply to what you’re doing, rather than trying to take as a whole, “We are just going to follow this process, and that’s the end of it.”

00:17:49

Peter

And it’s much more interesting too. You can actually do the anthropological work of getting to know the team. Understand them, understand their practices, the way they behave, the interpersonal relationships of that team. There’s so many small little facets of how a team works together, that it takes months before you really understand it fully. And in some cases, you’re constantly evolving that because people move in and out. If you’re doing your job effectively, you’re bringing small amounts of change in there. And so that will affect culture as well. Change reverberates slowly, but it’s something that we should just be aware of and watchful of. For me, I think that’s where the joy comes in for the work.

00:18:21

Ron

So your onboarding was, as you said, during the pandemic, and the coronavirus is still around, but it’s a very different world at this point, and among other things we’re largely back in the office.

00:18:30

Peter

Yes.

00:18:31

Ron

I’m curious what it was like getting to know Jane Street in this environment where you were not within 100 feet of anybody else.

00:18:37

Peter

I was in my dining room in Brooklyn, New York. That’s where I was. And frequently running out of the dining room, which I shared with my wife, it was our shared workspace to take calls and not be loud and to annoy her while she was doing her job.

It was very interesting. I mean, Jane Street at the time when I joined was just north of 1,000 people. For me, that was quite large coming from a small startup here in New York. So there was this sense of a global community of people who are all working for this one company and working very, very hard. And I knew 25 of them. And that was who I interacted with for the first—call it close to a year– that I worked here. Maybe that 25 people kind of shifted a little bit and changed here and there, but it was trying to understand this broader web of projects, and interactions, and nodes of connectivity from this very, very, very interesting context of doing all this remotely, only talking to people on the computer screen or on some sort of video conferencing device. And trying to learn the culture of the team made it even more challenging from that vantage point.

00:19:35

Ron

Did it feel really different? How did your feeling about the organization and the people you work with change when you got back into the office?

00:19:41

Peter

It changed dramatically. The first nine to 10 months when I worked here when interacting with those 25 people, the sense of culture as a company didn’t really exist to me. You just couldn’t discern it very well from such a small sample size.

I still remember very well coming in my very, very first day. It was two days before Memorial Day weekend in 2021. I had just finished my second round of Pfizer vaccine. I had just gotten to four weeks so I could actually come into the office.

And I remember walking in, being led to my desk by someone from Front of House, and looking down the row and seeing people. There weren’t a ton of people because we were still figuring out how to get everybody back to the office at that point. But actually seeing people that I didn’t know and didn’t recognize at all, that worked for some different part of the firm. And I had no idea what they were working on or what department they worked on at all.

And I remember being somewhat sheepish and asking one of them to help me lift my desk up so I could turn it into a standing desk, because I just didn’t know how to. And then knowing that I could go, and sit with people, and talk to them at lunch, I hadn’t done that for over a year. It was relearning all this at once.

The way that people were very welcoming of me when I came into the office was just this really interesting insight into this culture that I’d kind of heard about a lot over the previous year, but I’d yet to experience.

00:21:01

Ron

One of the things I really like about how Jane Street operates is people are generally really friendly and helpful.

00:21:06

Peter

Very.

00:21:07

Ron

You come by and you ask for something, and people’s default is to want to be there to help. But without those incidental encounters, it’s hard to see a lot of that.

00:21:15

Peter

It is. It is. And it’s those incidental encounters that I think people miss the most when they weren’t working from the office. And ultimately as a PM, a lot of this work is surprisingly or unsurprisingly very relationship driven. For me, the missing ingredient from when I was working at home was the ability to spontaneously interact with people and have conversations with them at their desk, to try to dig into a problem and understand what was happening and try to help them solve it.

00:21:38

Ron

So you talked some about your early work on networking. You actually also spent a lot of time working on our technology for remote and hybrid work, which unsurprisingly all of a sudden got really important in 2020.

00:21:49

Peter

It did.

00:21:50

Ron

Can you say a little bit more about what that looked like when you arrived and what part of that you contributed to?

00:21:55

Peter

Sure. Yeah, happy to. In fact, the reason why I was in the office two days before Memorial Day that year was we were planning our first big contingent of people returning to the office, and I had been there helping set up some of those communication tools that week to try to make that go as smoothly as possible.

At the time, the firm had two tools that we were using for communication. One of them was your kind of standard video conferencing software that we used. We just used Jabber for that. We had also in the early days of the pandemic, really figured out that that was not the most optimal method of communication for people who were used to just talking to each other on a row very, very quickly in a sort of real time way. So from there, we had found a low latency audio tool. It was mainly used for computer gaming, and we decided to adopt that.

00:22:40

Ron

Which by the way is both hilarious and makes a lot of sense. The idea that you pick up tools from the gaming world to get that kind of low latency audio, and not just low latency, but also you have some game where you’re coordinating, “We all have to do a siege, take the castle,” and stuff. It’s like there’s lots of different patterns, we need to communicate to each other, and need to be able to flexibly move around between virtual spaces. And this is not what corporate video conferencing tools were designed for.

00:23:08

Peter

Definitely not.

00:23:08

Ron

But gaming tools were right there.

00:23:09

Peter

Indeed. And if you look at our headsets, you would see that we have adopted many things from the gaming world, in fact. So we had these two forms of communication. Both were really effective, but were effective in very different ways. And some people were using one, some were using another. But ultimately, we needed everybody to talk to each other.

And so when I joined, it was right around the time that we were building these interesting tools that were sort of bringing them together, these two different systems and ways of talking.

So for example, this low latency tool that we use did not have video, and we needed a way to enable video but still have the low latency audio. So we created a tool that we use internally to basically bridge these two systems together so you could actually use one for audio, and use the other one for video and then mute the audio in the video one.

And it was very, very smart. And we had figured out a way that you could just click a button in your calendar, and the right things would happen. You’d be able to talk to people the way you wanted to, see them the way you wanted to. And then if you needed to move to the other tool to speak to somebody, maybe because you didn’t care about the low latency for whatever reason, you could do that easily too.

And that was the world of tools that we had just decided to make. Because as you say, emergent problems require these bespoke solutions. And we decided that it required some investment and we wanted to do that. And so we built that.

Going back to the office, you can imagine a world where, okay, we’re going back, maybe we don’t need these tools anymore. But we’re also entering a world that everybody is in right now where some people are working from home some days a week, some people are back in the office. And we wanted to bring the best of what we had developed working from home back to the office.

00:24:40

Ron

And one of the things that we realized during the pandemic, which is, while it was, I think, clearly the case that separating everyone made a bunch of things worse, there were some things it made better. And in particular, we’ve always needed good cross-office communication. There’s trading around the world. We have offices in London, and New York, and Hong Kong. And having those tools, a way of bridging across the offices was really valuable. And so figuring out how to preserve that seemed important too.

00:25:03

Peter

Yeah, it was like a new superpower. It’s all of a sudden you can use it in whatever ways you wanted, both in the office and working from home. And we wanted to enable that in a way that was very seamless and also maybe a little bit more mobile, so you could just walk around and kind of use these tools wherever you were.

So around the time that I came back to the office, we were building out those more lightweight versions of these communication tools. We hired an outside application developer to create an iOS app for us to use, and we bought a whole bunch of devices to put in people’s hands so they could actually use these apps to talk to each other. And we put them on everybody’s desks, and we got a bunch of desk mics and good video cameras and everything like that. Just to make sure that when you came back, you had everything that you had developed while you were working from home, and you could switch between them with ease. And so there was no difference in the environments that you had. It was really interesting.

And one of these projects that developed from the ground up, this was definitely an example of the people who were developing the tool and the people using the tool were sitting in the same room together talking to each other, and really trying to understand the best way to do it, and the best way to give it to as many people as possible. And that’s something that probably wouldn’t happen in a different environment where you might make the decision based on some other thing. Like, “Okay, we’re going back to the office. I know that I can get a discount on this huge package of licenses to use this standard communication tool.” And then that’s it. You’re done. You figured out your solution. It just may not be what everybody really wants. But this is much more of a, “We are building something that is emergent. We know that people are going to use it, and we know it’s going to enable them to work better in a more satisfying way.”

00:26:34

Ron

So one thing that strikes me about this story is the way in which it steps away from the kinds of things that we’ve traditionally built at Jane Street. Jane Street is not a product company. We’re not building collaboration software for the outside world. We’re not a customer service company of any kind. We’re mostly building our own trading systems, and we provide services to the outside trading world, but mostly mediated through the world’s anonymous electronic markets. So you don’t need a lot of good customer service for most of that.

But that’s changed over time. In a lot of ways, we actually have a lot more customer-facing work than we used to. But this thing in particular feels in some ways much more like building a traditional consumer product in that you have not tens of thousands of users, not millions of users, but hundreds of users.

00:27:14

Peter

At least over 1,000 potential users.

00:27:17

Ron

That’s right. And you’re doing it in a context where there aren’t a lot of people used to building that kind of technology, and getting the kind of feedback that you need to build that technology well. And I’m curious, what was the experience of helping the organization learn how to do that kind of work?

00:27:32

Peter

Part of it was really, really helped by the time constraint. Having a deadline is quite amazing.

00:27:36

Ron

There’s nothing like an emergency.

00:27:37

Peter

Nothing like an emergency. And knowing that hundreds of people are about to come back to the office and this thing needs to be available for them at that point was a really, really good forcing function.

One of the things that we really needed to focus on was finding that minimum viable product of that tool that we can put in people’s hands to use, and make sure it’s as usable as possible in that form. And in some cases, we needed to give something that was less optimal than what we ultimately wanted.

The design, the UX was not ultimately the thing that we arrived at, but it was something that could be used, and we decided to focus on the things that mattered most, which was the performance of the application, rather than the look and feel of it. And we had to make those trade-offs.

Part of that came from finding a set of users here that we could test this out with too. The two months previous to coming back, we had located a series of users from multiple parts of the firm in tech, in trading, and in other parts of the firm that could use it, and give us feedback, and make sure that we were on the right track. And we did a lot of that. We would share designs with people. And we would find the right proxies as well whenever possible, because going out and talking to 1,000 people is still hard. But if you can talk to between 10 and 50 people on a regular basis who have their finger on the pulse of their particular part of the firm, that feels pretty good. And we tried to do that.

Again, this is something that I think is very much in line with how a lot of startups operate. They try to be as scrappy as possible, infer as much as they can from the data that they have. Maybe they just can’t get access to anything more than that, and try to put early, small, simple things in people’s hands to use and get feedback and iterate from there. And that’s something that we did, and it’s something that we worked on for the next year and a half afterwards to try to get it right.

00:29:15

Ron

And keep on working today.

00:29:16

Peter

Yeah, in fact.

00:29:17

Ron

In fact, a new version of the Android app that dropped recently, that got a ton better, I’m very excited about.

00:29:22

Peter

Yeah, we were really happy about that too. That was one of the main things that we thought about as well was different sets of users, different devices, making sure that there’s feature parity across all of that. And sometimes for the Android version, launching a version that didn’t work as well as we wanted to, but we knew people would use. These are all things that you kind of have to do if you’re trying to be scrappy, and work fast, and get things in people’s hands as quickly as you can.

00:29:41

Ron

This reminds me of one of the things I really love about the work here at Jane Street, is that it’s a mix of different modes of operation. Sometimes, we are operating in kind of ‘ordinary mode’ of software that we’re building, technology we’re developing, where it needs to be super reliable, and it has to work all the time, and making iterative changes on things. And sometimes, there are emergencies. And in the middle of those emergencies, the field is much more like that of a startup. And there are sometimes big firmwide emergencies where everyone has to point in a new direction, because maybe there’s a pandemic, or there was a flood, or there’s a brand new market opportunity that kind of drives the whole organization to one direction. And sometimes the emergencies are sort of more focused of, there’s a place where it’s like, “We need to do this thing.”

But the ability to shift gears and for the organization to stop and look and say, “Right, we need to operate in this other way, and we have to break out of our usual structures to move fast and make more compromises.” I love both of those modes of operation. One of them gives you a certain kind of depth and technical excellence, and one of them lets you just get new things done quickly. And it’s important to be able to do both if you want the kind of technology in the organization to remain vibrant.

00:30:43

Peter

Yeah, very much so. And you would think maybe as an outsider, that a firm of our size, that would be difficult to do. But it’s actually quite the opposite.

I would say that there’s these two interesting currents that happen here. On one hand, a lot of our decision making is consensus driven, and it’s finding the right conversations to have around a thing before you move forward on something like that. And that can feel maybe a little slow at times. And not unnecessarily. Just, you need to make sure that all points of view are accounted for.

Now you can add in the same environment very, very, very quickly, coalesce around a particular problem or emergent opportunity, and move very, very fast. And the fact that we can do both in our size is something that is not anything I’ve experienced elsewhere.

00:31:22

Ron

So that’s maybe one marker of things that look different here. I’m kind of curious, a little more generally, what are the kinds of things that you’ve noticed about Jane Street that seem different? And I’m interested both in valuable things that seem worth preserving, and also things that seem like maybe problems where we could be better, things we’d like to fix.

00:31:38

Peter

Yeah. Going back to the example that I just gave around consensus-driven decision making, I think that there’s more of that here than even… I was at a firm that had a total headcount of 25, and I think that we were much more structured and bureaucratic than Jane Street at 2,000, essentially, people. The fact that that has been preserved in some way and continues is very unique and makes it very, very different.

The culture things that we talked about before where culture is built slowly but intentionally here, and that any new thing that’s brought in is thought about from a lot of different angles in terms of how will it impact our culture, and how do we introduce things incrementally over time to take the best of those things, but still preserve what we have.

00:32:18

Ron

Do you have a concrete example in mind of that?

00:32:20

Peter

As a PM, when I first joined, I was really thought of as an “other” in some ways. There was a team of engineers, and then here I was the PM who was a resource for that team of engineers. But I don’t know if they necessarily viewed me as part of the team. I mean, technically I was of course. But that has changed a lot to the point where I think most PMs here are part of multidisciplinary teams that include engineering, include a PM, and in some cases are starting to include other roles like designers for example.

And I think a good example, going back to those communication tools that we built, there were developers internally who were working on and improving those applications. We had a third-party firm that was working on part of it. I was helping to PM it. I was also writing release notes for it. And these are all different parts of the same thing that ultimately affect how people use something, how quickly it’s adopted.

And that has grown a lot over time, this notion that the final product is the result of many people working on it from a lot of different vantage points. But that hasn’t happened overnight. Three years on, this is where we are.

00:33:18

Ron

I do think adding PMs to an organization is a good example of a thing that’s culturally a little complicated. Part of the reason is I think you have all sorts of people who’ve been in other organizations where they’ve encountered PMs and they thought, “Oh wow, the way that PMs work in this organization is kind of terrible and I don’t like it.” And so people worry about bad things from other places getting imposed on them.

And also, the fact that it’s a place where when you’re working on software, you’re not alienated from the users, you’re connected to the users, and part of your job is to think about and understand the problem in an end-to-end way. One of the things that people worried about is us building a structure that disintermediates the developers and has PMs kind of sitting as a wall between one side and the other.

00:33:56

Peter

Oh, absolutely. I mean, the thing that you don’t want to have happen is somebody who comes in and basically adds multiple layers between you and either the product that you’re working on, or the end result of that thing, or the user themselves. What you really want is somebody who cares enough about this to optimize it so you are actually more focused on the thing that you’re building. You have more insight into the user.

And ultimately, if that person were to step away, all those things would sort of remain. It’s lightweight enough that you are building that process into the team’s culture. So we all buy into it. We all want this, because it makes our lives better and our jobs easier.

00:34:29

Ron

That’s right. And for the whole thing to work, you both have to not disintermediate and cut out the people who are doing the software development.

00:34:34

Peter

That’s right.

00:34:35

Ron

But also, I think the people on the development side need to understand the thing that the PM brings to the process. And there’s a lot of insight and understanding that really matters both around forming an understanding of ‘what are the things that we should build,’ and also around how to organize that process and make sure that things happen.

00:34:51

Peter

And how to work together, ultimately. It’s funny, I just read an article called How To Be A PM That Doesn’t Annoy Engineers. And then it linked out to another article that was How To Be An Engineer That Doesn’t Annoy PMs. And it was this flip point of view. And the funny thing is that in both cases, there was a paragraph that said, “Treat each other nicely.” That was kind of the point of the thing. And I think ultimately, it’s about respect and learning how to work together, is a big part of it.

Going back to the earlier question, just to get back to that narrative of some of the things that you think we don’t do maybe exceptionally well, and I think one of them stems from something that I think is really good here, but as we’ve grown gets harder to do, which is to say ‘yes.’ We say yes a lot. We say yes to new opportunities. As you’ve said many times, there could be a new strategy that is very, very exciting that we want to move on very, very quickly. But it doesn’t mean that we don’t have other things we’re also working on.

And this is a thing that happens at Jane Street quite often, where you find teams of people who are very, very willing to say yes to any request that comes to them because they want to help, because they want to enable their colleagues to do a better job. And that’s good, but it still comes at the expense of focus.

So especially when that new big emergent thing comes up, for example, building a new piece of software that’s going to enable communication between teams during a pandemic, if you are dropping 20 things on the ground in order to do that, it’s going to be a lot harder to pick those things up and know whether or not any of those are actually still important than maybe five or six of those things that you are actually just really, really focused on and going a little deeper on.

And sometimes it’s okay to say to somebody, “I understand. I want to help you. I’ll be able to help you in two weeks, a month. As long as that is not going to be too disruptive to you, I will make that time available. I just can’t right now.” I think that’s okay, because what you’re really doing is you’re saying, “I want to devote as much of my time as I can to the things I know are important right now.” And to do anything beyond that would be to dilute that. And so that’s something that I think we’re learning, and we’re evolving, and we’re getting better at, but we still have a ways to go there. But it comes from a good place.

00:36:48

Ron

Yeah, I definitely agree there’s an anti-pattern that we sometimes get into, in some sense, the organization is very ambitious. There’s a lot of things that we want to get done, and it’s very easy to almost reduce your anxiety by saying, “There’s a new thing to do. Great. We will take that thing on and we will assign it to someone.” And you end up with people being assigned to work on many, many different things, and the amount of actual progress you get on any one of those things can be really small.

So this whole thing is almost like a scheduling problem. The worst thing you want to do, if you want to get the maximum value out of your work, is timeshare between a million different projects. Because you don’t get any value out of anything until it actually works, and is usable, and is in people’s hands. And if you do a lot of slice of this, slice of that, then the timeline until anything gets done gets pushed out.

And I’ve seen lots of places where we’ve been able to improve things and make things better. We’ve been able to improve our workflows and deliver things more efficiently by reorganizing the work such that instead of having three people, each of them working on six things, having three people working together on one thing or two things, and trying to get the focus to get things done.

And you’re totally right that we’re an organization that wants to say ‘yes.’ You got to also be able to say ‘no,’ because in the end, you got to make choices. This is a trading organization. You got to make decisions. You got to distinguish between the expected value of the different choices, and pick the ones that maximize the value to the organization. Being nice is not a plan.

00:38:07

Peter

It’s true. Being nice is not a plan. It’s a good baseline, but it’s not a plan. Just trying to think about how to organize your day when you have 20 things going on, it’s just like you leave for lunch, you come back. What do you focus on? That’s a challenge.

00:38:19

Ron

So Jane Street’s a pretty different organization than lots of other places. The PM role is pretty different from what shows up in other places.

00:38:27

Peter

Yeah.

00:38:27

Ron

How do you think about the process of hiring people into this role? We’ve done a lot of it. We have a lot more to do. To start with, what do you look for in people who you think are going to be a good fit for Jane Street?

00:38:38

Peter

Yeah. I think the thing that I said before is absolutely critical, which is somebody who’s willing to meet the team that they’re about to join, where that team is, and is empathetic to them and their process. That I think probably trumps any other attribute for a PM that we hire.

I mean, we look for particular traits. Obviously, having done this work in some manner before is important. Having domain knowledge is really, really helpful, though not necessarily the determining factor. Having a technical background sometimes helps. Not always, but sometimes it does.

But really the thing that we look for more than anything is this kernel in somebody where they are able to really listen, really see how a team operates, be able to fold into their process as much as they can before they actually become an agent of change, and then figure out how to be an agent of change in a way that is going to make that team better rather than alienating them or feeling like something disruptive is happening. These are intangible qualities and are sometimes hard to evaluate, but we really try in our process to find people who have that.

00:39:37

Ron

Yeah, the evaluation story in general in interviewing is really hard. A lot of the interviewing that I’ve been involved with on the engineering side has been organized around trying to build little simulations of what it’s like to work together. Let’s work through a problem. Let’s talk through some of the designs on things you’ve built in the past. But generally, let’s try and do something that feels like simulating work. And I don’t know really how to simulate work in a way that pulls out how empathic are you, how much are you willing to dive in and do the anthropological work? How do we find the right kind of anthropologists?

00:40:06

Peter

This is an excellent question, and something we’re refining a lot over time. I mean, obviously when we’re evaluating candidates, we have to do the baseline skill set discussion. “Have you done this before? Do you know how to do this? Do you know what these words mean?” Things like that.

00:40:19

Ron

And what are those baseline skills? What do you think of the core technical skills for this kind of work?

00:40:23

Peter

The baseline skills are, have you been able to sit with a team, break down their work in such a way that you know can give it visibility, you can effectively track it over time, you can evaluate risk, you can focus on delivery, you can take the lessons that you learn from that project and apply it to new ones over time, you can think holistically about a product, and that you know how to work with people with totally different skill sets as you? I think those are the things that we try to measure in somebody, in the basic skill set conversations.

To get to your question about empathy, the other thing that we’re trying to really understand is, how well do they listen? We do a lot of interviews where we will pose a problem to somebody, and we’re not necessarily looking for a solution. We’re looking for, how do they dig into that problem and find out how we as a team really want to work together, and figure out working towards a solution, not just coming up with a solution. So we do a lot of that.

00:41:13

Ron

That echoes a lot how we think about engineering interviews, because we’re very much not playing algorithm bingo with people. We don’t want to know whether, “Do you know this particular technique?” We’re looking for things like, how quickly can they understand the things that we’re saying? How well can they explain back to us? How well do they model their own sense of confidence? Do they know when they’re making mistakes and when they’re not making mistakes? How comfortable are they being wrong? I mean, there’s core technical skills you want to evaluate at the same time, but there’s a lot of human stuff that’s really critical that we’re looking for in the context of these technical interviews.

00:41:42

Peter

I imagine that in both your technical interviews with engineers as well as our interviews with PMs, one of the things we’re looking for is how quickly somebody updates when they get some new bit of information and pull it into their worldview in the way that they’re talking about something. And that’s something we really look for. If we put a spin on a problem and we see that somebody is sort of banging their head against the same wall that they have, that’s sort of a signal to us that maybe they haven’t been listening as well.

And that’s something that we really want. We want somebody who’s going to come here and really be able to listen, really be able to show empathy to the people that they work for, and dig into a problem in a way that feels collaborative and not prescriptive.

00:42:17

Ron

Do you think about writing as an important part of the skillset?

00:42:20

Peter

I do, from a few different perspectives. I think on one hand, this is a generic answer, but good communication skills are critical for this work. Being able to write effectively and to summarize the problem space that you’re working in is really, really important.

But I also think of writing as a way to figure out how you’re going to solve a problem. And I think that somebody who is a good writer is somebody who, as they are writing, they’re thinking about the way they’re solving a particular problem. And somebody who at least can think like that in the process of building a narrative around whatever project they’re working on or the product they’re trying to build, they are finding ways of refining that and making that better, and scoping it correctly. Having that be part of your process for actually digging into the work that you’re doing I think is incredibly important. So yeah, I think writing is an important skill.

00:43:05

Ron

So you and I have talked a bunch over time about this question of how much technical background is required for this kind of work, and how much it’s helpful and where it’s helpful.

I’ve thought about this a lot from the perspective of the developer tools work, where from my perspective, it feels like having a technical understanding of the tools of software development seems to me relatively important in terms of building empathy for the users. It’s much, much easier to engage with the products that are being made if you can directly go in and use them. If you have some direct intuition for what it’s like to build software, one of the problems that you run into when building software. For that particular role, at least it nudges in the direction of someone whose background is more technical.

But we have a lot of great PMs here who have very non-technical backgrounds. We have some whose backgrounds veer more in the technical direction. And I’m kind of curious how you think about that all, and how do you think about finding the balance of D. E. Shaw getting a bunch of creative people with all sorts of interesting creative backgrounds to think about problems in different ways and approach it from other directions, versus getting people who have the domain knowledge that helps them navigate the space?

00:44:11

Peter

Yeah, it’s a very good question. One thing I will say is that Jane Street is methodical in the way we onboard people in order to build domain knowledge as much as we can. Even engineers who join Jane Street sometimes arrive and require some period of time before they actually are comfortable in the particular code base that they’re working in, for example, or on the particular product that they’re working with.

I think that’s true of PMs as well. Whether we have a technical background or not, that slow, methodical onboarding of learning context, and understanding the tools that people are using, and why they’re building them, and what they’re for, and really digging into that is important. You’re not expected to come in and know your products, and build something out on the second week of joining. That goes a long way whether you have a technical background or not. Even if you don’t have a technical background, there are opportunities to learn more and to get more technically proficient here.

So for example, we have an OCaml bootcamp that every developer, every engineer goes through when they join. But anybody who works at Jane Street can go through that bootcamp. I even went through it and I did a self-guided version of the bootcamp, and that was an incredible opportunity for me to understand this very, very important part of our world, and be able to work with engineers, and understand their workflow more.

Now, you’re right. I am not sitting in the code every day. I’m not coding at all. But I can review somebody’s feature very, very easily, and I know the entire release cycle that they go through every single time they put something out into the world. And that goes a long way with – what I hope – is my existing well of empathy to be able to work effectively with people.

Regardless of your background, I think empathy is a good core skill set. If you don’t have that, you could have the most technical background, and still be a PM, and not be as effective as somebody who does not have that background and is willing to do the work to try to understand how people build things and why, and really partner with them.

00:46:02

Ron

Absolutely. Empathy’s a pretty important skill for software engineers too, I think.

00:46:05

Peter

I think so.

00:46:06

Ron

I want to jump back to a different thing we were talking about before. So we were talking about various problems with how Jane Street operates, things that we want to improve on. One thing that has struck me as a real challenge for lots of teams is figuring out how to handle technical debt. It ties again back to your point about people really wanting to say ‘yes,’ and get things done, and accomplish new things. But it can be really hard to balance the endless stream of requests that come in. There’s a growing stack of work, and there’s kind of a tyranny of that stack where it can be hard to look away and do anything else.

And at the same time, if you are not thinking about what’s imperfect about your platform, retiring technical debt, cleaning things up, just figuring out fundamentally better abstractions and platforms that you can build things on, then you’re just going to find more and more grit in the wheels, and things are going to slow down, and you’re not going to be able to achieve as much.

So how can a PM help in balancing that and giving people the space to retire technical debt where it makes sense, and to make fundamental improvements where it makes sense? And at the same time, not lose sight of the importance of delivering new functionality for the business.

00:47:09

Peter

Yeah, this is a very good question. What you’re talking about is a few things. One, there’s just a pure quality of life issue that you’re talking about, which is if you’re ignoring tech debt just so you can focus on future development, it’s going to be incrementally harder and harder to do that future development, and you’re going to get bogged down more and more. The other part of it is, I think most PMs here are very much champions of their teams and don’t want their teams to have those quality of life issues. So one very concrete thing that we do is we try to take the long view of what the team is working on and find areas where we can actually just focus on tech debt.

So a good example is I work very closely with our Storage team. And every six months or so, we do what’s called a toil reduction sprint where we take two weeks, and we just focus on ways to automate operational toil. And the team curates a backlog of small automations that they can do. Maybe it’s creating a directory for your application in a way that is a lot more automated. It takes that away from the engineer who wants to go off and actually build new clusters, and think about tooling, and things like that.

So that’s something that we do. And in fact, we’ve done that on a group-wide level. In my group, Core Services, we’ve done a toil reduction sprint that dozens of people have contributed to.

And in fact, the interesting thing is some of our most important tools have come from those sprints, that people across the firm use now in order to build out applications, or request resources, or any of these things that we thought of as taking a lot of time from our day, and bogging us down, and contributing to our tech debt, but actually are things that are used across the firm.

And it’s also really a great way to build camaraderie across your team. You feel like you are accomplishing something for your team. This is something that you complain about every day and you can get rid of it. And I think that that’s really, really important to bear in mind.

And that’s something that I think a PM can be helpful with is knowing that, “It’s been a while since we focused on this.” And maybe in this little break that we have between projects, we can just do that for a little while instead of spinning up a new feature.

00:49:06

Ron

I struggle a little bit with this framing of, there’s the things that you’re delivering for the end customers, and then you want to reduce tech debt for the team? Because all the things you’re talking about reducing the manual labor that people need to do and the frustrating steps that they have to do every time some new thing happens, some of it’s about making the team happier. And I should say, happiness is like a first order concern. Right?

00:49:25

Peter

Absolutely.

00:49:25

Ron

You want the people here to be excited and happy to come in. People do better work, and do more work, and just stay at the organization for longer if the work is fun. But I think a lot of those things are also valuable for delivering to the rest of the organization. If it takes you less time, if you’re doing less toil, there’s more work that directly serves other people’s needs, that you can then get done.

One of the things I struggle with is, how do you convince people to think about these kind of toil reduction steps as not just a thing where they’re taking care of themselves, but a thing where they are optimizing the machine of their own work, which is an important machine. Their time’s really valuable, the ability to deliver things. It’s not like, “You have some extra time so you can knock off an hour early and go get a beer.” You’re making things faster so you can get more done.

00:50:09

Peter

Right, yeah. Just like anything else, it’s curation. You could for example, spend those two weeks just doing a handful of issues that you logged that you thought were particularly toilsome, and maybe you feel good about it, and you accomplish something and a few things get wiped off your list.

But ideally, if you’re doing it right, these are recurring themes that you’re coming back to. You’re seeing patterns in the way that you work that you know are particularly toilsome, or are taking additional time, or are causing frustration for your end users. And that sort of cultivation of those themes over time ultimately result in something that is very, very valuable you can get done. You could do 20 small little things that get rid of toil and in multiple places, but it’s probably more effective if you’re focusing on two or three things that are really, really deep. You’re going in, you are providing some value to yourself, you’re providing some value to the people who use your product in some way. And you know that you’ve made the world more holistically better. You’ve also made your time more effective in the process. You think about things the same way, but it’s just the lens is slightly different.

00:51:11

Ron

So let me switch gears for a bit. There’s another project that you and I have worked together actually a bunch on recently. So let me give a little bit of background.

So Jane Street has been doing machine learning of one form or another for a long time. I started more than 20 years ago doing research into statistical development of new trading strategies using all sorts of techniques that are totally out of fashion now. And in that 20 years, we’ve built out a whole Research team and lots of people doing research on individual trading desks.

But in the last few years, there’s been a shift in the set of techniques that we’re using, focusing much more on deep neural nets. And as a result, requiring us to really build out our infrastructure around training and executing those neural nets. And so in particular, we’ve done a lot of work on GPUs.

00:51:53

Peter

Yeah.

00:51:53

Ron

You got involved in that because a lot of the work that enables that comes out of Core Services, which is the team that you’re working on. And I’ve been thinking about this from the point of view of a team called Research and Trading Tools, which is responsible for building the kind of direct tools that support people in doing machine learning. And really in the last six months, there’s been a real shift in our focus there where you said, “Actually, you know what? This is a really big deal and we need to move a lot faster, and move in a lot of different directions at once.”

So there’s doing more in terms of our on-prem footprint of buying a ton more GPUs and figuring out the right kind of machines and setups for installing them, using them efficiently internally. Figuring out how to effectively leverage the large amount of cloud resources that are out there. Learning a lot about the details of how to do this all efficiently, and well, and at scale.

You talked before about storage. Storage is actually a big deal. Getting data into GPUs is really critical. GPUs are super data hungry. And so how do you set that all up effectively, is an important thing, and a thing that we didn’t know that much about. The scale of our GPU utilization was a lot smaller. And so we had a ton to learn, and we shifted the whole organization to have a lot of focus on it. It’s one of these emergencies where we really shifted gears and focused in a different way on this one thing, and tried to get lots of different parts of the organization at the same time to support this one buildout of technical infrastructure. And I’m curious how you think about enabling a shift like that from your role of a PM. What’s the role of a PM in making that kind of change happen effectively?

00:53:23

Peter

Yeah, it’s a great example. Coming from my vantage point, my team first encountered this by just knowing that there was somebody in the firm who was just buying more GPUs, and we needed to put them in our data centers. And that was maybe the first hint that this was a big effort.

I even remember I had a conversation with a colleague, and at the end of our meeting he said, “Last thing, I think we’re going to need to think a little bit more about GPUs.” And then he hung up the phone and that was kind of that. And then literally the next week was deep dive into thinking about GPUs and the infrastructure we needed to build them. So part of that is trying to get better at doing that thing, which is instead of just being reactive and knowing that there’s somebody at the firm who wants to buy a bunch of GPUs, and stick them in our data centers, think about why. And think about ultimately, what they’re trying to accomplish.

There were a number of conversations I remember you and I having with another colleague, Liz, who sits on Core Services as well, where we’d sit in the room, and you would map things out on a whiteboard. And it could have gone any number of ways in terms of what we focused on, in terms of whether we’re going to focus on on-premises resources, or in the cloud, how we’re going to schedule jobs to run on these GPUs.

00:54:29

Ron

Which concrete networking hardware we wanted to support it.

00:54:31

Peter

Exactly. And this was multiple narratives happening all at once. And I think the thing that we always left the room with was, we don’t want to be just at the receiving end of these requests. We want to be in the room when the decisions are being made.

One thing that PMs can do is to be an enabler of those conversations, and ideally to be a part of them. Because (a) it makes our job a lot more clear in terms of what we’re trying to accomplish, (b) it makes this kind of interaction more of a partnership, as opposed to a customer, and we’re at the store that is getting you the thing that you want. We can scope it a lot better, we can help guide that. And so that the teams that we are supporting don’t feel overwhelmed. They know what’s coming. They know the full timeline of things as much as we know them. They understand why they’re working on it.

Knowing that there is a set of teams that are working on many, many different things for the firm at the same time, and that this was an emergent problem that they could potentially look around and say no to a few things in order to focus on that, that has a huge impact on morale. It has a huge impact on knowing what you’re doing and why you’re doing it. And I think it’s just ultimately helpful for getting that project done completely, rather than many people feeling like maybe their time is being taken advantage of or you have somebody who’s asking many things of them all at once.

00:55:41

Ron

Yeah. It’s really important when you ask people to say yes to something that you understand that it means they’re going to have to say no to some other things.

00:55:47

Peter

Yeah, absolutely.

00:55:48

Ron

To make that happen.

00:55:48

Peter

Yeah. Yeah.

00:55:49

Ron

I think from my perspective, I think one thing I saw you guys do from the PM side is get people to explain themselves. And you were kind of mentioning before about how some of this is about making people feel like they’re on the same page. It’s just enormously helpful to getting to a good answer.

I think a common problem you run into is somebody wants something and they say, “Here’s what I want, and I want to buy the following thing. I want to build the following software, the following precise feature. Please go do this.” Nine times out of 10 when someone says, “I want precisely X,” and you try and build X for them and you give them X, it’s like “Ah, they didn’t really want exactly X. They wanted something else.”

And getting people to explain not just what they think they want but what they’re trying to accomplish, and a kind of more fulsome understanding of, what are they actually confident, and not confident about? What are the pieces that they know? What are the places where they’re just guessing?—makes it way easier for a larger set of people to understand the request and to be able to suggest alternatives like, “You think you want the following thing, but that thing turns out to have 98-week lead times. Should we really order it, or should we understand why you wanted that and see what else is possible to obtain instead?”

00:56:51

Peter

If we’re being honest here, maybe you want a thing, but maybe you’ve never used that thing before. And maybe you want to experiment with a smaller version of that thing before you buy the big shiny version of it, so you can actually understand whether or not it’s the right thing. What you want today might not actually be what you want six months from now, and we can help you get there a little bit slower and incrementally, without making decisions that we have to ultimately unravel at some point down the road.

A good example of this is, as you mentioned, the storage aspect of this. We could go out and buy a big, very, very powerful, very performance storage cluster specifically for GPU workloads. Or, we could use the storage we have here on a smaller networked set of GPUs, just to make sure that you actually are getting the thing that you need, and we’re not buying the thing that you think you want.

So that’s an example of small experimentation, asking the right questions in those meetings when you’re actually trying to make those decisions to understand what of the many things we’re trying to do really is the most important at any given moment, so that we’re not thinking that everything should be dialed up at 100%, and that we are building slowly and smartly so we can ultimately get the thing that we want.

00:57:54

Ron

Right. And I guess part of the point here is you’re not just optimizing for the end result, because you don’t know what the right end result is. You’re optimizing for the learning rate. And in part, we don’t know what the end result is because ‘Oh, maybe we’re not familiar with this hardware platform. We need to experiment with it some.’ In part, it’s because we don’t even know what models we’re building. The rate at which the approaches we’re using on the machine learning side is changing is super fast. And so there’s a kind of iterative process where you build some new infrastructure, you try it out, you learn some things, you get some new ideas about what you need, and you keep on iterating on that.

And one thing that I think has been uncomfortable, but in the end I feel like people have come around to, is understanding that we’re doing this in a way that’s guaranteed to make lots of mistakes. In some sense, I feel like the teams here that are responsible for the core foundation are used to building things and delivering them a steel box with bolts all around, that you could drop off the side of the building, it would keep on working.

The Networking team’s a great example. There’s this weird thing of, the network is foundational to everything, and nobody understands it. The only time you go to talk to the Networking team is if something fails and you think, “I don’t know what it is, so it’s probably the network.” I feel like the Networking team constantly has people wandering back saying, “Did something go wrong with the network?” And I’m pretty sure it’s you buddy. Most of the time, it’s not the network. I mean sometimes it is, but it’s usually not the network.

00:59:07

Peter

They take a lot of pride in it.

00:59:08

Ron

That’s right. But they’re very used to building the foundations for the organization, and building things at a very high level of quality. But there’s a real tension there. If you’re going to build everything at a very high level of quality, well, you can’t do it as fast. You need to take more time to really get things right and really machine things down until they work just the way you expect them to.

And in a project like this where there’s a lot of uncertainty, and we’re moving fast, I feel like part of the thing that we had to do was be explicit about the fact that we were making bets. We’re a trading firm, we should be able to make bets. And simultaneously giving people the freedom to fail, to try things out that they aren’t super confident are going to work, to deliver things that aren’t quite ready, and understand that the people on the other side aren’t going to try them for three minutes, say, “This is great,” and then immediately demand that they be ready. Right?

00:59:54

Peter

Right. Or even if when they’re being tried out, that maybe if they fail, that doesn’t mean that the Networking team needs to drop everything and take care of that problem immediately, because you know you’re in a place of experimentation. And so to your point, it’s okay to fail from time to time, and it’s okay for things to not work perfectly from time to time.

But that requires us to all be in a room and be comfortable with discomfort to a certain degree. And I think that takes a lot of trust, and also like anything takes some amount of curation. Because absent of that, I think you would have somebody ask for something, somebody would go off and try to build it in the way that they know how to build it, with the right kind of monitoring support and everything like that, that they wanted. And maybe that would take a lot longer for them to build, and maybe when you get it, it wasn’t the thing that you actually wanted.

Being able to stay in that discomfort to a certain degree, and be there together and cohesively as a group of many people trying to accomplish the same thing at the same time, I think that’s something that PMs can be really helpful with.

01:00:50

Ron

A little more concretely, as a PM, how do you help with that? What are the things that you’re thinking about to try and drive us to think about the problems in the right way?

01:00:58

Peter

Well, for one thing, we all have different domains that we all care about. So somebody in Networking really, really cares about the particular network design that they’ve implemented in all their data centers. And it is that very, very stable thing. And maybe they’re thinking about a project from that perspective, and maybe you as a researcher are just thinking about what models you can actually run on the GPUs that we bought.

Now, being in the room and having a shared understanding of what the end state is, and that we’re all responsible for it, and that the decisions that we make can’t be absent of each other, is very important. PMs can help create that framework where everybody understands that.

And I think without that, it’s harder to get there. You have your own set of concerns that are all very, very important. But sometimes you need somebody to say, “Okay, we can put those aside for now. We are a team of people who all care about this one thing. We might have different concerns. We might have very different disciplines, but we’re all going to the same place. And we need to understand each other and not leave the room with any assumptions about what we’re actually building and why.” Driving towards that, I think, is very important. Having somebody who really cares about that, who really wants that more than anything, is very important. I think a PM does that.

01:02:05

Ron

Awesome. Well, that seems like an excellent place to wrap it up. Thanks so much for taking the time.

01:02:10

Peter

Thanks for having me.

01:02:11

Ron

All right. Cheers.

You’ll find a complete transcript of the episode along with links to some of the things that we discussed at signalsandthreads.com. Thanks for joining us, and see you next time.