I recently sat down with Andrew Gorham for an interview and case study of progressive summarization, a method of summarizing notes in a form that is easily discoverable in the future. It is one of three techniques I teach in our online course Building a Second Brain.

Andrew works for BMNT, an innovation consultancy that introduces innovation practices to national security organizations, including the Special Operations Command, the Marine Corps, Cyber Command, and over a dozen others.

He also teaches progressive summarization as part of a Stanford course called Hacking for Defense, in the Technology Ventures Program of the Department of Management Science & Engineering. The course introduces undergraduate and graduate students to “lean launchpad” techniques, applied to solving national security problems. Here’s a short video about the course.

I thought we were going to stick to note-taking, but in just 30 minutes we somehow also touched on innovation in large organizations, conducting customer interviews, equipping Navy SEALs for missions, the importance of rapid iteration and testing with users, exceptions to “just-in-time” summarization, and the potential role of natural language processing in knowledge management.

Watch the video below, listen to an audio-only version, or read a lightly edited transcript (progressively summarized of course!) at the bottom of this post, including links to resources mentioned.

Transcript

Tiago Forte: OK. So here we are doing a case study on progressive summarization with Andrew Gorham. Thank you so much for joining us Andrew. It’s a real pleasure. Why don’t you just start with a little introduction, like your background, what you do now, and how you came across progressive summarization, just so people have the context.

Drew Gorham: Yeah Tiago it’s great to be here. Thanks for bringing me on to talk about how we’re using progressive summarization at the company I work for right now, called BMNT, and basically what we have done, what we’re trying to do, is bring lean startup principles into government and defense, national security problems. And so we’re taking a new approach to the innovation that goes on in that space.

But you asked about my background. So I came out of the app gold rush, you could call it, back in 2010 or 2011, when everyone was building apps. I started as an app developer and then moved into design, eventually started my own company called the App Factory which is basically an app development shop that helps non-technical founders build a product, and built about 100 or so apps over a four-year period. And what I realized over that time is that I had the easy job – building the product is really easy – and the really hard part is validating that there’s a real business need out there, that there’s a real market, that there’s customers that have a pain point that you were actually solving.

So after many failures of building apps that weren’t solving a real pain point, I was just looking for answers like, “Why was this happening?”, and got turned onto Steve Blank’s Lean Startup methodology. The whole idea of, you can’t come up with a solution for a problem in a room like this with whiteboards on the wall. You have to get out of the building. You have to talk to customers. And really at the early stages of an idea your goal should be to maximize your speed of learning. And so I started incorporating that into the work I was doing with non-technical founders and started to see a lot more success that way.

And I got pulled into BMNT about two years ago where they were noticing the same problem in these massive bureaucracies. I mean the largest human organization on Earth, the U.S. Department of Defense (DOD), with something like a 600-billion dollar budget. It’s just insane. They’ve got massive problems that they’re trying to solve. And they’ve got this legacy approach that is really committing all of the mistakes that Silicon Valley learned back in the late 90s, early 2000s, really their problem is they’re solving the problems of 10 years ago with technology of five years ago to be fielded into operation, you know, five years from now.

So the thesis at BMNT is that the key to our national security is not going to be the robustness of our defense infrastructure. It’s really our ability to adapt to change. And all of this rapid technological change that’s coming out after us. It’s really incumbent upon us to increase the speed at which we solve problems because the problems are changing all the time. And so our founding partner Pete Newell, he came from the Army’s Rapid Equipping Force where they would build prototypes out in the middle of Afghanistan to rapidly solve problems over the course of weeks rather than years. He came out to Colorado and basically had a collision with Steve Blank, the Lean Startup godfather, and they created this methodology called Hacking for Defense, which has turned into a university course, first at Stanford.

But we’ve also commercialized it into a kind of innovation methodology that we’ve since expanded to “Hacking for X” where X can be defense or healthcare – any kind of large bureaucratic organization in a highly regulated environment. We’ve taken the Lean Startup approach along with all the other great design methodologies, like design thinking, total quality management, all of these things need to be a part of how we solve problems in the future.

But to tie back. One of Steve’s big things is getting out of the building, talking to customers, doing a lot of interviews with people who share the pain point you’re trying to solve. And one thing we’ve struggled with is how to compress all of the data we’re gathering from these interviews and then make sense of it in the future. And the unique thing, the hard thing about doing customer interviews is that it’s really hard to be listening to a conversation in real time and also deciding what is worth writing down and what is superfluous or not really relevant.

So we’ve figured out that the best note-taking strategy for customer interviews is to write just about everything you hear. I mean it’s not quite a transcript but it’s very close to a transcript. So you’re just typing at the speed of sound and and we need a way to compress that down and so that’s where progressive summarization has become useful for us.

TF: What a fascinating journey you’ve been on. Wow. Gosh, so many ways we could approach this. I couldn’t agree more. It’s so hard to do multiple mental operations, to listen and to think and also to judge and to decide and filter and distill. It’s like people think that isolating their environment – I’m in a phone booth pretty much right now – they think that will keep them from multitasking. You can multitask right here in your head.

AG: Oh absolutely. Yeah that’s a great point, especially when you have a limited amount of time with these folks. You know you’ve got maybe 30 minutes and you know they have really good information that you have to extract. And it just takes so much mental bandwidth to decide where to take the conversation, to allocate some of that to capturing, it’s just overwhelming.

TF: Could you say more about the environment? Are you just sitting down one on one with potential customers of defense products or defense contractors or are you doing a workshop? What is the scene?

AG: Yes. So we are working with folks in government. They might be with the Marine Corps or government employees working at an intelligence agency for instance. And usually it’s over the phone. I mean ideally you’d be face to face or from there, slightly less good would be a video interview, and below that would be a phone call. But the folks we’re interviewing are often distributed all over the world so we’re forced to resort to a phone call, and they’re behind all kinds of technology firewalls where they can’t open a Zoom conference like we can because it’s restricted on the network. So we have to talk to them over the phone.

It’s typically one person driving the conversation who we call a “problem curator.” So we collect all of these problems from our customers. We’ve got a backlog of hundreds of them from across the DOD and across the intelligence community, and we pick one out and we figure out who are the “key beneficiaries” – who are the people that if you were to solve this problem for them, they would run up and give you a big hug. They’d be so excited to have this problem solved. So we talk to those folks, typically over the phone, and the person driving that conversation we call a problem curator, and sitting next to them is basically a note taker. And yeah like I mentioned we figure the best note-taking strategy is to just write down as much as you can and decide what’s relevant later when you can free up that mental bandwidth.

TF: I love that too, it’s like a tag team. So one person can be totally present and the other person can be running around, at least mentally, trying to pick up all the pieces and probably make sure that the details are filled in, that’s very interesting.

AG: We’ve tried doing it with a single person driving the conversation and taking notes and what we’ve found is it’s just way too much, and the quality of the conversation, that’s what really takes a hit. It’s just not worth it.

TF: So true. I loved what you said about problems. In fact, this is one of people’s favorite exercises from the Building a Second Brain course, is the 12 Favorite Problems exercise. Essentially I have people list their favorite problems. What are the problems they love solving? Which typically appear again and again and again in their life. In different permutations and different manifestations. I just realized this has a relationship to customer development, which is, most people specialize in particular solutions. So they have their hammer and they go looking for their favorite nails. But if you specialize in a problem, but you’re agnostic to how you solve it, that is powerful. That’s way more latitude that you have in recommending things to clients. Is that what you find?

AG: Yeah. That is. That’s one of our core value propositions to customers is getting them to break out of that mental model of being that hammer in search of a nail, which has been the default mode in this environment for decades, and I’m sure there’s a whole bunch of reasons behind that. But coming up with real problem statements and saying your problems out loud is actually very difficult and in a lot of environments it’s not rewarded. So we’re trying to flip that and force people to define their problem and who it’s affecting. And what we’ve found is that no problem statement survives first contact with these beneficiary interviews. They always change.

I mean the classic example we have, and this came out of the Stanford courses, they were working with the Navy SEALs and one of the medical officers who was in charge of making sure the Navy SEALs were in top shape when they would go out on missions. He was like, “Hey our SEALs are getting hypothermia because they’re underwater too long. So my problem is that I need them to wear a wristband that keeps track of their vital signs – their heart rate, body temperature, stuff like that.” That was the problem Version 1.0: “I need this wrist monitor.” And the team went out and they talked to the Navy SEALs and were like “Hey guys, if we gave you this wrist monitor how would that affect your day to day, your job down there?” And they were like, “Well, if you gave me that wrist monitor, first thing I would do is throw it at the bottom of the ocean, because I don’t want to be pulled out of a mission because someone in a back office says my heart rate is too high.”

And what they discovered was the actual problem was that GPS navigation is nearly impossible at this significant depth underwater. So the SEALs couldn’t find where they were underwater or they weren’t able to map out their route. And so once they figured out that was a problem they built a simple prototype, which is basically this little GPS unit in a waterproof container tied to a string. It’s literally like a GPS buoy that they would unravel. It would go up to the top so it could get a signal periodically. So now they can locate themselves underwater and, I mean, talk about a pivot. That’s just the power of validating a problem through interviews with real problem owners, I guess we’d call them. And that wrist band probably would have been like a 500-million dollar contract.

TF: Useless solutions are very expensive to develop.

AG: Yes. Especially when they take years and you don’t show them to the end user until you’re three years into development.

TF: So I’m wondering, how do we tie this back to progressive summarization? So once these two people who are in the interview take the notes, and then they capture everything, they write everything down, what does the process look like? I mean do you take them by hand in an app and then store them somewhere? What is the flow?

AG: What I’ve found is it’s way too difficult to transcribe the conversation by hand in real time. It’s just too fast. You have to type it out. So typically in Evernote there and just furiously typing. And what I found is, I like your approach of,  don’t use progressive summarization until you need to. Right. But what I found is that when the conversation is still fresh in my mind, that’s the best time for me to try and pull out the relevant parts of that conversation. So typically we’ll take like 10 minutes at the end of an interview to just go through the different layers of bold and highlight.

TF: Interesting. I wonder if that’s a potential exception to doctrine. Because with text that was written to be static text, it doesn’t matter when you summarize it, because it was created to be easy to preserve across time. But I’m guessing, do you find that with a conversation there’s all this messy contextual stuff, like their body language, their tone of voice, the feeling in the room, that you can’t quite write down but is present there? So by doing this summarization you’re at least capturing a little bit of that. Is that what’s happening?

DG: That’s a great point. I never thought about it that way but that’s so true. There’s all sorts of information in the tone of voice that is only in your working memory. And if you wait too long that will probably be gone, so maybe you are encoding that a bit in what you decide to summarize, what you decide is relevant. Yeah that’s a great point.

The other thing we’ve been sort of experimenting with is putting our notes through some natural language processing to see if we can pull out some interesting connections across these interviews. So you’re focusing on this one problem and you might interview five people. So by capturing real language…if we were just typing notes it wouldn’t really be natural language. It would be sentence fragments. But by capturing whole sentences you’re able to use these powerful techniques…and pulling out names and organizations. So I’ll keep you posted if that turns into anything. But I think adding the progressive summarization into that data model could provide another layer of relevance scoring, you could say. Have you heard of anyone experimenting with that or combining progressive summarization with natural language processing?

TF: I’ve seen some small experiments…this isn’t quite natural language processing, but I think one of the first things to be automated…I’m looking at the building a second brain methodology, there’s all these manual processes. And one at a time they’re going to be automated. It’s not like one day there’s going to be general AI and the whole thing is going to be automated. It’s going to be one piece at a time. It’s going to be like a modular process. So I think one of the first things is just going to be getting a text down to its essence. There’s a service called SMMRY, and I had this experience where I progressively summarized manually a text, and then I did SMMRY, where you basically just have a slider, and you input the text and you say, “How short do you want this to be?” And you can slide it down or you can slide it up.

DG: Really? That’s cool. I’m going to check that out. I have a note with the two side by side. I’ll post it when I post this recording.

Extras

Here’s a quick summary of the process Andrew used to teach progressive summarization at a U.S. intelligence agency:

The Process

We had 5 participants total. Each participant received a physical copy of the book Talking to Humans, by Geoff Constable.

It’s quite short, so they were able to extract Layer 1 in ~90 minutes. Some had their laptops open while reading and typed directly into Evernote. Others kept their laptops closed and copied notes onto Post-its, then transcribed the Post-its into Evernote afterwards (the former group finished faster and enjoyed the process more).

Once everyone had their Layer 1 complete, I explained the concept of Layer 2 and each participant bolded the passages that really stood out. This took ~10 minutes.

Then I used the same approach for Layer 3 — that took ~5 minutes.

By the end of the 2 hr exercise, each participant had familiarity with progressive summarization and their own personalized summary of Talking to Humans. These summaries became a great resource material for them later in the week when they got outside the building and ran their own customer interviews. Also sparked a lot of interest in the BASB movement.

Thoughts

I see a lot of applications between BASB and intelligence work. 90% of the intelligence community (~100,000 people) are pure knowledge workers.

They do Progressive Summarization to the extreme — a 15-page report is summarized into a paragraph when it reaches a Senior Leader, that paragraph is summarized into a color (red, yellow, green) when it reaches a commander. It’s kind of absurd.

Lots of interesting overlap I’ll continue to explore!

Slides

Click here to view or download the slides used in the exercise above

Follow us for the latest updates and insights around productivity and Building a Second Brain on Twitter, Facebook, Instagram, LinkedIn, and YouTube. And if you're ready to start building your Second Brain, get the book and learn the proven method to organize your digital life and unlock your creative potential.