Welcome back to Pentester Diaries. In this episode, longtime Core member and Cobalt Research Manager, Robert Kugler talks with Grahame Turner, an experienced security technical writer, about report writing, why it’s important, and tips on how to improve your writing as a pentester.
While many pentesters may say that report writing is their least favorite step in their pentesting process, Robert and Grahame will explore how this seemingly "boring" pentesting activity can be insightful and an interesting opportunity to showcase expertise as a pentester. Let’s dig into it.
Pentester Diaries is multi-formatted, with audio, video, and written versions of the episode below.
Watch the video podcast here:
Prefer to read the episode? Below you can find a transcription of the podcast:
Robert Kugler: [00:29]
Welcome back to the Pentester Diaries. My name is Robert Kugler, and I'll be your host for this episode. Today, I'll be talking with Grahame Turner about report writing. Thanks for joining us, Grahame.
Grahame Turner: [00:40]
Thanks for having me. It's good to be here.
Robert Kugler: [00:44]
Report writing is like a super boring, or maybe exciting topic so that's why we have you today. We'll dive a little bit more into what makes it exciting for you and how can we maybe, as pentesters, look at it a bit more positively instead of the most boring part of the whole pentest process? Let's get started here and maybe you can tell us a little bit more about yourself?
Grahame Turner: [01:12]
I am a senior technical writer for Cobalt. I work with the PenOps team on mostly editing reports and templates that make up reports. It's all reports all the time. I've been doing this for about four or five years now. I started with another pentesting company. Prior to that, I've kind of bounced around from corporate training, to journalism, to technical writing for a financial company, to a couple of stints in retail here and there. I've kind of been all over the place, and then kind of settled into security because it's fascinating, but also it's linguistically interesting.
Robert Kugler: [01:58]
Wow! I guess that's a pretty long journey coming from finance to pentest reports. Interesting one. What is the most exciting thing for you when you're looking at the pentest report? Because that's probably what made you stick to security here?
Grahame Turner: [02:17]
That's a good question. My skills, I wouldn't say are extremely technical. I've got sort of a passing understanding of a lot of things or at least enough to kind of make a mess and then fix it. But it is kind of interesting to just see the interface between people and computers. The fact that, on one hand, computers can do things that we couldn't really dream of, but then we can also pull the wool over their eyes and make them do things that they shouldn't do. Like, give people access to stuff they shouldn't have access to. In extreme circumstances, I've seen network tests where pentesters were able to look at the CEO while they were working on their computer. As smart as computers are, they are not thinking about these kinds of security things that we worry about. So, it's very interesting to kind of see that interplay. But it's also just kind of fun to work with testers from all around the world and get to know kind of what they're interested in, what their skills are, and how they make computers do stuff.
Robert Kugler: [03:30]
Definitely. And that comes right to our topic today with pentest reporting because it's important to be able to tell that story. It doesn't really provide any value if you just break stuff and you can't tell the story of how you got there, and most importantly, how you can fix it. That's super important. For you, what’s the most important thing that a pentest report should have? Then, we can later look a little bit at the findings individually.
Grahame Turner: [04:00]
I think the findings are definitely a very important component. I think the really important part of a report is that it's got to tell the story. Because a pentest - yes, it's a series of individual tests, and it has a scientific aspect of ‘we tried X and this didn't work. We tried Y with a different variable and that worked.’ So, you've got that, but then you're also trying to make it accessible to people who don't have the same level of technical background.
I may not know JavaScript version 1.3 or whatever - I may have just made something up that probably doesn't exist - and, I may not know what the specific vulnerabilities are, but I know that if I'm looking at an outdated piece of software, I do know what that means. When I'm looking at a report, one of the things I'm trying to find is, what is the stuff that I know generally people can understand? What's the technical aspect that underlies it and how can I make those two things speak to each other so that people who are reading it - even if they don't have a super technical background - can at least pick up like this is bad for this reason and I need to fix this by asking somebody who knows what they're doing to do X, Y, and Z.
Robert Kugler: [05:24]
You touched upon a pretty important topic here and that we are facing as pentesters all the time. When you're writing the executive summary or executive analysis, we tend to forget sometimes that we have a completely different audience here. That we are not actually talking to the security engineers but probably more C-level. How can we make this as understandable as possible?
Grahame Turner: [05:47]
I think some of it is thinking about the level of granularity that you need to get into. The version numbers are a good example - even if my example of JavaScript wasn't a good example. The specific version of a program for an executive probably is not going to make a difference, but the fact that it's past the end of life and is not going to receive any updates, that makes a difference to someone in these executive positions reading this report. To explain a story, a word I used earlier. The story, for me, is important because it's something that humans naturally do as we try to explain things. That's why fiction continues to exist because it helps us understand things, and framing things in a narrative structure helps. For example, saying that we started by doing X and that didn't work, then we tried Y, that worked and we got this far. I wasn't able to break your system completely but if somebody had more time and more resources, they could probably do more damage, for example, A, B or C.’
Robert Kugler: [06:56]
For writing that story, that puts everything together and kind of tells the reader how we got here. Where do you start with that? Do you really write a little summary of it in the executive analysis or do you put it all in the scope of work?
Grahame Turner: [07:13]
That's a good question. I think one of the things that might be useful is to take a second and define these sections. I said earlier that I've worked for a couple of different companies and I've seen these sorts of sections in different reports, and I think having a clear definition for those does help. The executive summary is usually the first section that somebody reads and that is for compliance purposes. To summarize and simplify the section is: It's just a kind of company A came, company A did a test, company A stopped testing on this date, and basic legal stuff Then we get into the executive analysis which is a little bit more of a high-level overview of the things that were tested and things that were found.
Scope of work is a more scientific-based section because we're talking about the things that were tested. This section goes into the details of stating ‘these are the things that were in scope and these are the things that were out of scope and so we didn't test them.’ You will often find a section in here that may include something like ‘we weren't able to test this for this reason’ or ‘we had access issues so we lost a couple days of testing, unfortunately.’ That kind of information. And then we usually have the findings, which are sort of building blocks. These are the individual vulnerabilities that we found, with a step by step guide. And then there is the recommendations section, which is the high-level ‘fix this by doing this’ kind of thing.
So, those are the sections and I find that the executive analysis is probably a very good spot to stick in that story, that kind of narrative, because if both of the sections are labeled executive then you probably have the C-levels attention. Since they're paying attention, this is where you really need to drive home that ‘this is bad for these reasons’ so that hopefully your customer can get the thing fixed. And then the scope of work, I do try to keep a little bit more dry, a little more scientific, a little more like ‘these things were in scope,’ ‘these things were out of scope,’ ‘kind of setting the parameters for the test’ and then working within that parameter.
Robert Kugler: [09:23]
I think it's pretty important also what I like to do is add a little bit of a snippet to the executive analysis about the state of remediation just to enable them to see at a later point in time when they go back to their report and call, we like to update this regularly, that they can always get an update of the report, that's super cool and valuable for them. When they look at it later then to see, ‘oh okay, I've already fixed 80% of all of that and there's only 20% left,’ it makes a much better impression on the client.
Grahame Turner: [09:55]
Oh definitely. At the end of the day, the vulnerabilities are the things that need to get fixed. Having that up front, that's very important. There's the phrase borrowed from journalism is burying the lede. You don't want to put the most important piece of information down at the bottom of the report or nobody's going to read it. This is one of the things that happened a lot in journalism. It is one of the things that I had to learn very quickly - not to do because I'd take a really cool and interesting bit and put that six paragraphs down and nobody read it and so my stories didn't do as well until I started throwing that right up front. That's why people clicked on the headline because they wanted to know what happened.
Robert Kugler: [10:36]
Exactly. And now this is kind of counterproductive. Why do we put the recommendations at the end then?
Grahame Turner: [10:46]
I've seen it done where the recommendations are a little bit closer to the top. I think it can vary. I like the format of sticking them down at the end because from a narrative perspective, it's kind of the ending. It's the call to action, the ‘go do something, to fix this’, that kind of thing. But I definitely could see an argument for moving it closer to the top because that's the stuff that people are looking for. One of the things that really helps too is having a well-structured document with a Table of Contents, especially with the reports that I work with Cobalt, we can just click and go straight to various sections because there's a little clickable table of contents, which makes our lives a lot easier because we can just send people and they can just go straight to it. That's a very nice feature that kind of lets us have a little bit more flexibility with the structure of the document.
I will say structure is one of those things that you don't really think about and that's probably a good thing because if you're thinking about the way a document is structured as you're reading it, then it means that it wasn't structured extremely well. So, that's the point of templating is you build it so it can be uniform and then those people can stick the information in the right spot. If somebody's kind of thinking about this on the backend then you've got the kind of freedom to just focus on making sure that your information shines through.
Robert Kugler: [12:10]
Absolutely, and that's also kind of what we learned here over the past years. It's super important to provide pentest structure. If you don't do this, you get so many variations of pentest reports and each of them varies so much that you can't imagine. That varies from super good to super bad and you want to really establish quality standards and that's only possible with templates.
Grahame Turner: [12:36]
That's one of the things that I have to balance is: on the one hand, I do want people to kind of have a little bit of their own voice and have a little bit of their - I don't want every single report to sound exactly the same. It just got churned out by a robot or something because that's kind of boring to read but also it does mean that you could just sort of copy and paste and just change the names and change the vulnerabilities and stuff like that. And so, it doesn't feel as sort of bespoke and unique to the customer so I do try to balance that with the fact that I do want things to be a little bit uniform. I want them to be consistent. I wanted a Cobalt report to sound like it came from Cobalt. I want a document that I write to sound like something I write. So, it's kind of that balance and it's something that I worry about. It's probably not something that a pentester needs to worry about per se.
Robert Kugler: [13:45]
I think at least myself, I really want to know that my pentest report sounds like a story. I could have just pasted this and the result out of Burp Suite to the report then and then I'm not calling myself a pentester. I'm a professional copy paster. That's not what my profession is and that's not what I want to do. I think it's really important that we, as an industry, come to the point where we really differentiate ourselves from the scanner output and the deliverable of manual pentesting.
Grahame Turner: [14:20]
It is definitely a key area where you can separate the automated results. I'm very much not a fan of people copying and pasting from CBE or OWASP or any of the other big websites because if I get a vulnerability that I don't have the right access controls and somebody's copied and pasted and stuck in the information that is on the open web app project, I could just google missing access controls, get the exact same information. That's another opportunity to say something specific to the test, say something specific like this is the way that your access controls work and this is why they're not working quite right as opposed to generally saying, ‘hey, badass access controls are bad.’
Robert Kugler: [15:19]
Absolutely, and coming to the findings report here individually, I think it's super important that you really provide a specific link to a specific resource that helps the client. What I really like to link is the OWASP Cheat Sheet. That stuff is just amazing because it's very specific, it really includes so much remediation details that can just be applied in most of the cases just out of the box and just a generic link to - I don't know, all of us saying okay there's Cross-Site Scripting that's why it's bad.
Grahame Turner: [15:53]
I think it's good to have both, definitely, because that's the thing. Customers want all the information that they can get. They want to be able to do the research so I think it's really important to speak. In the report, you want to speak to their specific instance but then that general information is super useful as well because it helps them kind of figure out what to do next. We're giving the recommendations. We're saying you got to upgrade this thing, got to implement this thing and then they're kind of stuck with ‘how? What does that mean?’ And so as you said, the cheat sheets are super helpful, information like that that really gives them not just fix this thing but here's where you can go to get the specific information. If you can't get it from there then maybe go to the vendor. That’s not just a command but also instruction.
Robert Kugler: [16:52]
I think also something to keep in mind is that we may even have differentiating audiences here too. We may have an engineering manager. We may have a security engineer and we may even have a startup C-level that's looking at the individual finding. So, it's good to have a generic finding description that puts it into context that quickly explains why and what it is. And then go into much more detail on the proof of concept section on the criticality later.
Grahame Turner: [17:20]
One of the things that I like to see when I'm editing a vulnerability report is I imagine if somebody were to just print out just this vulnerability report, is there enough information that I can just take this, print it out, hand it to someone and say go find this and fix this? And if there's a couple pieces of information missing, then that's probably a sign that I need to go back and do some editing. If there's too much information that might also be a good sign that there's some stuff to fix. But at the end of the day, I just want to be able to make sure that this one specific vulnerability gets addressed because otherwise why bother reporting it?
Robert Kugler: [17:58]
For sure and it can sometimes have a really crucial outcome for the client because if the vulnerability report is not good enough, not detailed enough, maybe people don't understand it, then people don't prioritize correctly and then it will maybe lead to a breach later. That's all because somebody didn't write the finding report correctly. It's a massive outcome.
Grahame Turner: [18:20]
It's one of the things that keeps me up at night - not really, but it's one of the things that I'm looking at, especially when it comes to like we were able to get this far but an attacker could go even further. That's the part that always makes me a little bit leery because I want to make sure that I'm communicating the risk without saying specifically this is the only thing that could go wrong. Because as I said earlier, computers are smart but they're also easy to fool and somebody with that kind of access could do something completely different that we don't predict. I used to talk about having a sort of ‘one step of conjecture’ where I just say, if I can guess somebody's password then I could get access to their account. I don't want to then go on and say, I can launch the missiles or whatever because I don't know. I don't know if that's what they're going to do or if they're just going to mine bitcoin. Who knows? Who knows what they're going to do?
Robert Kugler: [19:17]
Is this something where active versus passive voice could help a little bit to transfer that around? How important is it?
Grahame Turner: [19:25]
I will say I think active voice is super important generally throughout the report. But I do think that specifically attaching something to this hypothetical attacker and then different companies use a different term for that hypothetical person so just make sure you're following the style guide for that - plug for the style guide. But yeah, I think kind of associating it with something and giving the specifics is helpful. One of the things I've said in the past as well, to people when I'm talking them through reporting, is I think it's like if you've ever read like a cozy mystery, like a Poirot or another Agatha Christie hero, the report is kind of that scene at the end where they're in the parlor and they're talking about ‘I know who did it and here's why.’ ‘You went there. You did that.’ That kind of thing. Framing it like that can actually help because, again, you've got that narrative approach so the person is reconstructing logically and actively. And the active voice helps when reading it, ‘I did this thing. I was here when this happened. I saw this.’
One of the actors that's very useful to go back to active voice in a pentest report is the application itself or a server, or a specific piece of technology can also do things in the report. So active versus passive voice, passive voice would be like ‘an error appeared or an error was triggered’ would probably be a more clear example, as opposed to: ‘when I did X, the application triggered an error.’ That's kind of what the difference is there. And with that Agatha Christie example, I'm the detective: ‘I went and did this thing to test and then this is what happened.’
Robert Kugler: [21:21]
This was, for me, one of the most impactful things when I write a report that was reviewed by you because it really makes a huge difference. It makes it a lot more actionable when you read through that. I did this. You can literally think, ‘okay, you're reading it as the pentester right now’ and you're feeling that you literally do the pentest again and that's pretty cool. That makes it a lot more actionable and I hope that also the audience that's consuming the findings later feels the same - that would be amazing.
Grahame Turner: [21:56]
Yeah, definitely. That's actually probably a good time to ask what's some stuff that you try to make sure that you include? What are some tools that you use or tricks that you use to make sure that you're getting all the information in the right place so that when I come along, I can, at the very least, just kind of sand off some of the rougher edges.
Robert Kugler: [22:15]
Yeah, this is note-taking. That's a science, so it's crazy. I'm a really huge fan of Mind Maps. I like to go crazy with Mind Maps, especially if you have a big scope. It helps to keep the overview. I'm also a fan of text files. That's the other thing. When you have a huge scope and you have 20 text files, you go crazy and don't want to do that. So you basically link the files in the Mind Map to a specific target, screenshots all that you group together and then you have kind of - you're an artist at the end. You have a huge Mind Map and then transferring a really good place to enter and text replace and really enables you to think creatively.
Grahame Turner: [23:00]
That's interesting. That's a technique that I honestly haven't heard before, but I've heard similar things from some folks who do creative writing and write novels. That's the kind of thing that they use when they're planning chapters or what they're planning the overarching plot of the story. So, it's interesting to hear that in a different context. And I will say one of the things that - because I also read a fair amount of fiction - one of the things that I've noticed is everybody's got a different way to do it so if you found a way that works for you, there's no wrong answer, as long as you're getting the results that you're looking for. The Mind Map strategy sounds awesome, honestly.
Robert Kugler: [23:45]
Yeah, sometimes I feel like actually this would be nice as an additional appendix to the closing part. It's like a piece of art at the end that's just super big and helps you understand how the attacker thinks when they look at your text phase. But yeah, we are not there yet. I think as an industry to include Mind Maps in pentest reports - maybe at some point.
Grahame Turner: [24:08]
Maybe that would be really cool, honestly, because that'd be kind of just like a neat thing to look at. I also like this kind of visual representation. I was gonna say one of the other things that we used to talk about at one of my previous gigs was a lot of screenshots. Screenshots were super important. One of the things you want to keep an eye out for is take a screenshot for everything but also make sure that you're focusing in on where the specific action is, you're closing in on just the part that really needs to be focused on because if you've got the whole screen and you show that to the customer, they have no idea where they're looking. So, zoom in and add callouts like red circles or squares around the point of focus. Those can also be super helpful but really making sure that the font is big enough to read and that the specific area that you're trying to point out is called out. I think those are also super helpful for constructing stuff because the picture is worth a thousand words as they say.
Robert Kugler: [25:12]
Oh, for sure. And then if you include it in steps to reproduce like one, two, three, and you support it with screenshots, it's so much easier to reproduce. Screenshots are really a fundamental part of a good finding report and a good pentest report, too.
Grahame Turner: [25:28]
Definitely. Even when I'm trying to fix something on my computer, if I've got screenshots, and kind of walk through and get the same message, then I know I'm going in the right direction. For somebody who's actually trying to fix their network, it's got to be invaluable because the stakes are so much higher than me just trying to make sure that Word stops crashing.
Robert Kugler: [25:50]
They're pretty simple. They just try to see if they can reproduce something exactly as the person that reported it before. They look at the exact same output. Does it look the same? Is it the same color? Is it the same button? It's really this, what actually makes a good finding report and to get to that state where people can reproduce it exactly.
Grahame Turner: [26:11]
One of the things that came up a few times when I was working on some documentation purposes for how to write reports but also back in the financial company, too, was when people were using a Mac versus a PC, some of the feedback we got was ‘my error message was different because I'm on a different operating system.’ It's very easy to throw to someone if you're looking at something just slightly different. Those are the kinds of things that you kind of need to keep an eye out for and acknowledge. You got to maybe make sure you're including as many different options as you can because people don't like to think they're doing the wrong thing. It can make people nervous when they're trying to follow step by step.
Robert Kugler: [26:57]
Exactly. And then you’re in that weird sense of ‘okay, am I completely stupid not trying to understand this in the right way? I'm not that smart as a pentester’. You really want to be collaborative and make sure that everybody is consuming it as quickly as possible.
Grahame Turner: [27:16]
I think collaboration is another key point especially if you've got a technical writer that you can tap into and you're like I've just been picking at the sentence for 20 minutes and I don't know how to say the thing I want to say, just reach out and ping them. If you've got a teammate, having a second set of eyes on something - even if they're coming from the same level of grammatical prowess as you are. Just having somebody else who knows what you're trying to say and can tell you if you're not saying it is extremely helpful. For a lot of questions, where you may not be sure the best way to communicate this– ask the customer, or ask the person you're going to present the report to: what's the most useful way that I can give you this information so I can meet you where you need to be? That can be huge. Writing is definitely not done in a vacuum even if you are just kind of sitting and hammering away the keyboard on your own.
Robert Kugler: [28:15]
Absolutely, and sometimes the client may have a specific focus that they really would like to highlight a specific vulnerability that how to fix now for 30 years, but doesn't get the right attention. And then you can collaborate with the client to make it as detailed as possible in the pentest report and make a win-win out of it.
Grahame Turner: [28:35]
Oh definitely. I've definitely seen situations like that where it's like this is a vulnerability. I know it's bad and I know we need to hit this point so the pentester is usually telling me that that's what the customer said to them. I'm going, ‘okay, well you can provide just a little bit more detail here. You can repeat it elsewhere in the report if you've got a good way to do that’. Repetition can be helpful as long as it's not just copy paste, copy paste but saying the same thing in a slightly different way can really rhetorically drive that point home. Those are some little tricks where you're not doing a ton of work but still getting that message across and that's the kind of thing you can only really get by just asking like ‘hey, what do you need from this report?’
Robert Kugler: [29:25]
Nice.That's such an amazing end actually. We end up with collaboration is key. That's just perfect. It's actually what I also would take away from here is that writing a pentest report is pretty damn fun actually if you do it right.
Grahame Turner: [29:42]
Oh definitely. One of the things that I also like to tell pentesters when I'm working with them is, you're the expert. They brought you in to find some information to find some stuff. Yes, the report is this requirement. It may seem like it's hanging over you the whole time you're doing it but it's an opportunity to show that their faith in you is not misplaced. That you have an expertise, your expertise is pentesting and working with the customer, working with the people on your team, and working with your tech editors. You can create this document that really showcases your expertise in a way that produces results for the customer, because at the end of the day that's how you pay the bills but also is a good example of your work. And then you've learned a little bit more about how to communicate yourself which you can take to the next test.
Robert Kugler: [30:38]
Absolutely, and after all, the pentester's name is on the report. That's kind of representing you so you want to make sure that this is the best thing that goes out there.
Grahame Turner: [30:47]
Definitely.
Robert Kugler: [30:52]
Thank you everyone for turning this into another amazing Pentester Diaries episode. I want to thank Grahame today. It was super cool to chat with you about it. I can really learn a lot from your passion about writing. I really enjoy talking with you every single time. It’s super cool.
Grahame Turner: [31:09]
It's a pleasure.
Robert Kugler: [31:11]
Thank you so much. I'm the host, Robert Kugler and we hope you enjoyed the conversation.
