Joe Slowik, USN Vet, Adversary Hunter, and Digital Sanitation Engineer with a focus on ICS
This episode was originally streamed on Thu, 4-Jun-2020 to multiple platforms. You can watch the streams (along with the comments) on-demand on:
[00:04:56] Rik: Oh, good morning, good morning, the perils of life, I forgot to unmute my microphone. Good morning, good afternoon, good evening, good night, wherever you are.
I know from, previous shows we've had people checking in basically from everywhere in the world. So whichever continent you're joining us from, thank you again, for being with us.
[00:05:14] We have, an incredible guest this week. This is a, a person who- their- their- their job title is just one of the coolest, that I've seen, which is always a good start.
This is the Principal Adversary Hunter, responsible for finding, tracking, and defeating ICS-focused malicious actors.
[00:05:34] It's Joe Słowik. I have a, I have a real problem [laughs], pronouncing Joe's name correctly. Some of you might know that I have a Polish connection in my life.
In fact, I li- I used to live there for awhile. So whenever I se- see Joe's name written down, in my head it says Slovick and then I have to force myself to in- introduce him as Joe Słowik. Joe, help me out. [laughs] Wh- why, why don't we call you Slovick?
[00:06:00] Joe: You could blame, about 70 years of Americanization, I guess. Although, so, I, I grew up in Chicago, which has a very large Polish community. I think it's the second largest, city of people with direct Polish descent after Warsaw.
[00:06:14] Rik: Yeah, my wife says that Chicago Airport is Poland's first airport.
[00:06:17] Joe: Yeah [laughs]. Makes sense. But, yeah, and so growing up, I, it would alternate between people of the neighborhood I grew up in would say Slovick, but going to school, people would say, well, Slowik.
And it was usually a- an issue whether or not someone would tuck a CK instead of just a K onto the, end of the surname.
[00:06:33] Rik: Right.
[00:06:33] Joe: But, I do travel to Poland fairly frequently these days as well. In fact, I was hoping to be at CONFidence, right around now.
That's [crosstalk 00:06:55] would normally be and, you know, unfortunately with everything going on, things have been put off of it [crosstalk 00:07:02].
[00:06:45] Rik: ... Changed, right?
[00:06:46] Joe: Yeah.
[00:06:46] Rik: How's the whole experience been for you? I mean, like many people in our field, I'm sure that there's been a lot of canceled travel, a lot of canceled events or a lot of events moving virtual.
That's been my experience anyway. How has lock down been for you? Have you learned any important lessons from it?
[00:07:02] Joe: I've learned just how much I... Time I spent traveling and it's been very odd lately where it's like, "Oh, I'm not checking in for a flight."
Or, "I have a suitcase that normally is used every couple of weeks, at least." And now I've been in one place for awhile.
[00:07:19] Rik: Yeah.
[00:07:19] Joe: In fact it wasn't until yesterday for something that, I hadn't left the town that I'm in for weeks. And I live in a small town in New Mexico, Los Alamos right now.
So, you know, it's just been very strange, but at least we have the outdoors out here, so that's, that's good.
[00:07:35] Rik: That's right. I live in a really small place myself. I'm looking out the window right now. I mean, I'm in- in literally in a village of about 90 houses, so it's, I mean, it's tiny here.
[00:07:45] Joe: Very nice.
[00:07:45] Rik: But to, to, to ride out something like a global pandemic, it's kind of the best place in the world. There's very low risk and a lot of open countryside.
Anyway, you, you just mentioned where you live. You have, a, a storied employment history as well.
[00:08:00] And I know you started, this career kind of while you were in the military, but you went on then to work at a very famous facility local to where you live, right?
[00:08:10] Joe: Yes. So when, I left the U.S. Navy in 2014, my next stop was Los Alamos National Laboratory in Los Alamos, New Mexico. The, birthplace of such wonderful things like nuclear weapons, although they-
[00:08:22] Rik: Right.
[00:08:22] Joe: ... Do a lot of other work there, as well. But it is, you know, pretty wild, you know, there are parts of the town that have remained fairly untouched and are, you know, still housing tha...
The nice housing that was built during the, the Second World War where Oppenheimer lived or [inaudible 00:09:02].
[00:08:38] Rik: Yeah.
[00:08:38] Joe: And, you know, all these other famous physicists that had worked on the Manhattan projects.
That's, you know, a really cool sense of scientific history, that you get exposed to on a daily basis here, which is pretty, pretty neat.
[00:08:49] Rik: Is it kind of a weird tourist attraction thing or it just...
[00:08:52] Joe: [inaudible 00:09:18].
[00:08:53] Rik: It just happened to remain that way?
[00:08:55] Joe: I mean, it's not easy to get here which, kinda makes it a little bit diff... Difficult for just someone to drop on by. But, you know, there is sort of a nuclear tourist circuit, that you can do in New Mexico.
[00:09:05] Rik: Okay.
[00:09:05] Joe: 'Cause you can visit the, the first test site out at Alamogordo and then drive up to Los Alamos and, the US Park Service created like a Manhattan, historical district national monument to try to capitalize on that.
[00:09:20] So it's interesting, the town's trying to become more than just a, a company town. So it's just really, that's the only thing here is...
[00:09:25] Rik: [inaudible 00:09:54].
[00:09:26] Joe: This giant laboratory. So...
[00:09:27] Rik: Right. But, and- and, and when you were at the, the national lab, you were running the [inaudible 00:10:00]?
[00:09:32] Joe: Yes. So I started out, just, I got out of the military, I was an officer and I was like, "I don't wanna lead anyone anymore. So please just-
[00:09:39] Rik: Right.
[00:09:39] Joe: ... Put me on the IRA team."
[00:09:41] Rik: Just leave me alone.
[00:09:41] Joe: Yes. And, then after about a year, due to circumstances, I ended up leading that team for a couple of years, until I left in 2017 to join Dragos. So...
[00:09:52] Rik: So that's, that's an experience that not many people a- are gonna be able to lay claim to. Running the [inaudible 00:10:27] at a national lab, which is clearly going to be of interest, as an attack target to various different groups, individuals, nations, whatever.
[00:10:07] Was that an eyeopening experience for you? I mean, what... Did you expect something different when you went into it than you had when you walked away from it? And if so, what was that?
[00:10:16] Joe: I mean, there were certainly some surprises, but I mean, really, I think I got precisely what I was looking for because...
And I tell this to people who are trying to get started in the United States in the information security field that the...
[00:10:28] Rik: Mm-hmm [affirmative].
[00:10:28] Joe: The national labs in the United States are really great places to learn and to really get exposed to a lot of things because, you know, Los Alamos is famous for nuclear weapons design, but it also has, significant other research.
I mean, they've done lots of work on, using supercomputing complex here to map the COVID-19 genome and other sort of health research.
[00:10:49] Rik: Right.
[00:10:49] Joe: So there's lots of stuff going on, which makes it for a very interesting target or a very interesting-
[00:10:54] Rik: Yeah.
[00:10:55] Joe: ... Attack surface because you have this super secret national security mission and nuclear weapons design programs, but then you have this globally, connected research organization that brings in visiting scientists from places you might not expect, like from the People's Republic of China or collaborating-
[00:11:11] Rik: Mm-hmm [affirmative].
[00:11:11] Joe: ... With physicists in Russia, which means that you can't just say like, "Oh, we'll do a geoblock for all traffic from these locations."
[00:11:17] Rik: Right.
[00:11:18] Joe: [inaudible 00:11:52] you can't do that, there's very important and very vital collaborative research going on.
So it's a very interesting, environment to be in to see, you know, the sorts of things you'd expect in a defense or military perspective, but also something that you might get from a university perspective as well.
So, it's [crosstalk 00:12:09] exposing you everything from crimeware all the way up to your state sponsored, trying to steal secrets activity.
[00:11:41] Rik: What- what I noticed, which kind of stood out to me when I was looking through your bio a couple of days ago, is that, when it mentions your time there a- at the national labs, it said that while you were running the [inaudible 00:12:28] you were trying to get them to shift from a passive to a more active sort of threat hunting methodology.
[00:12:00] Joe: Mm-hmm [affirmative].
[00:12:00] Rik: So when you started it, the impression you got was that it was purely reactive and- and [crosstalk 00:12:42] what does shifting to a threat hunting model mean?
What would you offer other organizations or professionals that want to learn from that experience?
[00:12:14] Joe: Right. And I think this is part of a broader industry shift. So it's not as this, you know, that we were doing something completely novel.
But, you know, something, I think that was being reflected across security organizations where you would have a SOC or an IR team.
[00:12:27] And you're either playing sort of whack-a-mole with alerts day in, day out, or if there's not an incident like, "Oh, work out a project," or something along those lines and not trying to dig into data sets to try and figure out like, "Okay, is there weird stuff going on? Is there something that we can investigate,"
Or find or better know our own, network as a result, and maybe dig up some things that, you know, instead of waiting six months or a year after an intrusion to realize that, "Oh, that was actually the MSS.
[00:12:56] Rik: Right.
[00:12:56] Joe: We maybe should've caught that earlier." and to try and dig these things out earlier.
And, I had the pleasure of working and being mentored by, another individual in the name of Kelsey [inaudible 00:13:42] who may be familiar to some people.
[00:13:06] Rik: Yeah.
[00:13:06] Joe: He's mostly spent his time in the labs as well. And working on that with him was a very good experience to try to, you know, an, an incident response team shouldn't only be...
It's not quite like a fire department, which I think is an analogy that's often used that, oh, like, you know, when there's something bad happens, we go out and we fix it.
[00:13:23] Well the fire department, when they're not fighting a fire or typically training or sitting around the firehouse and doing stuff well, instead-
[00:13:29] Rik: Yeah.
[00:13:29] Joe: ...This should be more trying to go out and doing controlled burns or checking the building codes to make sure that if the fire does happen, that it's [crosstalk 00:14:16] contained. So-
[00:13:40] Rik: What are the practical steps?
[00:13:41] Joe: So the main thing is, first understanding your data. Like what kind of visibility do- do you have?
I mean, it's always enlightening bringing in someone new and it's like, "Here's the threat intelligence report, what do you do with it?"
[00:13:51] It's like, "Oh, oh, I'll look for the hashes." It's like, "Well, what does that even mean?" You know, can you even see that you have a hash value within your network? Or if you do, what's the context around it?
[00:13:59] Well, that means that it... You see it during execution, perhaps as opposed to during download. So based on that data point, where do you go from there?
How do I start moving out to, better articulate what do I use this information for or how I, can build that into my environment?
[00:14:13] And then based on that information, well, what things am I not miss... Am I missing?
So if I see that, oh, you know, adversaries still love to use PowerShell, especially a lot of very major adversaries, still love using PowerShell-
[00:14:25] Rik: Yeah. More and more actually [crosstalk 00:15:07] recently, right? I mean, over the last sort of 12, 18 months, maybe two years, we certainly a trend, we've seen those numbers significantly ramp up in terms of events detected, did our PowerShell, like-
[00:14:38] Joe: Yeah, exactly.
[00:14:39] Rik: ... Living off the land activity, anyway. I'll put it that way.
[00:14:41] Joe: Exactly. And, and learning like, "Well, what do we do with this?" Like, "Do you actually have visibility into things like PowerShell command execution?"
Especially at the interpreted level, after it goes through the PowerShell engine, as opposed to this big Base64 encoded-
[00:14:54] Rik: Right.
[00:14:54] Joe: ... Blob of mess that you can't really make heads or tails of initially.
And identifying those things like, "Okay, these are the things that we're not seeing, these are the things that we need to see. How do we start bridging that gap then in order to start identifying both, how to shift what it is that we can do from a security alerting and response mechanism to address those gaps and then longterm planning to fill those gaps."
[00:15:15] So it's like, "Okay, I know I can't get PowerShell Command visibility right now, but maybe I have full network visibility, even, you know, breaking SSL or whatever. So maybe I could look for scripts as they get downloaded and try and build something along those lines.
[00:15:28] And then we can work on getting CSMA and, or getting, or, you know, whatever other endpoint visibility tools in place or capturing PowerShell commands, upgrading everyone to B5.
[00:15:38] Rik: Right.
[00:15:38] Joe: to get that as part of the longer term strategy. So it's a question of, you know, both identifying what sort of things are going on within the threat landscape.
How do I then look for them within the tools I have available?
[00:15:49] And then in the longterm, from a planning perspective, how do I inform my leadership, my, you know, [inaudible 00:16:37] or whoever I... It is that I'm reporting to.
Of these are the tools that we're going to need to make sure that we stay at pace with the threats that are out there.
Otherwise we're going to be in trouble because we've already either be, you know, limited in what we can see or some things that are gonna be very important will be completely invisible to us.
[00:16:12] Rik: So it's not just really operationalizing threat intelligence. And I know you gave a talk at RSA, was it last year?
About exactly that [crosstalk 00:17:01] it's not just operationalizing it, it's also making it comprehensible to the people outside of that niche...
[00:16:26] Joe: Yeah.
[00:16:27] Rik: In the organization.
[00:16:28] Joe: Exactly. Yes.
[00:16:30] Rik: Yeah. [crosstalk 00:17:13] One of the things that we read a lot about, hear a lot about, and I think I've spoken about several times as well is for one of a better term is alert fatigue.
[00:16:39] Joe: Yes.
[00:16:39] Rik: And one of the statistics that I dug up from somewhere I don't even remember where now, I think it was, a survey by an analyst company, but I can't remember which one.
They had done a survey of financial institutions. And they, over 60% of the respondents said that they were dealing with, more than 100,000 alerts, everyday. Is that credible?
[00:17:00] Joe: that seems like a lot, but for a very large organization, I could believe it. And it, it is a real problem. I think that's why you have an entire industry built around this now.
[00:17:10] Rik: Yeah.
[00:17:10] Joe: With your automation, that's SOAR security orchestration and whatever that breaks into, because people are getting buried in- in these sorts of items.
And it's really a question like, do I need to see an alert every time that... You know, like, "Oh, IP connected to the Tor network contacted?" Like, that could be really interesting.
[00:17:29] Rik: I was, doing the maths on it. I wish I could remember the numbers off the top of my head. I didn't even know I was gonna reference this so, I haven't got it written down anywhere. [laughs]
And it was something that if, you know, say 25 minutes, per alert, just to do the triage, right?
[00:17:43] Like, is it a duplicate, is it critical or not? You know, doing that basic triage of the alert.
[00:17:48] Joe: Mm-hmm [affirmative].
[00:17:48] Rik: Say 25 minutes par, you've got a quarter of a million minutes of triage every day on 100,000 alerts.
It's clear that as an organization, if you're looking at that level of noise, there you've got no chance of seeing the signal, right?
[00:18:02] Joe: Exactly.
[00:18:02] Rik: I mean, you need to do something to whittle that away.
[00:18:05] Joe: Yeah. The big win, that a number of people are working on whether as public projects. I mean, you see some of this with like the Sigma project.
I don't know if you wanna call it a project, but the Sigma rules.
[00:18:16] As well as some of the other, commercial orchestration sets is really about trying to correlate and enrich data.
So it's no longer looking at, "Hey, alert fired," but rather like these sequence of alerts [laughs] fire in contexts that are all linked to something.
[00:18:29] Look at that instead and then you could roll up maybe 20, 100 items into a single item that the analyst already has triaged to a certain extent-
[00:18:38] Rik: Yeah.
[00:18:38] Joe: ... And both minimizes the ex- extra work, but also adds contextuality around something so that you're not just, you know, treating everything as being an isolated atomic incident.
[00:18:47] Rik: That's I think what- what the- the... Certainly the analyst community and- and I think it's fair to say the industry now is kind of coalescing around calling XDR, right?
[00:18:54] Joe: Yeah.
[00:18:55] Rik: What used to be EDR is low well, you know, it's way beyond just the endpoint.
We need to get intelligence from all these other sources from the network, from the mail servers, from gateways of various descriptions, and we need to be able to correlate, and we need to be able to present that as a story that you can drill down into and that-
[00:19:11] Joe: Mm-hmm [affirmative].
[00:19:11] Rik: ... That you can then use to pivot from, to go and do that threat hunting that- that we spoke about earlier on. Right?
[00:19:16] Joe: Yep. Yeah. And- and you see this also on a conceptual level too. I mean, you know, we're infosec, we're special.
We don't have one kill chain, we have like three or four of them and then you throw on MITRE attack on top of that.
[00:19:26] But, you know, we could joke about it. You know, how many frameworks do you need and add in things like diamond model and Lord knows whatever else too. [crosstalk 00:20:26]
But I think it shows that there's a lot of energy around like, okay, we've realized that what we're doing today, and we've realized that for a few years now, or whatever, this doesn't work.
[00:19:41] So how is it that we can start putting things in a framework where we can better sort, analyze and disposition or, articulate what it is that we're seeing?
So it's not just, "Hey, we saw this IP address," or, "Hey, we saw, command [inaudible 00:20:48] with a call to ping local host with count of one." Like, okay, that's weird. But you know, on its own-
[00:20:01] Rik: Yeah.
[00:20:01] Joe: ... It doesn't say anything, but you start piecing these things together.
And now you start either building up something through an attack sequence or a set of behaviours that, correlate with bad actor behaviour, as opposed to just something that's anomalous. Because an- an anomaly-
[00:20:13] Rik: Yeah.
[00:20:14] Joe: ... Could be anything. It's an anomaly, [laughs] not necessarily suspicious. It's not necessarily [crosstalk 00:21:12].
[00:20:19] Rik: ... Don't know what this is, right?
[00:20:20] Joe: Right. Well, which is why, you know, there's a lot of energy in the machine learning side of things, which, you know, 10, 20, 30 years from now, maybe it's gonna put us all out of work.
Hopefully that far, because then maybe I'll be retired by then. But, [laughs] you know, there, there's definitely hope there, but it's not there yet because identifying just something that's weird, only tells you, you found something that's weird, you still need someone, a human in the loop to do the enrichment, to figure out if weird equal suspicious equals malicious. So-
[00:20:45] Rik: Yeah, and I think, you know, when I look back at the history of the industry, we, we as security practitioners, I think we have a tendency not to trust automated reactions anyway.
So I think maybe the, the machine learning aspect of it, it's already important.
[00:21:00] It's already baked into a bunch of products and will only increase in importance that's for sure.
[00:21:04] Joe: Yep.
[00:21:04] Rik: But I think we have a natural suspicion of allowing, automated reactions based on, intelligence, however it's derived. You know, I think back to-
[00:21:14] Joe: Yeah.
[00:21:14] Rik: ... In the early days when it was like, "Oh, we can automate firewall rules based on threat intelligence."
And then you look back and think, "Well, how would an adversary use this?"
[00:21:22] Joe: Yep.
[00:21:22] Rik: Oh, suddenly I could find myself, denied of servicing myself as a result of responding to a spoof's attack for example, you know, stuff like that.
And just thinking about, one of the technologies I worked on in its very early days was, was IPS.
[00:21:35] When IDS was already a thing obviously IPS was the new kid on the block. And it was my experience in...
Back in tech support that basically everybody that bought IPS continued to operate it in IDS mode.
[00:21:50] And as far as I know, that's largely the case today as well. People have a fear of, of a false positive, no matter how unjustified now.
[00:21:59] Joe: Mm-hmm [affirmative].
[00:22:00] Rik: That [inaudible 00:23:03] on, right? I think we have that suspicion. God, I haven't even got to my first question that I actually wrote down.
[00:22:06] Joe: Well, okay.
[00:22:07] Rik: [laughs] this, that's how it happens. This is why I, it just goes on.
And I'm still not gonna get to the first question I wrote down because you, you made me think of something else just now, when you talked about how many kill chain models do we need.
[00:22:18] Joe: [inaudible 00:23:25].
[00:22:19] Rik: And obviously the latest greatest, biggest baddest, I guess, is MITRE attack.
[00:22:24] Joe: I don't think you're allowed to have a security conference these days, if there's not at least one attack presentation.
[00:22:28] Rik: [laughs].
[00:22:28] Joe: I love my team [crosstalk 00:23:39] by the way, [laughs] so I know a number of them.
They're great. [laughs]
[00:22:33] Rik: Is the kill chain dead?
[00:22:35] Joe: No, I think I look at them as being complimentary and I think the MITRE attack team will be the first people to tell you that this isn't a replacement for the kill chain, but rather an alternative mechanism for defining behaviours.
[00:22:47] Because a kill chain is a sequence of events whereas MITRE attack is a catalog of behaviours.
And so the two can be overlayed on one another. But I think, and- and this is where, you know, you know, the, the classic XKCD comic or whatever, it's like, "We need a new standard."
And it's like, there are now 17 competing standards.
[00:23:04] Rik: Yeah.
[00:23:04] Joe: you know, some synthesis between these is going to be necessary. 'Cause even internally for the position I'm in now, we refer to both kill chain and attack because they communicate different things.
[00:23:14] Rik: Right.
[00:23:14] Joe: you know, attack can tell you what it is you need to do or how it is that an adversary is behaving within certain stages of the kill chain or, you know, overall. Because you could use scripting at a variety of levels of the kill chain.
[00:23:26] Whereas the kill chain tells you maybe where in the stack, you need to apply certain controls or where you're going to observe it.
So, I think attack is a huge step forward, in trying to actually come up with a standardized lexicon of what's going on.
But it's still not going to be able to stand on its own for awhile, which makes things kind of confusing, unfortunately.
[00:23:43] Rik: Yes. We're bad at standardized lexicons in, in our industry in general. I think so for me, MITRE is a huge step forward. And if you look at it, you know, macro level, it has to kill train across the top anyway, right? Or a kill chain across-
[00:23:55] Joe: Yes.
[00:23:55] Rik: ... The top anyway. And then it breaks down into the matrix that it, that it, that it represents, which is, I think it's a fantastic model.
And obviously we, in- in Trend, we, we, we map to that now whether-
[00:24:04] Joe: Yeah.
[00:24:04] Rik: ... We're talking about products, whether we're talking about, you know, blog articles and white papers, we definitely see the value and kind of adopt it. I guess that's the same for you guys.
[00:24:12] Joe: Yes. And, you know, going back to, even what I was talking about earlier in terms of hunting methodologies, in terms of identifying gaps, it's one of the things that attack has been quite good at.
Is that in highlighting certain behavioural types, as opposed to very specific instantiations of them is that you can start getting an idea like, "Oh, I see defense afu... Evasion obfuscation. Like how do we find an obfuscated binaries?" "Oh crap, we don't have a-
[00:24:33] Rik: Right.
[00:24:33] Joe: ... Good way because we're not doing binary inspection." We're just doing AV that might be a problem, or scripting, in...
Which is I think in every step of, or in every column of the, attack landscape or [crosstalk 00:25:59].
[00:24:45] And so it really calls attention to like, what is it that we can see and what is it that adversaries are using.
And then being able to overlay those on top of one another to figure out what your detection and response gaps are at that point too.
[00:24:56] Rik: So I'm gonna finally get, I have a list of questions over there. I'm gonna get to some of those now, but I need to, 'cause I'm really bad at this part. 'cause I, I end up just having a chat with you and I, and- and-
[00:25:07] Joe: Yeah.
[00:25:07] Rik: ... In the case, in every episode so far [laughs] I get so interested and carried away that I just end up sitting and having a chat with you like it was a FaceTime call or something. Which I guess is great for comfort level of you and I.
[00:25:17] But what I need to make sure that I'm aware of is that there is an audience out there. Hi, thank you for joining us.
And I need to let you know that, this is open for you to ask questions as well. I'm hogging all the question space so far. LinkedIn is there for you to post your questions, and they will appear in front of me down here.
[00:25:33] And I'll pose those questions to Joe or if you've got questions for me equally, equally valid, LinkedIn, Twitter, YouTube, all of those platforms, please feel free to use them and drop questions on us.
The more the merrier that would be fantastic.
[00:25:46] So, we spoke earlier on Joe about stuff and one of the first things that came out of your mouth was Sandworm and how it's actually kind of a, a big time for Sandworm right now for- for, if anyone is unfamiliar with the term or maybe uses another term, Sandworm is NotPetya, Sandworm is [inaudible 00:27:26].
[00:26:06] And Sandworm, was Olympic Destroyer as well. And it was recently recognized by, as a group, as an APT group, by both the UK and the US governments as being, GRU Main Center for Special Technologies group 74455.
[00:26:20] So they have done a very definitive state attribution of, of Sandworm. And we all know that attribution is the most complex part of, of threat telemetry in general.
But there's some new stuff going on with Sandworm right now and you've been looking at it recently, I guess.
[00:26:35] Joe: Yes. That's been most of my previous week, or I think that came out last Thursday, is when the, National Security Agency of all entities just dropped a public report of, hey, these guys, which is this very specific Russian o- organization, which, you know, that alone is always an interesting data point when you start talking about the attribution question.
[00:26:56] Because in, except for very rare instances, like private industry can't really do that level.
And so it's been an interesting period of time just since 2016, between legal, cases, indictments, other sorts of proclamations by governments or whatever of really getting a fairly, refined look at just who is responsible for...
[00:27:17] Allegedly responsible for, however you wanna phrase it, a lot of this activity. So that was interesting.
And then we got like, "Hey, here's what they're targeting, Exim mail transfer authority." Like, hmm, that's kind of an odd one, but you look at the vulnerability itself and what sort of systems that would typically expose you to and what access it gives you.
[00:27:33] It's like, "Ooh, this is a pretty potent way either to, if something's externally exposed," I hope not. Or even inside a network, to enable some interesting attack paths.
And then, and I was talking with some analysts at a couple of different companies because, you know, there's always the- the threat intelligence community or whatever.
[00:27:50] Isn't just, we all sit in our own little corporate silos, but instead there's this little community underneath that we're- we're all kind of working on stuff together.
That's the National Security Agency report had, I think two IP addresses and a domain name but [crosstalk 00:29:33] that doesn't sound like much.
[00:28:04] Rik: Yeah. There was three IOCs, in- in that particular statement or report it was three IOCs.
[00:28:09] Joe: And I think I've, you know, myself and others have managed to turn that into about 300 discrete items at this point. Just ba... I mean, some of them we kinda knew about already, like definitely knew that they were bad, pretty sure that some of these might've been really bad.
[00:28:24] We're not 100% sure that it... Who it correlated to. And now, yeah, this is always the interesting thing, both from a threat intelligence and a threat hunting perspective of, you know, there's on the one side, a very interesting, almost sort of academic, way of looking at this. It's like, "Oh, this gives us an insight into Russian intelligence operations-
[00:28:43] Rik: Yeah.
[00:28:43] Joe: ... And what they're, targeting and tasking might be." I'm like, "Okay, that's cool. I can do that in my spare time." my customer would get very angry with me.
If I gave them a book report about like, here is what, you know, the history of the GRU, it's like, "This doesn't help you at all. What the hell are you doing? Why am I paying you?"
[00:28:56] But then there's the... This other tactical, operational side of like, "Okay, I've got two IPs and a domain, what do I do?" It's like, "Well, okay, well who registered it? How was it registered? What are the hosting providers here? What sort of services are being exposed? How do I find others that look similar to these and then start building that out with multiple sources?"
[00:29:13] And a lot of them are actually, you know, everyone talks about, "Oh, I can't do this because I don't have enough money for a, pro VirusTotal account."
Or, "I can't pay for Sensus or any other of a number of tools and data refining, or, datasets."
[00:29:27] But a lot of this you can do with freely available stuff and suddenly turning a, you know, fairly small number of observables into not just a long list of IOCs 'cause IOCs are they're limited in far as what they get you.
[00:29:41] But revealing instead, a class of behaviour it's like, "Wow," for like, from about 2017 up until last weekend, a couple of cases, this entity was performing credential capture operations using very standard, very repeated patterns for how URLs were crafted, where these were hosted, what sort of infrastructure was used.
That you can really start to articulating how a- an actor behaved to figure out not just the IOC, but if you see this combination of o- observables in the future, outside of that list of indicators is like, "Whoa."
[00:30:15] Rik: You start putting it together, dropping it and again, MITRE attack, right?
[00:30:18] Joe: Yeah.
[00:30:18] Rik: Is a great tool for helping you to- to do that.
[00:30:20] Joe: Mm-hmm [affirmative].
[00:30:21] Rik: Dropping things in the right buckets. Th- the other thing that really surprised me when, when I was, you know, when I initially read the, the NSA, statement was how widespread Exim is as an MTA, I started looking into it.
[00:30:32] And it's almost 60% of MTAs on the internet-
[00:30:36] Joe: Yep. [laughs]
[00:30:36] Rik: ... Are running Exim. So no wonder, you know?
[00:30:38] Joe: Yeah.
[00:30:38] Rik: it's not the first vulnerability, even in recent months, in Exim I was looking, through some of our internal mailing list where we talk about those kinds of things and it's, "Oh, here's one, here's another one."
[00:30:49] So you know, it's not surprising that something that widespread a- and that critical to communications, if you can man in the middle on an MTA you stand a good chance of finding a lot of useful information or pivoting out to other areas with- within the organization, right? Which is the big [inaudible 00:32:42].
[00:31:04] Joe: Yep.
[00:31:05] Rik: Is it, is it a big deal though that the US government, effectively and the UK government are pretty much the same time coincidence, were willing to make that super distinct attribution? Is that a big thing?
[00:31:23] Joe: I think it is. And- and this starts getting us into, you know, less the operational, how do I defend a network and more into the policy and norm setting aspects of, cyber, and you know, Sunroom's pretty popular.
It's been, Andy Pel... Greenberg published his book last November, October. I can't remember exactly when, late 2019.
[00:31:42] Rik: Yeah.
[00:31:42] Joe: And so that kinda got Sandworm in a historical fashion in the news. And then we start seeing, you know, in addition to some of the election interference items, like this comes out as well.
What's interesting about this class of activity though, is that you have two governments that are responding not to something that seems like it was fairly like, you know, thou shalt not interfere in elections or thou shalt not cut power to civilians in the middle of winter, or thou shalt not execute a worm that ends up disrupting life, all, you know...
[00:32:09] Rik: All over the planet.
[00:32:09] Joe: Globally. Yeah, exactly.
[00:32:10] Rik: Yeah.
[00:32:11] Joe: This kind of, I put this in a similar category as the, German legal case that's being brought, also against Russian interests with the Bundestag, intrusion from a few years ago.
[00:32:21] Rik: Mm-hmm [affirmative].
[00:32:21] Joe: Because now we're starting to get into States, calling out items that really seem to align with traditional intelligence activity.
[00:32:28] Rik: Yeah.
[00:32:28] Joe: Like, all right, we can all say that it's like, you know, we don't want people spying on each other. You could scream that all day long, it ain't gonna happen. [laughing]
[00:32:34] Rik: Yeah.
[00:32:35] Joe: spy is gonna spy, people gotta gather that intelligence. And to get to this level of disruption or public declaration by entities that would normally keep quiet about these sorts of things or respond to these only in a fairly, circumscribed or in very protected, networks way.
[00:32:53]that's really interesting because it's almost like we're starting to see perhaps people trying to set limits on what's acceptable and what, people will stand for when it comes to potential cyber intrusions.
Or alternatively, maybe and this is pure speculation at this point, but speculation's fun so let's do it.
[00:33:11][laughs] That the Exim comment came out because Sandworm is a very interesting entity because they're also an entity that is associated with some of, and most of the actual disruptive operations that have taken place via cyber, whether you're talking about the Ukraine power events.
[00:33:26] Rik: [crosstalk 00:35:13] Yeah, some of the biggest in recent years, there's, e- even recently there's the- the- the- the attacks on Georgia, right?
[00:33:31] Joe: Yeah.
[00:33:31] Rik: The- there were, were defacements, and-
[00:33:33] Joe: Mm-hmm [affirmative].
[00:33:33] Rik: ... Took a couple of TV stations offline for a while as well.
[00:33:36] Joe: Yes.
[00:33:36] Rik: [inaudible 00:35:24] And the stuff I mentioned, NotPetya, Olympic Destroyer [inaudible 00:35:27].
[00:33:40] Joe: Mm-hmm [affirmative].
[00:33:40] Rik: and more many, plenty more, right?
[00:33:42] Joe: [crosstalk 00:35:30].
[00:33:43] Rik: They, some of the real ticket items, in- in cybersecurity news of recent years.
[00:33:47] Joe: Yes.
[00:33:48] Rik: Without [crosstalk 00:35:37].
[00:33:48] Joe: Which is why, if you read between the lines of the Exim notification, that it might not just be, it's like, "Hey, bad guys are doing bad things." Like, "Okay, cool. We know this."
[00:33:55] Rik: Mm-hmm [affirmative].
[00:33:56] Joe: But this could be an indirect way of saying like, "Hey, not only are bad guys doing bad things with this vulnerability, but a very specific group that is associated with critical infrastructure disruption is using this."
[00:34:09] Rik: Yeah.
[00:34:09] Joe: So maybe some of the victims in question are just, you know, your standard intelligence targets, but actually, you know, it... Who knows? TV stations, power stations, et cetera.
[00:34:19] Rik: And I guess, given the nature of the- the software in question, I suppose, you know, my- my- my initial thought was, this is kind of a, this is like an early warning because the, the kinds of activities that are initially possible with this kind of compromise are more like reconnaissance type activities than disruptive type activities.
And yet we're talking about a group which is associated more...
[00:34:42] Joe: Mm-hmm [affirmative].
[00:34:43] Rik: With disruptive type activities. So this is like an early warning, hey, disruption is coming...
[00:34:47] Joe: Yeah.
[00:34:48] Rik: Because all the reconnaissance has been happening. I need to step back in time because I…a question's come in from David on YouTube.
And he, we spoke, earlier on about, XDRs and emerging technology, and we spoke about MITRE attack.
[00:34:59] And David wants to hear, I think your views, maybe I'll offer mine as well on, those systems as they exist today.
[00:35:07] Joe: Mm-hmm [affirmative].
[00:35:07] Rik: Well, how do you live out XDR as a solution set right now? Is it, you know, is it mature? Is it maturing? Is it easy to use? Does it add value, all that kind of stuff.
[00:35:16] Joe: Right. And I think it's, you know, first off I preface this by saying that, I haven't done like secured a network.
I've been doing the third party, helping others or whatever for a few years. So I haven't been on console with any of these in a while.
[00:35:28] So, you know, give that disclaimer, but from what I understand, and even looking at like some of what we do at Dragos as well, it's definitely the way ahead, just as we were talking earlier, if you're not contextualizing and combining datasets you're wrong or you're gonna be buried in individual atomic alerts that you can't possibly respond to all of them.
[00:35:45] So it's the right idea. It's getting better, [laughs] but it's also one of those sorts of solutions that you're only going to get as much value out of it as you put into it, which makes them prohibitively expensive and difficult to deploy.
[00:35:57] It's not just something where you can drop it into a network, set it and forget it and walk away.
But instead requires lots of adjustments, care and feeding and initial setup time to make sure that, you know, if you're pursing the datasets correctly, you're naming fields and com... You know, common lexicon or whatever.
[00:36:13] That you're feeding the right data streams into a product and continuing to update those as other things kind of get slotted in and out, within different solutions.
So it's the right way forward but I think marketing has gotten in the way of showing [crosstalk 00:38:23] just...
[00:36:29] Exactly, I know. Like marketing would never mislead people [laughs] of presenting things as sort of miracle solutions when in fact, there's, there's still a lot to be said for what needs to be put in to make these into effective products, I think as a result.
[00:36:41] And kind of like what you were saying with the IPS example is that, you get organizations that might procure these solutions not realizing that sort of hidden cost of installation or management.
And then as a result, you're only using a fraction of the overall functionality.
[00:36:55] Rik: Yeah.
[00:36:55] Joe: And thus the idea gets a bad name because it was never properly implemented in the first place.
[00:37:00] Rik: Yeah. I think one of the, the- the other emerging areas that's complimentary to XDR, which I think is designed to address some of those issues for maybe organizations that don't have the resources is MDR, managed detection and response.
[00:37:14] Joe: Yep.
[00:37:14] Rik: So, that takes care of they're not using it to its fullest capabilities side of the equation. Because-
[00:37:20] Joe: Mm-hmm [affirmative].
[00:37:21] Rik: ... You're leaving that part to- to the people who designed and built it, for example, or to a trusted reseller or whatever the model is that you're using.
But I think MDR certainly for, for those organizations that don't have the budget to, to, to justify, you know, full-size threat hunting organization.
[00:37:39][inaudible 00:39:36] Are really a good way for us, but of course, you've gotta do competitive comparisons and look at features and functionality and best fit for your threat model, critically, best fit your industry and all those kinds of things.
Hope that, answers your question, David, thank you for asking.
[00:37:53]yeah, what else did we talk about? You mentioned air-gap jumping malware, right? We didn't, dive into any great detail.
I assume you're talking about the recent spate of, 'cause you said to me, air-gap jumping malware it's not what people think it is. So like you were referring to things like Ramsay, the USB based stuff, right?
[00:38:10] Joe: Yeah.
[00:38:11] Rik: The- the ones that I can think of, that we've seen recently is, Ramsay, I know Trend Micro have blogged about USBferry, which was another, another one, there was USBCulprit, I think, which came from a Kaspersky blog recently.
[00:38:23] There's certainly been a spate of, yeah. Malware, certainly the, the blog titles and the press coverage around them...
[00:38:31] Joe: Yeah.
[00:38:31] Rik: Didn't focus on the malware that jumps the air-gap thing. Why did you say it's not what people think it is?
[00:38:36] Joe: Well, and I think, you know, I just wanna emphasize this very strongly. Like the reports you mentioned in the research have been great.
The issue has been translation into more public, [laughs] descriptions of these sorts of things.
[00:38:47] And this is where [crosstalk 00:40:54] problems always seem to come up. Is that, I think a lot of people see, oh, you know, rights to USBs, it jumps air-gaps.
It must be like Stuxnet or targeting nuclear reactors or super secret sensitive government networks. And that may be true, but it ignores that just because something can write to removable media like a USB drive or something along...
Else along those lines doesn't mean that you have an effective air-gap jumping tool.
[00:39:11] Stuxnet was an effective air-gap jumping tool because not only could it write to removable media, as well as some other interesting things like the project file item, which continues to get overlooked with the Stuxnet incident.
[00:39:21] But it was also autonomous in operation once it got into the network that it wanted to be in. So you didn't have to worry about command and control.
I can't remember which one exactly. It does like document harvesting or whatever, sorry, they're all like blending together right now. Like that is a little bit of-
[00:39:35] Rik: Yeah [inaudible 00:41:43] USBferry definitely did [inaudible 00:41:44] harvest. But I think at least one other did, did similar. What, what we've tended to look at. I think in- in the recent ones is basically a Trojan that propagates to removable drives.
[00:39:48] Joe: Right.
[00:39:48] Rik: Effectively.
[00:39:49] Joe: That goes back to like [inaudible 00:41:58] you know, you're talking like 15 year old EXE overwriting, or binary overwriting worms that are...
[00:39:56] Rik: Yeah.
[00:39:57] Joe: Not really fancy and not really, they're still disruptive in certain environments.
That really, if you wanna start talking about air gap jumping, you know, going back to our MITRE, attacking kill chain items [laughing] that unless you can have something that either enables command and control in some fashion or can operate semi-autonomously based on pre-programmed defects or ideas, then it's like, okay, I have a propagation mechanism.
[00:40:18] I don't really have any control, I don't have any ability to deliver an impact or, have some sort of effect on that follow on network. So it's only once you start layering in the ability to take advantage of, or somehow make that malware useful.
[00:40:34] Once it's gotten into a disconnected network in some fashion, that you can really start talking in my opinion, at least intelligently about something that can jump an air-gap.
[00:40:42] Rik: Right.
[00:40:42] Joe: otherwise you just have a f... A spreading mechanism that's especially proliferating. I don't know. However you wanna phrase it.
[00:40:49] Rik: Yeah. It- it- it- it seems when, you know, when you talk about the Trojan that writes itself to a removable drive scenario, it seems that you're looking far more at a kind of spray and pray approach.
[00:41:02] It's like, I really hope this lands in the right place. And I really hope that, that USB gets taken to the right place. And then I really hope that same USB gets brought out of that right place and put in the wrong place.
[00:41:11] Joe: Exactly.
[00:41:12] Rik: ... Like the big chain of events that you rely on all of these different things happening, in order to get you a payoff.
[00:41:17] Joe: Right.
[00:41:18] Rik: So it's... So would you say they're kind of less expert than Stuxnet?
[00:41:23] Joe: so far-
[00:41:23] Rik: I really hope no one's playing, [inaudible 00:43:36] with us 'cause we're gonna hit all of them, I'm pretty sure.
[00:41:29] Joe: But I mean, it's still fair because, and- and I wrote a paper about this last year, you know, that spent several pages get... Digging into Stuxnet again and it's like, "Oh, haven't we talked about this enough?"
[00:41:38] [inaudible 00:43:50] wrote a great book about it. We have, you know, semantics really good report on the malware. Like what else do we need to understand?
There's still misconceptions over how that malware operated.
[00:41:48] And some of the tactical details around it that are important for discussions just like this, because, you know, just saying that Stuxnet was air-gap jumping malware misses the point that it had multiple propagation mechanisms.
Not all of which are frequently discussed in a couple of different versions and that-
[00:42:05] Rik: Yep.
[00:42:05] Joe: ... Autonomous functionality within a very discreet and limited target set, which makes it, which translates it beyond like, "Oh, it spreads like wildfire on USB."
Like, "No," but it also makes sure that it's operating in this sort of specific environment with this sort of specific software, beto... Before it does anything important and we maybe saw reflections of that in some other things like Flame and [inaudible 00:44:39] is still a big mystery.
[00:42:27] Be fun when someone cracks that one, we'll see what one Andreas is up to these days now that he's back blogging again.
But, you know, other sort of mystery items that are out there, that's some of the things that we're seeing lately are certainly very effective tools and effective at spreading and in some cases effective in gathering information.
[00:42:42] But like you said, unless I have a way of getting it out, what's the point? I mean, there's still maybe a point, but it makes it far more difficult than...
And we see the same sort of issue, I'm gonna mention the big hack, to make fun of it. [laughing] So...
[00:42:55] Rik: [inaudible 00:45:10] so get in there, we've- we've done Stuxnet, so everything is on the table now.
[00:42:58] Joe: Right, exactly. But like, you know, the idea of inserting a rice size chip on the motherboard as it leaves the- the factory or whatnot. Oh, that sounds really s- scary. It's like, "Yes."
[00:43:07] Except unless I can either target it or figure out a way to operationalize it, then it's a notional attack value that, you know-
[00:43:14] Rik: Yeah.
[00:43:14] Joe: ... You start getting into the things like air-gap jumping, supply chain hacks, and hardware back doors and all of these sound, you know, real a- and they are concerning and they are difficult to detect in their own ways.
[00:43:25] But, unless you have something that can be de- developed, deployed and inserted where you want that can then operate completely autonomously, we've only got one example that's done that successfully, that we know of so far.
[00:43:35] Then you still have a lot of work to do and you still need to be able to communicate with it.
So I can, look for network traffic, I can, you know, why is my switch getting a nonstandard communication or whatever on a port that it shouldn't be listening on and responding back.
[00:43:49] Rik: Yeah.
[00:43:49] Joe: I should probably investigate that. You know, if- if you had that [crosstalk 00:46:09].
[00:43:53] Rik: ... Putting a presentation together, I think it was last year. It might have been the year before. And I do...
But I don't think much has changed in regard to this particular point, in the meantime. And I was trying to do a- a timeline of, malicious tools that had been specifically written to target ICS environments, okay?
[00:44:10] Joe: Mm-hmm [affirmative].
[00:44:11] Rik: To target industrial environments.
[00:44:13] Joe: Yeah.
[00:44:13] Rik: Because it- it's a big, you know, everybody wants to hear about it. It's a big conversation point, there is risk. But I'm not talking about the, the [inaudible 00:46:40].
[00:44:21] Joe: Yeah.
[00:44:21] Rik: the [inaudible 00:46:41] of the world. I'm talking about the Stuxnets of the world. And I think I came up with like five, does that sound about right? It's... There's not a massive population of specific ICS targeting...
There's malware, like Flame and [inaudible 00:46:56] and all that, that have been used to target those environments, but they weren't written for those environments. I'm thinking more like, [inaudible 00:47:03] more like, Trisys.
[00:44:45] Joe: Mm-hmm [affirmative].
[00:44:46] Rik: Triton, Havex.
[00:44:48] Joe: Yep. Like Energy 2, kind of.
[00:44:49] Rik: Like [crosstalk 00:47:14] Yep. [inaudible 00:47:15] beyond those names, there's probably not much else. Right?
[00:44:56] Joe: outside of things that we only know about from labs or [inaudible 00:47:23] presentations [laughs] or whatever, but like, you know, no kidding in the wild.
Yeah, it's a small handful and that's also another really...
[00:45:06] Now going back again to the idea of there's a MITRE for IC... A attack for ICS down as well. But, of really distinguishing between industrial tar... So, distinguishing between targeting and effects.
So the five, we just lifted have some effect in terms of either gathering, you know, Havex, being able to pull IP... OPC servers in order to extract information from them.
[00:45:28] Okay. Or something like a industry or crush override, Triton traces, or Stuxnet that could actually have a manipulation to cru... Cause damage disruption or something else along those lines, very scary.
[00:45:39] And then there's an entire family of industrial interested, like for example, Kaspersky released a very, Kaspersky's, ICS, specifically, released an interesting report on some, very targeted phishing activity that, they noted was focused on industrial organizations.
It gets to the press and then it's like, "Oh, it's targeting industrial systems." Like, "No.
[00:45:59] Rik: Yeah.
[00:45:59] Joe: Hell no." Like that's not exactly what this is phishing.
[00:46:01] Rik: This is phishing. [laughs]
[00:46:01] Joe: It may, it's very much targeting organizations related to this, but it's focused on their IT networks. So you could theoretically leverage that access to then deploy something after it.
But the id... The stuff we're talking about right now is just interested in the vertical and so your motivation can be anywhere from, you know, ransomware, maybe.
[00:46:24] Probably not in this case, given how the malware look through, you know, intellectual property theft, all the way to the far side, then of trying to pre-position and gather information to prepare for some sort of future disruptive attack.
There's a lot of... The- there's a long road between, you know, phishing someone and then blowing up an oil and gas refinery. So...
[00:46:45] Rik: Yeah. We, Trend Micro Research, did some, they did some great, honeypot work over the years.
[00:46:51] Joe: Mm-hmm [affirmative].
[00:46:51] Rik: Most recently was the- the fake factory and they had fake employees. So they use photos from ThisPersonDoesNotExist.com [inaudible 00:49:24] webpage.
And it's a really nice honeypot set up, that was the most recent one.
[00:47:01] Before that there was a water facility that they accidentally published online and checked out the results.
And I think the first one that we did, was the gas pot,
[00:47:12] Joe: Yeah.
[00:47:12] Rik: ... Which with the gas pumps.
[00:47:14] Joe: Mm-hmm [affirmative].
[00:47:14] Rik: And I'm pretty sure that without exception, there was no specific look at this scary piece of industrial malware that, you know, there were ransomware attempts, there were extortion attempts, there was information theft.
[00:47:25] Joe: Mm-hmm [affirmative].
[00:47:25] Rik: There was definitely manipulation by the attacker or the curious individual, but it was, it was human on machine manipulation, not malware on machine, if you know what I mean.
[00:47:36] Joe: Yeah. Well, and that's a really important distinction as well, because if the only goal that I have is that I want to disrupt something like, I wanna turn the lights off or I want to, you know, cause some factory owner a lot of pain, I don't need malware to do that.
[00:47:49] I can just harvest a bunch of credentials, log on to an engineering workstation or better yet an HMI and then just turn it off. [laughs]
[00:47:56] Rik: [inaudible 00:50:27] BlackEnergy 2 the, like the first blackout [00:48:00] was more like that. Right?
[00:48:01] Joe: [crosstalk 00:50:31].
[00:48:02] Rik: You could see the, the remote desktop and the mouse moving around and the people going, "What the hell's going on now?" With like, what's happening.
[00:48:07] Joe: Yeah. Although there's, again, you know, it's one of those things.
So with the BlackEnergy 3 facilitated blackout in 2015, you know, again, there are multiple things going on in those environments.
[00:48:18] So on the one hand you had the DNC connection, like, okay, that's stupid, simple, or whatever.
Two factor authentication, or God, please don't use DNC or [laughs] you know, so many other things, or TeamViewer, you know, take your pick.
[00:48:29] Rik: Yeah.
[00:48:30] Joe: But, but then there's the other i- item where they stood up a rogue SCADA master within the environment. That's a little bit more complex in order to facilitate an intrusion to that level.
And then after that, you know, after they, you know, opened up things or whatever, and cut the flow of electricity across three distribution sites, roughly simultaneously.
[00:48:47] The really interesting part of that attack and probably the more technically sophisticated part was pushing the malicious firmware update to the serial to ethernet converters in that environment.
[00:48:56] Rik: [inaudible 00:51:27] the- the- the serial to ethernet, yes.
[00:48:58] Joe: Yeah, 'cause that's blowing the bridges afterwards. Like we're gonna open these up and then we're gonna take these out so you can no longer remotely control these. But they didn't-
[00:49:05] Rik: [crosstalk 00:51:36] on site and flip manual switches, right?
[00:49:07] Joe: Yup. [crosstalk 00:51:38] The thing is, is that, that is a realistic operation in Ukraine. You talk to the [inaudible 00:51:44] and, you know, some of the other operators in that part of the world, they're used to having to enter manually in- in order to operate a lot of this equipment.
[00:49:23] You do this in the United States where you have like a single control center, that's controlling a very large geographic area.
You're talking about putting someone in a truck and driving for hours to try and get into a position of restoring power, as opposed to, you know, the 2016 was about an hour, 2015 was a little bit longer.
[00:49:41] You know, that they were set up in order to manually intervene. Whereas a lot of the, you know, Western Europe and the US and Canada, and especially we're not set up that way. [laughs]
[00:49:48] Rik: I had a really, eye opening conversation with someone who works in the power generation industry in Europe. And I said something about the National Grid 'cause that's the name of a company in the UK-
[00:49:57] Joe: Yeah.
[00:49:57] Rik: ... The National Grid.
[00:49:58] Joe: Mm-hmm [affirmative].
[00:49:58] Rik: And, but I used it conceptually, not as the name of a company. And he looked at me and said, "You know, there really is no such thing as a national grid within Europe.
[00:50:05] Joe: Yeah.
[00:50:05] Rik: it's all one thing." So if you, if you can imagine an attack like that across a geographical area, the size of the US, so you've got the same distance problems, but multi jurisdictional, multilingual, multinational. [laughs]
[00:50:20] So the same thing. I mean, if it, if, if it's, if it's really the case, that the kinds of things that we've seen, and I think this was what was intimated or even stated directly in the, in the Wired article, about, Sandworm in this particular attack.
[00:50:32] If Ukraine really is being used as a testing ground, playground, whatever. We've got a really strong vested interest in making sure that we start getting stuff, right.
So, what do we do? What, what, I, you know, what are the, what's the low hanging fruit? What are the easy wins and what, what should be the longterm goal for operators in those environments?
How do we rapidly up the security game in those environments?
[00:51:00] Joe: Yeah, the easiest and quickest win, although it's a more difficult than it might seem as you start getting really deep into industrial networks because you have such wonderful things like hard-coded passwords that are, you can find online, that you can never change and, you know, wonderful things along those lines.
[00:51:15] But at higher levels, as long as you're not directly exposing this sort of stuff to the internet, please don't do that ever, is authentication security.
So, you know, it always sounds so trite, it's like, "Oh, multifactor authentication, robust password security."
[00:51:28] Like honestly, that would at the very least make most of the things that I see in the industrial space from initial access and lateral movement significantly harder than they are right now, because so much that we're observing, even in things like, you know, US, UK, German governments, even, of, or German media, at least talking about, Russian linked intrusions into critical infrastructure.
[00:51:51] If you look at the intrusions in questions, it's all facilitated by password capture and reuse.
So why are we making this easy? So, combination of make sure you're, you're using robust authentication mechanisms and multi-factor as many things as possible in order to make sure that just, you know, harvesting one password doesn't then blow the doors open.
[00:52:08] After that, then to try and tie in, try to really tie in, how we start off this conversation.
Then it's a question of, we have a small set of examples, but a meaningful set of examples of industrial intrusions right now, how does that map to what we can see to what our visibility is?
[00:52:25] And right now the visibility is shit, [laughing] for lack of a better way of putting it.
Maybe you get NetFlow maybe a little bit more network based, visibility, but if we're really concerned about not just defending, sort of at that crunchy outer shell of the external IT network, but really moving into, defense in depth and getting into industrial networks, we need to start building out network and process security at that level.
[00:52:47] And I say process and I'll highlight that because it's not just a question of, you know, your standard Windows based like, "Oh, let's, you know, do, do Windows event logging,"
And, you know, all these other things and try and push those, but also monitoring the industrial environment as well. Because, like looking at the 2017 Saudi Arabia incident with Triton/Trysis...
[00:53:06] Rik: Yep.
[00:53:06] Joe: being able to perform the sort of root cause analysis that would identify that, "Whoa, wait a minute, why would this have happened in the first place?"
'Cause there were two outages that took place and it wasn't until the second one that it was identified that there was a malware issue.
[00:53:18] So there was very much a missed opportunity there because sufficient industrial specific and process data wasn't being incorporated into the security investigation, as well as it could have been that might've identified that, "Hey, why did I see this connection to the, safety instrumented system workstation then followed by the push of a new update," as best as I can tell a lot of that visibility didn't exist in order to allow for that to be, you know, fairly quickly dispositioned.
[00:53:43] The result was, is that, you had a possible... Didn't end up working out for them, thankfully.
But a possible incident that could have resulted in a loss of safety and potentially even loss of life as a result.
[00:53:52] Rik: [inaudible 00:56:34] Yeah, I think so. Yeah. I think I would throw into that- that mix as well, segmentation as being criti-
[00:53:59] Joe: Yep.
[00:53:59] Rik: 'cause it- it's a relatively easy win as well, right?
[00:54:02] Joe: Yep.
[00:54:03] Yes and no, [crosstalk 00:56:47].
[00:54:04] Rik: ... Segmentation, it just means look at the bits that you can hide from the other bits and- and-
[00:54:09] Joe: Yeah.
[00:54:09] Rik: ... Make sure... The problem is, that there isn't much out there, that's being designed for those environments either from a physical perspective in terms of it, you know, you don't have server racks on the factory floor.
So you can't put rack mounted stuff down there.
[00:54:22] Joe: Mm-hmm [affirmative].
[00:54:22] Rik: you need din rail stuff.
[00:54:23] Joe: Yep.
[00:54:23] Rik: So you'd need something that has a different, you know, it attaches in a different way. Has different power requirements, temperature, dust, vibration, all those different things.
So from a physical perspective, there's not a lot, and from a digital perspective that is actually capable of pausing the relevant protocols.
[00:54:37] You put a standard IPS on a factory floor and they'll go, "Everything's fine, I can't see a thing." Right?
[00:54:43] Joe: [laughs] Well, and that's a very good point. You know, on the one hand I try to explain this, you know, I work at a, a company that does industrial security right now. It's like, "This is the problem we're trying to solve."
[00:54:52] And on the one hand, the problem is better because like encryption doesn't exist that much when it comes to industrial environments.
'Cause one, you don't really need it. If that's your main concern at that point, then you've seeded a lot more character [inaudible 00:57:49] to your attacker then as a result, if that was your primary concern.
[00:55:06] So you have a lot of things that are passing in the clear, the problem is, is that you also have a lot of things that are nonstandard or even better yet.
This is a fun thing with the 2016 Ukraine event, is that the malware industry or crush override, or at least the- the modules that did the breaker manipulation were designed based on the publicly available IEC 104 spec.
[00:55:27] However, the actual implementation on the ADD gear in the environment was not specific to the spec, but instead was proprietary and had a few adjustments into it. So as a result, parts of it did work. [laughs] So there's also [crosstalk 00:58:25].
[00:55:39] Rik: ... That's exactly back to something that we had in the last episode where I was talking about one of the perennial things in [inaudible 00:58:31] was looking at why don't people write things according to RFCs and that's kind of, that's kind of-
[00:55:51] Joe: Yeah.
[00:55:51] Rik: ... A good outcome from it, right?
[00:55:52] Joe: Yeah, and- and- and it's good and bad. But on the other end, it means that if you're trying to understand like, oh, like, you know, what function calls am I seeing to try and identify is someone deploying industrial specific malware?
It's not just a question of knowing the spec.
[00:56:03] It's a question of also doing like, "Well, what did the vendor write to meet the spec as well?" Because it might be different in ways that, are unexpected and can throw things off.
So it's a, it's a very diverse world on the one hand, it's nice in the, in the respect that creating industrial specific malware is very hard and we've only really seen one truly successful, publicly known incident so far, Stuxnet.
[00:56:26] Rik: Right.
[00:56:26] Joe: every other event, again I wrote about this last year in a fairly lengthy paper has on some level failed in some way, because this stuff isn't easy.
The problem is, is that, these sorts of attacks are fairly sophisticated. I hate using sophisticated because it's- it's a loaded term and it's so poorly used often.
[00:56:41] Rik: Yeah.
[00:56:41] Joe: But in this case, trying to do things in a multistaged way to really undermine integrity and industrial environments, that's not easy.
[00:56:48] Rik: Mm-hmm [affirmative].
[00:56:48] Joe: But if all you wanna do is turn the lights off or cause some sort of disruption, again, all I need to do is log onto a terminal because it's using poor or even limited authentication.
Like for example, if there was the story about a middle Eastern country with water treatment facilities that were being targeted and you're like at reporting like, "Oh, this is a big deal. This is so complex and nasty."
[00:57:10] If you look on Shodan and look for who's exposing certain protocols online, you start seeing lots of control panels towards things like chlorination of water that have-
[00:57:20] Rik: Mm-hmm [affirmative].
[00:57:20] Joe: ... Minimal or no authentication. So we're not exactly making it hard for certain entities in order to do things, which is very concerning. So...
[00:57:27] Rik: And that's kind of across the board. Do you know what? I knew this was gonna happen.
We're coming up on the hour, Joe, I could go on talking to you for hours. Would you come back in season two and we can talk some more?
[00:57:37] Joe: Certainly.
[00:57:37] Rik: [crosstalk 01:00:26].
[00:57:37] Joe: If you'll have me.
[00:57:39] Rik: ... Left in season one. I feel like we've covered half the stuff. If that scratch the surface of things that we could be talking about, it's been an absolute pleasure having you join me.
And I'd be super grateful i- if you would come back, for the next season.
[00:57:53] Joe: Awesome. I feel happy, I would be happy to hang out.
[00:57:56] Rik: I to, you know, I told you this would go by in a blink of an eye and this is why these things end up lasting an hour. [laughs] again, thanks so much for joining us, Joe.
And, and we'll talk to you next time.
[00:58:05] Joe: Sounds good. Thank you so much for having me.
[00:58:07] Rik: See ya. Wow. Again [laughs] it happened to me again, we did another hour. I know I did a Twitter poll after the last episode and, and the results.
Last time I looked, I probably didn't look at the final results, if I'm brutally honest with you, were mixed.
[00:58:23] Some people said, "Make it shorter, you fool. These things are far too long." And some people were like, "No, it's lock down. I have all the time in the world." So, I know myself, I talk too much. I'm passionate about the subject.
[00:58:32] My guests have all been amazing. And they just leave me astray. And we end up talking for an hour.
So here's another one for you that was Let's Talk Security, episode five. It was exactly as I thought, I said earlier on today on Twitter, it's gonna be a doozy.
[00:58:49] I would encourage you to go and follow Joe, on Twitter. He's very active, pushes up some, some great content.
I would encourage you to go and follow me if, if you enjoy my own particular brand of lunacy. My Twitter thing is, there, down there.
[00:59:03]thanks for joining us. We'll be back next week same time, same place with Let's Talk Security.
What can I say? I'm looking forward to that as well, we have another great guest lined up. In the meantime, I've been [Ron Burgundy 01:02:15] you stay classy, San Diego.
[00:59:41] This is the bit when, if you stay in the cinema until the end and after the credits have run, this is that bit. Thanks for watching.