Log4Shell impact and remediation strategy

A conversation with Igor Tsyganskiy, CTO at Bridgewater Associates and Yinon Costica, Co-founder & VP of Product at Wiz, discussing the immediate dangers and issues with Log4Shell, strategies for remediation, and its long-term impact on cybersecurity as a whole.

Log4Shell impact and remediation strategy

A conversation with Igor Tsyganskiy, CTO at Bridgewater Associates and Yinon Costica, Co-founder & VP of Product at Wiz.

In this fireside chat, the two discuss the immediate dangers and issues with Log4Shell, strategies for remediation, and its long-term impact on cybersecurity as a whole.

Complete transcript

​​Yinon Costica 0:05

Hello, all, I'm super excited to have with me today, Igor Tsyganskiy, Chief Technology Officer at Bridgewater Associates, one of the largest asset management firms in the world and known to be also one of the most sophisticated and advanced in technology.

Igor, the largest software companies like Microsoft and Amazon are following your guidance, and I remember clearly from my time at Microsoft that every email from your team always became a guiding principle. So thank you, again, for taking the time and joining me for this fireside chat.

Igor Tsyganskiy  0:39

No problem. Glad to be here.

Yinon Costica 0:41

Just as a starter, can you tell me a bit about yourself, your role, and how you're using cloud today?

Igor Tsyganskiy 0:47

So I run technology at Bridgewater Associates. We’re the largest hedge fund in in the world. And we have a very large compute infrastructure. So my responsibilities include everything from running our T data centers, software in our data centers, all the investment systems and risk, being responsible for our investment platforms that focus on identifying and understanding how the world works, creating opportunities that we then trade on, and all the appropriate trading systems. Everything from the architecture of those systems all the way to the operation of them.

Given the fact that our customers are mostly institutional investors, state-owned funds, pension funds, we have a huge responsibility in front of them to keep everything that we do secure. Given the fact that we're a systematic fund – and what that means is that all of our trading, all of our ideas, we persist them in software systems, and then we use software systems to trade the market. So we have quite a big infrastructure that we use to trade the market, our footprint is very large. We've been fortunate enough over the last four to five years to move most of our footprint into the cloud. And we have both, we use both Azure and AWS to operate the fund.

Yinon Costica 2:23

We are here today to talk about the Log4J vulnerability and the crisis it created. And maybe we should start with how did you hear first about this vulnerability? And what was your initial reaction?

Igor Tsyganskiy  2:36

Well, I first heard it from reading an alert about it. And my reaction was mostly “Wow.” You know, we went back and forth inside of the team. So okay, that looks pretty big. That was, I think, sometime last week, so that was about seven days ago. Like, that's exactly what we need before the holiday season.

Yinon Costica 3:04

Exactly. And I think NIST gave it a score of 10 out of 10. And US officials are saying this is the most serious flaw ever seen. And when we chatted before this conversation, you called it a digital COVID. And maybe you can share your perspective about why is it such a crisis today?

Igor Tsyganskiy 3:26

Right? Well, I'm saying that it can be, has a potential to be, digital COVID. And the way I describe it is, you know, imagine that all the interconnected computers in the world – and now we're learning that even some on Mars – computers, servers, devices. Imagine that all these are just houses, and those houses have windows and doors on them. And then one day, someone comes in, does something like this [*claps*], and says there's no more locks on any doors or windows. And by the way, you don't know how many doors and windows you have in your home. So that's the way I would basically describe what has happened.

Now, since someone just clicked and made the whole cyber world not as secure as it used to be, then the second question comes up of what does that mean? Then the question is, you have people who may attack you or enter your home without knocking, right? Just enter the home. And then you kind of need to go out and discover everywhere where you have doors and windows and put locks on them. And so the question becomes, what can happen faster? Can you discover where your doors and windows are and put the locks in faster than someone will walk into those open doors?

So the reason I am saying that it has the potential to be a digital COVID is mostly because the time it takes to attack someone is asymmetrical to the time it takes to protect. And attacking is a lot easier, given this vulnerability, than protecting and discovering what you need to protect. And therefore attacks can spread very, very fast. And if they spread fast, bad guys can have access to too many resources all at the same time. And then the whole system becomes really, really fragile.

Yinon Costica 5:47

This is exactly what we talked about. So around one, it's a very pervasive use of the library, it's everywhere. And the second, it's really easy to exploit it. And now we're basically going to see attackers using it to target different systems in specific campaigns. And the question is, first, how do we detect all of these systems? How do we identify all these systems across all of the environments? I read today in the news that even NASA's Mars helicopter is vulnerable to Log4j. So basically, how do we identify all of these systems across the estate in the most rapid way?

Igor Tsyganskiy 6:27

Well, the reason I'm here is because I'm grateful for the partnership that we have between the two companies, and being able to use your software to quickly identify the places where we may have, or actually where we do have a problem. I reached out when this happened a couple of days ago, so we became a Wiz customer, when was it in November? Yeah, I think in November, and we, I would say, signed a pretty big deal with you guys. And I reached out to the CEO of your company and said, hey, you know, you just paid for the whole deal in one week. And the reason I said that was because by using your software, we're able to identify the places where we can find vulnerabilities extremely fast, so we can patch it, right? So you can get an inventory of all those doors and windows.

So generally speaking, we're very fortunate that large portions of our infrastructure are running in the cloud. And given the fact that they're running in the cloud, both in AWS and Azure, you have cloud-native software that we were able to use. Maybe it's serendipity, not in a great way, but this is what your software was designed to deal with. And so, you know, static analysis, and dependency analysis across a vast number of computers trying to figure out, you know, producing a dependency graph and security graph trying to figure out where you might have issues. Once you know where you have issues, then patching it is not actually as hard as just knowing where they are. So we used Wiz, and after going through the whole experience, and actually focusing on fixing the problem, I agreed to do this podcast as a big thank you for the partnership.

Yinon Costica 8:31

Thank you very much, Igor. I really appreciate it. And I'm so happy to hear that this was so helpful. Maybe this is in striking difference to what we have in the on-prem world, right, where it's very hard to get an immediate view, even to endpoints and other devices that don't have cloud.

Igor Tsyganskiy 8:46

And, yes, I'm hopeful that tools like yours, you know, that you will have support for VMware soon. And you will be able to do the similar dependency analysis in the static world of on-premise, where you still deal with images. Is it correct that you're working on it?

Yinon Costica 9:10

Yeah, absolutely. That will be I think…

Igor Tsyganskiy  9:12

Now I'm gonna just interview you!

Yinon Costica  9:14

–Uniform view across any type of cloud. I think the cloud offers us a really unique opportunity to have visibility into what's going on much faster than what we had in siloed machines and devices.

Igor Tsyganskiy 9:27

That's right. And you know, that goes to the benefit of both AWS and Microsoft, you know, the power of the cloud and other providers. The power of the cloud allows you to act a lot quicker on some of these issues across the board. But I hope that the solutions will emerge for on-premise as well. The on-premise work is harder across the board. And that in itself is what I'm mostly fearful of on behalf of what is happening across the enterprises around the world.

Yinon Costica 10:07

So what are maybe the biggest misconceptions or the mistakes that you have the opportunity to provide your point of view to security leadership that is now watching us? What are the things that they should be mindful of when they try to tackle this Log4j crisis?

Igor Tsyganskiy 10:25

What we've discovered, and we have a pretty tight environment, is that whatever we thought our exposure was, was probably 1/10 of what our exposure ended up being. And knowing that quickly was the game changer. So that's mostly what I'm worried about.

The reason I'm somewhat relaxed is because we did scan everything, we did use your tool. And we've discovered that our footprint is way bigger than we thought it is. Mostly, not just because of the software that we've written, but the third party software that we use, and knowing how all that interconnects and how that basically may have exposure to the external services. And basically being able to look at the whole Security Graph and see the dependencies, see the trees, and see the exploits that someone might take and trying to close that down really quickly.

Yinon Costica 11:23

Yeah. So one is the discovery of every instance of the Log4j, then also the ability to maybe see it in the context of what should be fixed first, and how to prioritize it: what is externally facing, what has high privileges, and then closing it out one by one. But what you point out is that basically the long tail of systems that we are not aware that they are vulnerable, this is what's going to haunt us for…

Igor Tsyganskiy 11:49

Well, I worry about two things. One thing and you know, I'm going to be working over the next few weeks to bring the whole industry together, because I don't think you just may be able to help by yourself. One of the aspects of having open source software is people reuse libraries, which is what's happening here. And then the second thing is very frequently people cut and paste the source code. And they put it into their systems. And knowing that exposure since the component is so widely used, and I know that's going to be pretty hard.

Yinon Costica 12:20

So we're going to see variations of the same…

Igor Tsyganskiy 12:24

What do you think, you're also in security? I think that there's a lot of variations. And that's where the industry needs to come together to help the rest of the world deal with the issue.

Yinon Costica 12:37

Yeah, I agree 100%, I think that from our experience with vulnerabilities, usually when there is a high profile vulnerability in one area, it shines it, and then we know how to patch it. But also others are looking for other vulnerabilities in this area, and it becomes like a hotspot. And I think this is exactly what we're gonna see across copy-pasted code, across the same libraries, versions. And because it is so pervasive, I think this is what's gonna really change the way we do security.

I think that the ability to really identify and remediate such systems in the shortest time before an attacker can get to it, I think that's what’s going to be the measuring factor for the industry in the upcoming months. Because the vulnerability is so easy to exploit, you log everything, and you basically change the user agent. And that's it, you're logging a user agent, and you can run code.

Igor Tsyganskiy 13:35

It's important to mention that the component has been in existence since 2013. So this is not a new vulnerability, it's a new vulnerability that has been made public recently. In some ways, maybe some state-owned actors may not have known about this, but I'm pretty sure one way or the other, state-owned actors across the globe, you can assume that they had an idea about it, right?

The danger here is it's so simple, that everyone from a state-owned player all the way down to a 13 year old kid or 12 year old kid who just wants to go out and play can go out and take over the server. You need to go cut and paste the code, there's plenty of libraries that are available, and you can mount your own attack. And you know, this does not need to be some country that we're afraid of, it might just be a middle schooler who's just exploiting something and accidentally does something wrong. And then you see that you can go from middle schooler to people who just adversarially want to do stupid things, to professional criminals to state-owned actors. The number of these people is so hard and so large, that that creates the fragility and that's what I'm mostly fearful of, right? That is just basically an ecosystem. It's like instead of mining Bitcoin, you're going to go mine this and you know, maybe you'll get some money.

Yinon Costica 15:03

So the World Economic Forum already put cybersecurity as the second most serious threat to our existence after global warming, of course, or climate change. But basically, I think that this actually accelerates it.

So we've covered already two things that we have to do: one, we mustn't assume that we have identified all of the instances because we had one tool that told us we are good. We basically need to now think about it again and again, every time we ship, every time we deliver, and identify how are we using that component and make sure we replace it and patch it, that's one.

Second, we need to go and find any instance that might have proliferated through code into other systems, so we can identify all of the impacted components for real.

And third, we need to be able now to monitor what threat actors are doing. So we know where we need to be quicker, what exploits are available. What other type of measures or security controls should we put in place beside patching when we think about countering this?

Igor Tsyganskiy 16:09

So you know, just to answer this question and build on the question you've had before. As you might know, we work a lot for Microsoft, to move into the notion of zero trust networking environment, right? Where basically your assumption of what you trust, and what you do not trust is very different. What you're talking about is what does it mean to have a production environment from the security standpoint of view? What does it mean to do pentesting? What does it mean to do red teaming? What does it mean to deploy software? And how do you assess risk?

So the way I like to think about it, as you know, to build on your previous question, I call it continuous risk assessment. Continuous risk assessment is essentially what you guys said: the risk is changing every day. And you need tools to basically assess your risk and adjust. The risk is not static. Since the risk is not static, you have to constantly monitor for the shape of risk to change. And for that, you need tools, and you need development techniques. And I believe you're on the forefront of that particular thing.

So between zero trust, modernization of your environments, this continuous risk assessment, all those things need to be done at the same time. My belief is people who are building mission-critical applications, you know, there are no shortcuts. One of the benefits of commercial software is that someone takes responsibility for commercial software. The shortcut with open source software – it's great, and it gets fixed over time – but what we're seeing now is one of the shortfalls of open source software, where it's so widely used, yet no one is responsible. If it would be a commercial piece of software, at least you can call someone, you know, you can call Microsoft, you can call AWS, you can call Google, you can call Oracle, there's someone to call and they can patch it. In here, you know, it's free, great. Everyone deployed it, it works great, these are the benefits. And then the downside is the fact that someone has to be able to then fix it and maintain it over time.

Yinon Costica 18:28

So the continuous risk assessment that you mentioned is actually based on the risk graph that takes into account all of the environmental context, such as the permissions, the network exposure, the vulnerabilities, the importance of the workload, the deployed, and basically correlates it all of the time. So you have a prioritized risk. This is a machine that is Internet facing and vulnerable and has admin permissions, obviously, you need to identify it and tackle it first, before you go to the siloed machine that doesn't have any permissions anyway. So creating this continuous risk assessment is identifying what are the key elements? If you have one hour to do security, how do you become most effective and do it continuously every day? Because the environment is so dynamic. This is like the third element you discussed. And I think that it actually gives a good segue.

Last year, ironically, same time, same period, it was SolarWinds. And all of the learnings we had over the past year from SolarWinds, and I think you touched on it, but when you try to fast forward like a year from today, what would have been like the learnings we can start having from this Log4j vulnerability?

Igor Tsyganskiy 19:47

Well, and I just want to highlight that the vulnerability we had last year was a state-sponsored attack – very, very sophisticated and very targeted. And in that sense yes, the downside is the attack; the upside is at least there's someone to negotiate with. You know, in here, there's no one to negotiate with. So as we're going to look into a year later, my belief is that we're living in an extremely interconnected world. People talk about the thing, that software is eating the world. And as software is eating the world, hey, software is buggy. And not all the software is secure. And so that means the world is going to be buggy, and the world is not going to be as secure.

Third, very important thing, Moore's law, essentially, everything gets more and more efficient over time. So the world is getting more efficient. And that means also that hackers are getting more efficient. And we need to protect against all the attackers in the world. But one attacker, you know, the one guy or one girl, they can go and do whatever they want to do. And so if they're getting more and more efficient, this interconnected world, as it changes over and over and over again, the asymmetry of it, I think that over the next year, and the years to come, people will realize that there is no more separation between cyber threats and global threats. And those two things are combined into one, that is just basically a threat assessment. And cyber risk is equal to real risk.

You know, we went from malware to ransomware, to now killware if someone takes over the hospital, and that has happened multiple times this year, when, during the pandemic, whoever is taking over the hospital, and whether they know it or not, that leads to people dying, right, because you know, pharmacy is not working, some digital equipment is not working because the networks are offline. That's an example of killware. And so, basically, you know, I think it's a loss of innocence. That's the best way I can describe it. There will need to be a lot more security assessment, especially around mission-critical applications across the world.

Yinon Costica 22:30

No, it's a really powerful statement. And I completely agree with that direction. Loss of innocence is a good way to say that. I think we looked at security for a long time as something that goes after the fact. And now we came to the realization that security needs to be by design, by the way we build the stuff, but also by the way that we respond to it fast enough. It's a necessity that we have to build this muscle. And again, as a community, because as you've seen, it's the dependencies, the interconnectedness. And this will become a challenge.

Igor Tsyganskiy 23:08

Yeah, it definitely both takes a village. And because the world is interconnected, the nature of security tools cannot be isolationist, you know, it's not enough to just secure yourself, you have to secure others. Just because my software is running in any cloud, whether it's Microsoft Cloud, Google Cloud, AWS cloud, or in the data center, if my software is secure, it's not enough. All the other software around me has to be secure. And that software might not be something that's produced by me or even used by me. Which comes back to that notion of which I think you guys are pushing on and pioneering, which is “Security Graph is everything.”

Yinon Costica 23:49

Yes, Security Graph includes also the components that you rely on, and what's their security posture. And if you have a vendor ecosystem that connects your environment, you basically need to be aware of it and analyze it as well. Maybe the Security Graph is a response when you think about how security responds to the interconnectedness. When we think about the world as graphs and connections between elements, then we are able to basically bring it together and assess the risk in a more, let's say, asymmetric way to how attackers are thinking, right?

Igor Tsyganskiy 24:26

I just thought about another thing you asked me: what I think will happen. One of the reasons I'm excited about your company, and that's what’s being showcased right now, is I believe that the same impact that social graph had on a bunch of industries, whether it's advertising or a bunch of others, Security Graph is going to have a huge impact on the world. And a company who owns the largest Security Graph will end up having quite a lot of leverage and ability to help its customers deal with upcoming threats. And from everything I'm seeing, you're the closest. I mean, you've been thinking about it for a long time. You have the best implementation of that so far.

Yinon Costica 25:25

How much time did it take you to deploy Wiz across your entire environment?

Igor Tsyganskiy 25:30

Well, that's what I'm saying, under threat, it took us hours to deploy Wiz. So you've paid for yourself in a matter of one week – paid for five years of subscription to Wiz. One week, because we were able to, from the time that the risk got identified, to the time when we could assess the impact of the risk on our organization took two or three days. And last year, given what we knew, it would have been unknown, it would have been infinite, we would have closed 15% of what we've learned we have. If you don't know where your threat is, if you don't know where your doors are, the only time you find out is by doing different types of scans – it would be a while. And that while grows proportionally with the size of the environment that you have. And so it was very fast to bring it up.

We were fortunate enough that we were doing pentesting on your environment right prior to this, so, we spent the whole month pentesting your environment and making sure that it's actually secure to let you scan our stuff. And we came to the point where your pentest got cleared, we worked everything out. So then we could actually use the software. And from use to result, it was minutes to days. Now what we did is we accelerated deployment and deployment was very, very fast. And in some ways, it was fast for both multi-tenant deployment on those instances where we do not think that we need the security to be the highest level, and Bridgewater’s own deployment, where we don't want anyone else to see what is happening. So the fastest deployment to value that I've seen, period, especially during the crisis.

Yinon Costica 27:44

And we're seeing already that it's being used in numerous campaigns in ransomware, in deploying malware, installing backdoors. And yeah, this is the challenge – how do we keep our ears open and respond as fast as possible to the risk we identify?

Igor, in general, as we wrap this session, any final thoughts that you would like to share or guidance you would like to give to other security leaders encountering this?

Igor Tsyganskiy  28:19

Yes. And then I'm going to do that publicly, outside of this forecast. I haven't been able to find another tool, if you're deployed in the cloud, right now, and you need to close down your issues, go talk to Wiz. And this is not a paid promotion. I’m mostly saying this because they have helped me and I have not seen anything else right now that can give you as big of an impact.

And that's why I agreed to do this podcast, mostly, to give you guys enough credibility, you know, in the most efficient, fastest way, for people who are searching on how to solve that problem, to know where the solution might be. And we're not getting anything out of it. But my advice is, if you're dealing with this problem, and you have currently been running software in the cloud, or services in the cloud, and you need to assess your risk, Wiz will probably get you there faster than anyone else.

Yinon Costica 29:25

Thank you so much, Igor and for joining us and sharing your views on the threat landscape and where we are heading. And also I want to take this opportunity to wish all the teams out there, the responders and the security teams, the fastest path to remediating this vulnerability and I hope that this won’t evolve to be a long-lasting crisis that we will need to face.