Introduction
This week I’m joined by Mike Kail, the Chief Technology Officer for Everest.
Mike starts off by outlining what immutable infrastructure is in a cloud-based world. He digs into why this is so critical to the enterprise. He discusses the problems with Layer 8 and how it differs from existing processes an enterprise might use. Mike goes further into the dangers that exist in a cloud-based world. He talks about specific changes enterprises can make to address these risks.
Speaker Profiles
Mike Kail Twitter: https://twitter.com/mdkail
Mike Kail LinkedIn: https://www.linkedin.com/in/mdkail/
Everest: https://everest.org
Audio File
Featured Hashtags
#CIO #Leadership #CyberSecurity #Cloud #DigitalTransformation #CxOitk
Podcast Transcript
Tim Crawford 0:00
Companies are looking for new ways to transform their business. Technology plays a critical role in this transformation. Speed and innovation in both technology and thinking are key to this shift. Hello, and welcome to the Cxo in the Know podcast, where I take a provocative but pragmatic look at the intersection of business and technology through the lens of leading CxO executives. I’m your host Tim Crawford, a CIO and strategic advisor at Avoa. This week, I’m joined by Mike Kail, the Chief Technology Officer for Everest. Mike starts off by outlining what immutable infrastructure is in a cloud-based world. He digs into why this is so critical for the enterprise. He discusses the problems with Layer 8 and how it differs from existing processes an enterprise might use. Mike goes further into the dangers that exist in a cloud-based world. He talks about specific changes enterprises can make to address these risks. Hey, Mike. Welcome to the program.
Mike Kail 1:02
Hey Tim, thanks for having me.
Tim Crawford 1:03
So Mike, CTO of Everest, back again to talk about cybersecurity, and this time we’re going to talk about immutable security for immutable infrastructure. So let’s set the stage and talk about what is it and how does it apply to the the enterprise?
Mike Kail 1:23
I think let’s start with defining and level setting on what immutable infrastructure is and why that’s important, especially in a cloud native world. So the basic definition of immutable infrastructure is that the instances or containers and the applications and configurations are not modified in your running infrastructure. If you make any modifications, you redeploy. So you shoot the the running instances in the head and deploy quickly. So the changes are tracked properly and not done out of band. And then, if you think about infrastructure being the base of your stack and security posture, you want to have the same immutability to your security policies and how you approach that as you do with infrastructure and have that tightly coupled.
Tim Crawford 2:11
So, why is this so critical to consider? I mean, there might be the obvious that someone could look at, but from your perspective, I know there’s more to it. Why is this so critical?
Mike Kail 2:25
You really want to have good rigor and discipline around making configuration changes and removing any temptation of the layer eight of your infrastructure, which is the people layer, to make changes and and all of a sudden, start testing configuration changes in production because we all know how that movie typically ends.
Tim Crawford 2:48
What you’re talking about is different than what we’ve traditionally talked about in change management or some of the other processes that enterprises have put in place for decades.
Mike Kail 3:00
It probably goes against some of the continuous delivery or A/B testing in the software application world, which can be much more flexible. I think infrastructure is typically very fragile and breaks often, which is why we need to design for resiliency and and use some chaos engineering approaches. And security also needs that level of rigor because it’s too easy to make mistakes in policies and not be aware of it.
Tim Crawford 3:29
And I can imagine how that’s really critical. I mean, especially as you think about production infrastructure within the enterprise. So let’s maybe pick that apart a little bit. What are some examples of risks that you might encounter if you don’t step into this. Let’s say that someone goes, “Okay, well, that seems a little far-fetched for me. What are some of the things that maybe they’re not thinking about? I
Mike Kail 3:53
think the first classic examples are that you expose something that you didn’t intend intend to. AWS s3 buckets are often in the press, network access control lists, security group configurations. Those are some of the the main ones that happen. If you don’t have that level of discipline, you run a pretty high risk of what’s called configuration drift. Because if you make a change against the running infrastructure and policies and don’t commit that back, then the next time it’s not deployed, you run into weird bugs, and it’s very hard to debug. If you think about the infrastructure assembly pipeline, you can now potentially include components such as container images that have libraries or or packages that have known vulnerabilities. So you really need to think about an infrastructure software delivery pipeline. The concept of least privileges and not allowing infrastructure to be deployed and managed with escalated privileges. To think about the super user in the Unix world, you don’t always need to have that level of access. To deploy, and you should use proper role-based access control and manage that.
Tim Crawford 5:05
Okay,
Mike Kail 5:06
and then the final one is both security and cost. So call it shadow cloud or orphaned resources. You now could have long-running instances that have misconfigurations that allow a potential hacker to breach, and your cloud cost meter is always running as well, and that won’t be a fun conversation with your finance team and CFO.
Tim Crawford 5:30
Well, especially when the bill comes and you hit the sticker shock, as we often hear in the press today in the public space, sticker shock from cloud and and whatnot. If I kind of step into this maybe a little further for a second, and talk about your first item on that list, exposing something you didn’t intend to. I mean, this is a risk that every enterprise has for anything that’s critical. So, are you suggesting that this is a concern that you should really be applying like peanut butter across everything you do, or are there certain things that this makes more sense for versus others?
Mike Kail 6:09
I think it makes sense for literally everything, and I’ll sound like a broken record for especially for those who follow me on Twitter and call it zero trust, meaning trust nothing, verify everything.
Tim Crawford 6:21
And we talked a bit about that in the last podcast. So if you missed that, definitely go back and take a listen. You know, one of the things that often comes up in these conversations is this concept of infrastructure is code, and so what I’m curious about is how does infrastructure as code kind of play into this, and what does it mean to secure that? Especially as you’re talking about immutable security for immutable infrastructure.
Mike Kail 6:49
So, infrastructure as code, as most people probably already know, is a declarative way of outlining how you’re going to configure and deploy your infrastructure. You need to take the same approach that you do with your application code, meaning do static code analysis, dynamic, and shift left and do that as part of your delivery pipeline instead of doing pen tests periodically or config checks periodically, because otherwise you have an unknown window of potential exposure,
Tim Crawford 7:23
but this is really talking to being more proactive when you think about security as it pertains to infrastructure, as opposed to reactive.
Mike Kail 7:32
Absolutely, I think if you’re using something like GitHub to manage your application code, you should use that same methodology to maintain your infrastructure code plans, and once again have proper discipline and rigor around it, where your DevOps engineers perform code reviews or plan reviews as part of the pull request process. So then nothing gets deployed without it having at least a second pair of eyes.
Tim Crawford 8:00
Yeah, and I want to talk about DevOps in a minute. Do you think that infrastructure teams today, you and I have both led IT organizations over the course of our career? How well equipped do you think infrastructure teams, specifically the folks that that have responsibility for this infrastructure, how in the enterprise specifically? So trying to exclude web scale and startup for a minute. How prepared do you think they are to wrap their arms around this and actually execute upon some of the things that you’re talking about?
Mike Kail 8:33
I believe we’re in the very early innings of the cultural understanding of how to do this. There’s still a lot of click and forget in GUIs, whether you know it’s in Microsoft Windows deployments, VMware. There’s a lot of GUI management without understanding how how things work under the hood, which means I think there’s significant opportunities for vendors to help address this, both from the education and awareness to other technical solutions,
Tim Crawford 9:06
but do you think this is something that should be addressed by vendors, or should be addressed by infrastructure teams, or both?
Mike Kail 9:15
It first should be addressed by technical leaders to build that muscle in their team, and perhaps augment that with vendor solutions. But a vendor solution is not going to fix your your leadership challenges and cultural challenges,
Tim Crawford 9:33
right? And I kind of go back to it’s not trust but verify. It’s trust nothing, verify everything, right?
Mike Kail 9:42
Yep, exactly. Never trust, always verify.
Tim Crawford 9:44
That’s gonna hit. It’s the Mike Kyle never trust, verify everything show. Okay, so let’s move on to DevOps because I think DevOps is something. It’s been a hot topic for. A number of years now, and there’s been this newer topic called DevSecOps. And before you literally throw something through the screen at me while we get into this, help me understand: Are they different? Are they not different? Should we be talking about them separately? What’s your take on that?
Mike Kail 10:22
I think some people tend to have a visceral reaction to the term DevSecOps, and it’s called Sec DevOps or DevOps Sec. I think let’s avoid the religious debate here, and let’s settle on that in DevOps the Sec part is silent, but should always be part of that culture. And let’s remind everybody that the term DevOps really should be a cultural approach. It’s not a job title, a set of tools. It’s a cultural approach to a few ways of deploying and managing infrastructure and collaborating with your teams. And the key part there is collaborating, and security should be part of that overall culture, not a team or organization outside of it.
Tim Crawford 11:08
You know, one of the challenges in organizations, enterprise organizations, and IT has been the dev team, the application development team, and the infrastructure operations team have been like church and state, like they never come together, and that’s really where one of the areas that DevOps was intending to build a stronger bridge between security has always been this additional pillar that sat off to the side and was an afterthought. How do you bring that security piece into the culture. I mean, it seems like that would be easier said than done. But how do you bring it in, and who starts with that conversation?
Mike Kail 11:50
Yeah, it’s church and state, and then the Department of No with security was been unfortunately kind of the the cultural existence.
Tim Crawford 12:00
Well, to be fair, let’s call a spade a spade. IT as a whole has been called the Department of No. It’s not just the CSO and their organization, but I understand your point.
Mike Kail 12:11
Yeah, absolutely. It’s been a contentious handoff in past life. It was development handed things over to operations. Things broke. Operations yelled at development, who yelled at somebody else, and and within operations you had the network systems and storage teams all infighting, and then you had security trying to keep up with all of this, and coming in and laying down their set of rules with no one speaking the same language. So once again, this comes down to leadership or lack thereof, and setting the proper context, developing the proper culture and team alignment, and eliminating that infighting and reminding people that they’re all there to solve for advancing the company, and fighting does not help that most certainly.
Tim Crawford 12:58
But arguably, security sometimes. sometimes works against the usability or the speed in which an organization may want to work, and I’m not saying that I agree with that, and I I understand some of the complexities to it, but I could just imagine folks listening to this podcast going, “Well, wait a second here, you know, ops and and devs-they just want to go, go, go. They want to run with scissors, and security gets stuck with the bag, going, “Hold on, let’s think about this. Let’s be responsible. You know, it’s almost like the security teams come into this conversation saying, “We’re the responsible adults for the rest of the children in the organization. I know that’s a very detrimental or negative way to look at it, but I’ve seen this culture play out in a number of organizations. How do we get to a point where you actually do have a set of equals, and everybody’s working collaboratively to this? Is this where the CIO or head of IT needs to jump in and and say, okay, hold on, enough. This is the way we’re going to move forward.
Mike Kail 14:06
I think the the main challenge is that you want a development team delivering feature updates at a high velocity. Oftentimes, the security team is not involved, or this whole meme concept of shifting left. So the security scanning, testing, and verification is not done early on in the development process. So then they’re overwhelmed because you have this massive amount of changes being delivered continuously, and they don’t have the proper staffing or tooling to keep up. So you have to take one of the core tenets of DevOps being automation to automate security as early as possible in the development lifecycle, and have a continuous feedback loop, so that security team is not caught flat-footed, and can work in conjunction with the dev team to resolve any high critical vulnerabilities as quickly as possible. And to answer your last part, once they can, that’s the Cxo’s job to foster and drive that culture.
Tim Crawford 15:07
And to be fair, I mean these are all good people. It’s not like they’re trying to to undermine others intentionally, but it does kind of work against each other when you have different priorities and initiatives that don’t necessarily align. So I get your point of the leadership having to step in or not having to, but really kind of setting the tone and cadence for the organization as a whole.
Mike Kail 15:32
Yeah, and there needs to be a continuous level of collaboration as well, and and people talking and removing assumptions because I don’t think I think in general humans tend to assume the worst, especially if they’re not talking. So having that open channel of communications, especially in this work from anywhere situation, all of us are in.
Tim Crawford 15:52
That’s right. That’s absolutely right. So Mike, you know, you talked a little bit about shift left. I want to ask you about where some of these key areas are to look for. So, you know, someone listening to the podcast is going, “Okay, immutable security for immutable infrastructure. Where should I start? What are some areas I should look for? Is kind of points that kind of get me the biggest bang for the buck to get started. Let’s start with shift left for a minute. So, shift left. Can you articulate what exactly that means if someone hasn’t heard that term before?
Mike Kail 16:25
Yeah, I think the first use of it was in the traditional QA process, where QA testing would happen at the last stage or the far right of the software development pipeline, or even post deployment. So by shifting left, that means you were moving the QA testing process as close to the code commit of the pipeline as possible, and so then the the same thing has been applied to security testing, whether that’s static code analysis, container scanning, dynamic scanning, and then pen testing, and the same should be taken in your infrastructure delivery pipeline, meaning I want to start doing my infrastructure code plan reviews at Code Commit because that is the farthest left you can get in the the pipeline.
Tim Crawford 17:12
So if I’m still leading an organization, so let’s assume for a minute I’m a CIO, I’m the head of IT, I’m still leading a more traditional organization where I have my infrastructure teams and I have my application teams, right? You and I have both seen this organizational model. How do I start to align those infrastructure teams to be able to get into that that mode of thinking? Because these are traditionally these are not folks that that do development. They might have done scripting in the past, but not necessarily development like an app developer would have, or someone that that’s more ingrained in this. How do they get more involved in that infrastructure delivery pipeline?
Mike Kail 17:52
Yeah, I think the first thing you need to remind them is that they don’t need to be hardcore developers. Like scripting and configuration syntax is good enough to get started. It’s how do you take an automated, repeatable approach? So the first thing I would do is get you know some of my dev leaders and IT infrastructure leaders together and talk about best practices around automation and repeatability, and how do they assemble their application and collaborate on some ideas on how to start defining their the IT infrastructure as code or templates, and automate that assembly and deployment in stages. You’re not going to go from A to Z, and you need to set that context, and you know have them collaborate together. So then, once again, they come back to they start speaking a common language. There’s trust and respect being fostered between the two, and you also are getting best practices for your infrastructure pipeline.
Tim Crawford 18:51
That’s great, and I love two of the things you said there that I think are incredibly important: trust and respect. That seems to only come when you start getting that communication and that collaboration wheel going, that flywheel going, of conversation. What are some other key areas that you would suggest folks take a look at as they’re starting down this path?
Mike Kail 19:14
I think you have to understand that that things will still fail, or configuration changes will break and be wrong. It’s how quickly can you remediate them? Like failure happens, how can you build more and more resiliency into your infrastructure?
Tim Crawford 19:31
Okay, great. Are there other aspects though that come into play, especially as you think about the people and the process aspects?
Mike Kail 19:40
I think in a proper infrastructure delivery pipeline, you eliminate people from being involved in that process by using the proper technology. People define the code or templates, but then the assembly and delivery of that is done completely automatically, and then you have some checks to make sure that. That it’s all running, which you should have metrics and monitoring in place as well. I think that’s an important component here, and so a tight feedback loop. I think that’s really what to think about. Once again, going back to let’s call it traditional IT infrastructure, where it’s point and click on a GUI and deploy and and hope. This is the complete opposite of that,
Tim Crawford 20:23
and a lot of what you’re saying is really trying to build repeatable process into the mix that includes the development, the infrastructure, and the security functions together collaboratively.
Mike Kail 20:37
Yeah, I think you could take kind of the Uda loop construct here. If you’re starting green field here, that might be a good methodology to look at.
Tim Crawford 20:48
Excellent. Well, this has been informative. I think there’s a lot that lot more we could unpack here, but in the interest of time, we’ll have to leave it right here. Mike, hey, thanks so much for taking the time to talk about this today,
Mike Kail 21:01
anytime. Thanks again for having me on.
Tim Crawford 21:05
For more information on the Cxo In the Know podcast, visit us online@cxointheknow.com You can also find us on Apple Podcasts or wherever you listen to your podcasts. Please subscribe and thank you for listening.
Discover more from AVOA
Subscribe to get the latest posts sent to your email.