How DevOps and Security Can Work Together

During his career as a cybersecurity engineer, Matthew Rosenquist saw his share of frustration. There was the CEO who dismissed his own team after it showed him vulnerabilities in the company’s main product. There were the software engineers who lacked security experience but didn’t care to acquire any.

“In their minds, they were brilliant coders and that was enough,” says Rosenquist. “They said, ‘That’s not my problem. I just code it to where it works and the rest is some security guy’s problem.’”

That antagonistic attitude is understandable. Software developers live in a world of iterative design, rapid releases and “minimum viable product.” In the interest of time and a focus on code, they aren’t concerned with much else, including security. They often perceive it as a drag on the schedule, or even on the product itself.

“If you’re a developer or engineer, this is your baby,” says Rosenquist, a cybersecurity strategist and board advisor who spent 23 years at Intel and is currently CISO at Eclipz.io, a software encryption company. “You’ve got deadlines. You’re being evaluated on how well you get your code to work on time. Now throw in a cybersecurity person who doesn’t know your product. ‘You want me to do X, Y and Z? That’ll blow my timeline.’ This friction builds up. The only thing worse is not to do any security at all.”

Mending the rift

The friction Rosenquist describes isn’t new. It has grown out of a longstanding rift in the enterprise between DevOps and security teams and worsened by the challenges of a global health crisis. With millions of remote workers accessing sensitive data systems over tens of millions of endpoints, including laptops, PCs and tablets, vulnerabilities and security breaches have skyrocketed. According to a survey that Tanium released in 2020, North American businesses lose $700 billion a year as a result of IT downtime. The rift among the teams isn’t helping.

But work together they must. An emerging practice called DevSecOps, for example, aims to build bridges between the dueling fiefdoms by integrating security team members and processes into the software development life cycle to catch vulnerabilities early. Even so, deep cultural issues, a global skills shortage in security and organizational structures still handicap cross-team harmony. As IT leaders look at ways to overcome these hurdles, several best practices have emerged, beginning with clearer buy-in at the executive level and developing a collaborative ethos. Here’s how to get started:

Set the tone from the top

After two debilitating worms hit Microsoft in 2001, Bill Gates sent a memo the following year to the entire company, announcing that security would become an integrated part of coding of all of its products: “When we face a choice between adding features and resolving security issues,” Gates wrote, “we need to choose security. … We’re in the process of training all our developers in the latest secure coding techniques. … Our software should be so fundamentally secure that customers never even worry about it.”

The result was the company’s groundbreaking and much copied “security development lifecycle,” a set of practices that builds security into the product as it develops and matures. Companies across the tech landscape have since adopted — and adapted — these security practices.

[Read also: Strained Relationships Between Security And IT Ops Teams Leave Businesses At Risk]

The lesson: Adding steps for security to the product development lifecycle may very well impact timelines. And if higher-ups aren’t making it a priority, it’s going to be a much more complicated sell lower down the food chain. “From the top, you have to get a business understanding that is in alignment with corporate goals,” says Rosenquist.

From the top, you have to get a business understanding that is in alignment with corporate goals

David Hahn has seen this firsthand. As a CISO who has worked in cybersecurity at Silicon Valley Bank, media conglomerate Hearst, Intuit and others, he has successfully partnered with IT leaders by agreeing that cybersecurity must have a seat at the table and conveying to their teams the importance of different skill sets that ultimately must work together. “You can’t come in later and have your role be the bad guy,” he says.

Shifting accountability

One way to bridge the divide is to make security a more integral part of the development process. Nearly two-thirds of companies today don’t involve security teams at early stages of IT projects, according to research by EY.

One way to help close that gap, experts say, is to adopt “shift left” software testing—prioritizing testing much earlier in the development cycle, rather than bolting it on after the fact. It also encourages, if not requires, more collaboration between security teams and developers.

Those kinds of practices help break down old silos. “It’s about creating a new team,” says Hahn. “Everyone should adopt development, security and operations skills, but some will have more of a background in one than others.”

One way to get them there is with training. Providing software developers with security skills is “a growth opportunity,” says Hahn. Likewise, many security analysts would benefit from learning basic coding. “They can start with a coding language like Go or Python to understand procedural programming and the intricacies of software development,” says Ax Sharma, a security researcher and engineer at UK-based software developer Sonatype. “This will help reduce skepticism of security’s ability to participate in the team deliverables.”

DevSecOps: Finding new ways to think about security and development

If security and software development teams can discover a new sense of kumbaya in the coming years, it’ll be in part because they embraced a style of working that has been wildly successful in repairing another old enterprise rift—between software developers and IT operations teams. Over the last decade or so, developers have worked much more closely with IT ops to integrate the two functions and ship new software out the door more swiftly and efficiently than ever before.

It’s a methodology called DevOps, now in practice in some form by an estimated 62% of enterprise organizations, according to GitLab. DevSecOps builds on the same notion, bringing security testing into the process during development, not afterwards.

DevSecOps doesn’t just involve security teams earlier in the process, it allows the collective group to explore risk tolerance. For software engineers concerned that their cybersecurity colleagues will slow development, Hahn says there’s a way to get everyone on the same page and keep code moving: lay out for them what happens if vulnerabilities aren’t caught early. “Bring in a threat model,” he says. “When you start mapping it out, it becomes something tangible. It becomes an engineering discussion versus ‘David’s opinion was not great, I don’t like that.’”

A threat model gives both sides a way to look for compromise. For example, software engineers are keen to create an MVP or beta version of a new product that includes a full database of proprietary records. Security doesn’t want that database anywhere near the product until a full test for vulnerabilities has been completed. A threat model could point to the risks inherent in making sensitive data available in an unfinished product.

And then it’s up to the security team to speak up. “They need to say, ‘We don’t need to take that risk. We could put in dummy data,’” says Hahn. The EY report similarly finds that security teams could do a better job of communicating such risks.

When bringing the two sides together, Sharma suggests starting with small-scale efforts, such as reviewing the state of security processes and the threat levels on all products. That will create awareness of where vulnerabilities get introduced. Then the two teams can use security automation tools, such as malware detection software, to hunt out remaining vulnerabilities. “Think of the first month of your efforts as an observation phase so that teams can see the effect, work together and iterate to remove waste,” says Sharma.

When things go off the rails

Sometimes things will work. Other times, they won’t. When the two sides fail to work together, look for two causes, says Hahn: scope creep that throws a project off timelines and process, and ego. “As in people not adhering to principles they’d agreed to,” he says.

At that point, it’s time to pause development, even if it means pushing out your release date. Better to discover a security vulnerability that sets a project back two months, says Hahn, than to face a potential catastrophe after a public launch.

“There’s always going to be a cost,” says Rosenquist, the CISO at Eclipz.io. “But you’re mitigating against losses or ensuring you’re compliant with regulation.” Uniting your security and IT crews, he adds, isn’t just good for the employees. “It can be a competitive advantage.”