Diving into DevOps

The DevSecOps Skills Gap

Without baking in proper training and education, a new study suggests that the rush to adopt DevOps practices might be leading enterprises to an insecure place.

• By John K. Waters
• August 17, 2017

Few enterprise IT trends have evolved from buzzword to must-have as solidly as DevOps. Virtually everyone agrees that a software development and delivery process that bridges the traditional gap between dev teams and operations professionals is a good thing for the enterprise, an approach that is almost certain to deliver software faster and more reliably.

And yet, the results of a just-published survey ("DevSecOps Global Skills Survey: Trends in training and education within developer and IT operations communities") suggests that the rush to adopt DevOps practices might be leading enterprises to an insecure place.

Sponsored by application security firm Veracode and DevOps.com, a site dedicated to DevOps education and community building, the survey of IT professionals uncovered the disturbing fact that "developers today lack the formal education and skills they need to produce secure software at DevOps speed."

Among survey respondents, nearly one in three said the IT workforce is unprepared to securely deliver software at DevOps speeds, and just over half said they believe it's only somewhat prepared.

The report's authors warn that organizations that "plow forward with DevOps makeovers that don't bake training and education best practices into their transformation strategies" risk accumulating a kind of "personnel debt," analogous to the kind of technical debt that accumulates when an organization continues to choose "quick-and-dirty coding or architectural shortcuts over more difficult but thorough approaches."

"Organizations that fail to address these debts now will likely have to pay later with interest," the authors wrote. "Such as surcharge could come in the form of stalled-out or failed DevOps efforts, as well as heightened risks to software infrastructure that could even cause costly breaches and theft of intellectual property."

Among the survey's findings:

• Nearly 40 percent of organizations in the survey said the hardest employees to find are "all-purpose DevOps gurus with sufficient knowledge about security testing."
• Almost seven in 10 developers said their organizations don't provide them with adequate training in security.
• The top two skills hardest to find in IT Ops talent are vulnerability management and containerization skills, respondents reported.
• More than 76 percent of college educated respondents said they weren't required to complete any courses focused on security during higher education.
• An overwhelming majority of DevOps professionals -- more than 64 percent said they learned their most relevant skills on the job.
• Only 4 percent said they learned their most relevant skills from third-party training -- even though more than one in three respondents said this kind of training, in the classroom or through e-learning, would be the most effective way to gain new skills.
• About half of respondents said their employers paid for additional training since their entry into the workforce.

The results of this survey are sobering, but not surprising. Security has long been the ugly stepchild of the software development process. Fortunately, this version of that old story has not gone unnoticed. Secure DevOps -- increasingly, DevSecOps -- was a hot topic at the big RSA Security conference held in San Francisco in January. One of the presenters at that event, Jeff Williams, CTO and co-founder of Contrast Security and a founding member of the Open Web Application Security Project (OWASP), described the DevOps security challenge to attendees. Among other things, the transition to DevOps disrupts traditional app security practices, he said, which typically involve specifying the requirements up front and then testing just before deployment.

"With DevOps, this process simply won't work," he said. "The feedback is too slow, too late, and too inaccurate, and it slows down development."

Because code is released continuously in a DevOps shop, security must also work continuously, he said. Developers need instant, accurate feedback on their code, and security should be confirmed before every release. The only way to make this happen, Williams said, is to "turn security into code," by fully automating the testing process.

"When the security tests are fully automated, DevOps projects can build and deploy software with confidence even without involvement from security experts," he said.

Unfortunately, finding a source for training appears to be another challenge. The report cites the observation by Daniel Cuthbert, a longtime security researcher, expert in penetration testing and COO at security consultancy SensePost, that continuing education opportunities for DevSecOps are still few and far between.

"[T]here just isn't much out there," he said. "And a lot of the developer training courses we've seen are still teaching methods from a decade ago -- waterfall security, pen testing at the end. "[They're] expecting developers to be hackers, and they're not. Developers build stuff. They don't want to be hackers."

The Veracode/DevOps.com report emphasizes that the DevSecOps skills gap isn't exclusively an app dev problem.

"DevSecOps will require security practitioners to re-evaluate tooling and processes to fit better into the continuous delivery process," the authors wrote. "And that will require security personnel to bone up on their baseline knowledge of how developers operate and the constraints these software engineers face. Security people must meet the development team halfway on bridging the knowledge gap."