Diving into DevOps

Continuous Delivery Enables DevOps Reality

According to the self-described "Code Curmudgeon," leveraging the advantages of DevOps will require organizations to shift from test automation to continuous testing -- which are very different things.

• By John K. Waters
• December 26, 2016

Arthur Hicken bills himself as "The Code Curmudgeon" and, to be sure, he does not suffer security bumblers lightly.

On his personal Web site, he maintains "hall-of-shame" lists of organizations that have fallen prey to SQL injection ("totally preventable") and IoT device hacks ("some call it the Internet of Evil Things"), but when he's not ranting about software security, he's actually a pretty nice guy.

Hicken is Chief Evangelist at Parasoft, which develops automated software-quality solutions, including the Parasoft Continuous Testing platform. He was one of our presenters in the App Dev Trends track at this year's Live! 360 event in Orlando. During his session, "Continuous Testing in a DevOps World," he argued that leveraging the advantages of DevOps and the new, accelerated software development lifecycle will require organizations to shift from test automation to continuous testing -- which, he insisted, are very different things.

"'Continuous' actually refers to a spectrum of things," Hicken said. "Continuous build, continuous integration, continuous test, continuous release, continuous deployment. To me, 'continuous' is the holy grail, and it really builds on top of the DevOps movement. If you are going to move beyond continuous build and test into continuous deployment and release, you've got to bridge that gap between the guys who write the code and the poor people who have to make it work somewhere. Continuous enables the DevOps reality."

The essential goal is the DevOps Pipeline, he said, in which the planning, coding, build, test, release, deployment and operations processes are integrated and as many things as possible happen automatically.

But just because an organization is employing automated processes, doesn't mean it'll find an easy path to true continuous testing. "I often hear people say, 'I'm automated, so I can easily do continuous,'" Hicken said. "Well, you have some of the pieces, but automation is only a first step. Then you have to look at your processes, things like intermittent manual dependencies, buttons you have to press. Your tests need to start producing binary decisions. When the tests run, am I comfortable saying it's done, it's good, it's safe? When can I push it out the door? With continuous, you have to have 'yes/no.'"

"If any of you are dealing with a noisy test suite that's got a 10 or 20 percent failure rate that you're used to living with, you shouldn't be comfortable with that," Hicken added. "In fact, you should turn it off. That test is not meaningful. To get to continuous, you have to stop managing the noise; you have to manage to zero."

Complexity is one of the chief barriers to continuous testing, Hicken pointed out. Nowadays, pieces and parts of a system come from everywhere, definitions change with upgrades and new versions, components are not always available, and in many cases, you can't test with real data without causing problems. And technology is faster and more distributed than ever, thanks largely to the advent of containers and microservices.

But often the problem is one of process maturity. "Typically, when you find you can't get to continuous, it's because you're doing something immature," Hicken said. "A mature development process has a stable regression test suite, where they know at least things didn't get worse."

Hicken advised attendees embarking on an effort to implement continuous testing in their organizations to use test stubs (which simulate software behavior) and mocks (special case objects that mimic real objects for testing) to facilitate test automation and increase code penetration; to employ service virtualization, which emulates the behavior of specific components in heterogeneous component-based applications, to improve testing automation; to do API testing; to implement test data management ("extracting data from the test, so I can expand it, grow it, manage it, share it, fix it -- absolutely essential for continuous"); to pay attention to environment management issues; and to look to self-service test environments.  

"We're all talking about breaking down the barriers between Dev and Ops and the different things we can do that can help us to become continuous," he said. "We're no longer releasing once or twice a year. At some organizations, the gap between feature request and release can be as little as 15 minutes. That's amazing, and it's happening now."