Diving into DevOps

New DevOps Maturity Survey: You Might Not Be Where You Think You Are

One finding: Companies implementing DevOps are often "sharing tools, but not knowledge."

• By John K. Waters
• April 18, 2017

If you had to rate your organization based on the maturity of its DevOps processes -- say, Beginner, Intermediate or Expert -- chances are you'd get it wrong. And you really need to get it right.

That's one of the conclusions reached by the authors of the newly published "2017 xMatters Atlassian DevOps Maturity Report." The report analyzes the results of a survey in which more than 2,000 people at organizations ranging in size from 500 to more than 10,000 employees, gave themselves a maturity ranking within a range of DevOps disciplines.

"And then we asked them specific questions about how they did things," explained Abbas Haider Ali, CTO of xMatters, "everything from testing to dealing with cultural issues and managing service disruptions. Through those questions, we came up with a more accurate picture of their actual status."

Nearly twice as many survey respondents received a Base or Expert rating after completing the second round of questions. And fewer respondents ranked as Beginner, Intermediate, or Advanced than in their pre-survey evaluations. "This tells us that many businesses don't really know where they are in their DevOps journey," the report concluded.

Among the more useful findings of the survey: Companies implementing DevOps are often "sharing tools, but not knowledge." The organizations surveyed are establishing strong DevOps cultures by removing boundaries and sharing tools; 80 percent reported that development and operations share at least some tools, and they do tend to share information generated from those tools. But only 17 percent of companies reported having open information available to cross-functional teams in dynamic formats, such as wikis and chat rooms. Most respondents indicated that information is shared only when requested, and only in static formats.

"Operations and development teams are leveraging the same tools," said Aileen Horgan, who leads Atlassian's DevOps Growth program. "They've got the tool sharing down, but when it comes to sharing knowledge between the teams, that's an area where their still major room for improvement."

"Tools and processes can support and drive collaboration," Horgan added, "and we strongly believe that our tools at Atlassian do that, but it really is about nailing culture first, especially if you're just starting out on your DevOps journey."

The survey also uncovered a lack of consistency around major incident responses in many organizations implementing DevOps practices. "Incident prevention and rapid response is easily the cornerstone of a mature DevOps operation," the report states. "But when the servers are on fire, many companies are stuck waiting on dependencies and getting bogged down in the process."

"In about half the cases, enterprises are actually waiting for other people to declare major issues, Ali said, "even though we know that customers are having disruptions. Often, they have all sorts of manual processes in place, which means they lack an effective way of keeping internal stakeholders up to date. This wouldn't be a problem if they had the cultural thing down as well as they think they do. You wouldn't be waiting around for the application experts to help you out, if there was a cultural element that said, hey, let's get on this thing and fix it."

Atlassian, a software development and collaboration toolmaker, and xMatters, a provider of an intelligent communications platform integrated with several Atlassian products, co-sponsored the survey and published the results in this accessible and insightful report. The two companies developed a framework for assessing the maturity of an organization's DevOps processes from observations of the efforts of their own customers. The framework provides five categories for investigation: Culture and Alignment, Operations and Support, Continuous Delivery, Design and Architecture, and Test and Verification. It also provides five maturity rankings: Base, Beginner, Intermediate, Advanced and Expert.

"These were the categories that stuck out consistently among companies investing in different areas to improve their DevOps processes," Ali said. "It gave us a model we could use to measure our survey results."

The report concludes with a discussion of "the DevOps gap" between "a ready culture and lagging functions" and "companies doing DevOps, and companies who still consider it a buzzword." Approximately 60 percent of survey respondents either didn't know what DevOps is, or weren't sure if their companies were doing it. (The report states that these companies were disqualified from the rest of the survey.)

The report ends on a positive note: "The good news is: For the 41 percent embracing and practicing DevOps, they have a strong competitive advantage over their peers in the market. And to keep that edge over the market, companies must continue evolving their DevOps practice, focusing on the divide between culture advancements and the functions required to run a full-fledged DevOps machine."

Atlassian currently provides an online "Playbook" for team building and management, and Horgan said her company plans to follow up this report with a playbook for DevOps. xMatters expects to launch an online self-assessment tool next month, Ali said.

"This was a first step," Horgan said, "identifying where you truly are in your DevOps journey. We plan to follow up with support for the next steps."

"Think of the survey as the first chapter in what we expect will be a long book," Ali said.