AppTrends

Sign up for our newsletter.

I agree to this site's Privacy Policy.

2014 Developer Challenges and Opportunities, Part III: Extreme Automation, Service Virtualization, Don't Be Afraid of SOAP, More

See Also:

There's a lot on the horizon for developers in 2014, and I just couldn't let the "predictions" thing go without passing on the observations of two more top industry watchers.

Theresa Lanowitz, founder of industry analyst firm Voke, for example, points out that catastrophic software failures in 2013 could make 2014 the year enterprises begin dialing back the pressure on app dev teams to get to market at warp speed. Why? Because businesses are being held accountable for these software failures. The shutdowns and glitches that plagued U.S. financial exchanges last year resulted in a credit rating downgrade (Goldman Sacks), a major contract termination (CGI Federal for healthcare.gov), and IT people being put on administrative leave.

"I believe that's the first time we've seen repercussions for the technology side," Lanowitz said. "The constant relentless push to get things to market faster is coming back to haunt us. Software is now such an integral part of the enterprise that failures have a serious impact. Businesses can't afford to absorb these big failures. Customers will give you only so many passes. And when Standard & Poor's says they are going to downgrade your credit rating and that you have to have a liquid capital reserves on hand to pay out for damages your faulty software causes…  Well, the C suite is suddenly interested in software."

Time-to-market still matters, Lanowitz said, but more than ever, so does software quality. Those conflicting demands are likely to lead to enterprise developers to embrace what she calls "extreme automation."

"Developers are going to have to say, all right, what can I do to make sure that we are delivering on time, but also with a high degree of quality, while understanding my cost and knowing what's going to happen when we have a defect in production when there's a catastrophic event," she said. "I think we're going to see developers answering that question more often in the coming year with extreme automation across the entire application lifecycle."

Essential for this level of automation, she said, is service virtualization, which she believes, though currently underutilized, could become the hub of the modern application lifecycle.

"The app lifecycle needs to be built using things like service virtualization, virtual lab management, and dev/test cloud," she said. "You want to give developers and testers environments as close to production as possible, so that the testers can test and immediately give developers the defects to remediate. This approach also reduces the provisioning time; developers and testers don't have to wait for lab environments. And you want to be able spin up a platform very quickly, and then replicate that platform throughout your software supply chain."

According to Voke's research, organizations that employ service virtualization in this way see significant benefits, Lanowitz said, including fewer defects going into production, shorter wait times, and an increase in the availability of services for end-to-end integrated testing.

"I think the term 'virtualization' has become too closely associated with the data center, and people don't think of it in this context," Lanowitz said. "But this is a technology that spans the entire application lifecycle. Now it's just a matter of education."

Forrester Research analyst Randy Heffner says 2014 should be the year developers begin to focus seriously on what he calls "digital business design," which is an integration strategy that sees the trends that will likely dominate the coming year -- big data and predictive analytics, mobility, and API management -- as pieces of a larger picture.

"There are no stand-alone applications anymore," Heffner said. "So don't design your application as though it is one. Your app will be integrated across a multi-organization ecosystem of business activity. So design the business activity first. Design the transaction, and then figure out how to embody it within the delivered solution. This business-first approach is going to be the key to bringing these trends together in a coherent way that enables sustainable business flexibility going forward."

Heffner also advised developers not to let their enthusiasm for REST architectural style cause them to miss the continuing value of SOAP.

"There are a lot of what I call quarter truths out there about SOAP," he said. "Of course, REST is very important, but don't fear SOAP. Understand when and how to use them. On the open Web, you'd better be using REST, because that's what developers demand. For BtoB situations, companies are often willing to invest a bit more, and often they're using toolkits where SOAP is easier to use; they're not scripting-language-based, but Java- or C++-based. The truth is, there's a lot of expansion still happening with SOAP."

"Just be wary of those who get very religious about REST -- you've got to have these particular models and you have to use verbs in this way and that way, etc.," Heffner added. "What will rule the day is pragmatic REST, designing to fill the right kind of model based on what you need to do."

Posted on 02/05/2014 at 11:06 AM0 comments


The Eclipse Foundation Turns 10: An Interview with Mike Milinkovich

It was during a break in the action long ago at the 2004 JavaOne conference that I found myself sitting in the W Hotel restaurant in San Francisco across from an earnest Canadian dude whose name I mispronounced twice during the interview, as he explained how IBM would really (no, really) keep its Big, Blue mitts off its relatively recently open-sourced, Java-based tooling platform, code-named Eclipse.

"They'll be involved, of course," said Mike Milinkovich, the first (and to date, only) Executive Director of the then-spanking-new Eclipse Foundation, "but as a member of the community only, without undue influence. With a little time, we'll prove that. We want people to have confidence in this technology."

I was skeptical (so was just about everybody), but IBM kept its distance, and the budding Eclipse community bloomed and spread far and wide over the next decade, virtually terraforming the IDE landscape in the process. Milinkovich and the Foundation cultivated that growth with vendor-neutral governance practices and what proved to be one of the most effective models for community open-source development.

Today, the Eclipse Foundation marks its 10th anniversary, and Milinkovich took the time to put up with my questions yet again. I reminded him of our first meeting back when he'd been on the job for about a month.

"Looking back, I'd have to say that credit for a big part of the community's early success should go to IBM, which did an exemplary job of setting Eclipse free," he said. "We became the first open source organization to show that real competitors could collaborate successfully within the community. When BEA and Borland joined in 2005 as strategic members, they effectively validated our claim of vendor neutrality. Both were important, but BEA in particular was a fierce IBM competitor at that time. Having BEA demonstrate that it was comfortable enough with our governance model to participate was a major milestone."

The impact of the Eclipse Foundation over the next decade is hard to overstate. When IBM released Eclipse to open source in November 2001, it was essentially a version of its WebSphere Studio Workbench, which the company then characterized as "a kind of universal tool platform -- an open, extensible IDE for anything, but nothing in particular." That release was a strategic move, Milinkovich said, to ensure that the fragmented Java tools market would have sufficient market share to compete with Microsoft's Visual Studio franchise.

"Given that there are now effectively two major tooling ecosystems -- Eclipse and Visual Studio -- I think it's fair to say, mission accomplished," Milinkovich said.

But perhaps more importantly, under the auspices of the Foundation, that "open, extensible IDE for anything, but nothing in particular" became the jumping off point for a remarkably diverse community responsible for, as the organization puts it, "a wide range of technologies, including rich client platforms; modeling; Web-based development tools; Java server runtimes; and frameworks, protocols and tools for the Internet of Things."

That most people still think of Eclipse as primarily Java tech continues to vex Milinkovich.

"If there's one thing I wish people would recognize, it's that with projects like Vertex and Jetty and Orion and EclipseLink and Equinox and on and on, there is an enormous variety of technology coming from the Eclipse community," he said.

Since the Foundation was established, it has grown from 50 members engaged with 19 projects to 205 members engaged with to 247 projects, including the totally-not-Java CDT Project, which provides a fully functional C and C++ IDE that has become the de facto standard development environment of embedded systems.

The foundation has also been responsible for a set of best practices for open-source IP management that has led to "pervasive use of Eclipse open source technology in commercial products," the organization pointed out in a statement.

But in my view, the Foundation's greatest accomplishment so far is its annual Release Train. This simultaneous, synchronized launch of Eclipse projects was one of a kind in the open source world when it was announced nine years ago. The first Eclipse Release Train, dubbed "Callisto," comprised 10 projects; the "Kepler" release last June included 71 projects comprising 58 million lines of code written by 420 committers in 54 supporting organizations.

"I agree that this is a big one," Milinkovich said. "The Release Train allowed us to show that it's possible to have a vendor neutral, open-source organization predictably ship commercial-ready software that companies can adopt in their products. And we did it in a way that provides for an explicit role for corporate members, while also making sure that the community is still represented in the governance. It's been a huge part of our success."

Much of the credit for the Foundation's efficacy should go to its highly professional staff, Milinkovich insisted. He also insisted on naming everybody during our interview, but I'm just going to do it with a link to the staff page.

Milinkovich has been in the catbird seat for an awful lot of changes for developers, and I asked him what stood out for him as the most significant in the past decade.

"One of the things that has completely changed over the past ten years is the way technology is being procured," he said. "If you don't have an open-source offering that provides for frictionless adoption by developers -- where they don't have to involve procurement to try your technology -- you basically can't sell. There are still a few exceptions to that, but when was the last time you heard somebody say that they were going to do a six-month proof of concept, hire two consultants, and get a vendor to come in and show them how the technology works? Ten years ago, that's how it was done. There has been a fundamental shift in that process that I think has put developers in the driver seat in selecting the technologies for the applications being deployed."

The Foundation issued a press release today that included a comment from Stephen O'Grady, Principal Analyst at Red Monk, that summarizes the importance of Eclipse Foundation pretty well, and bears including here: "Having begun its life as a bold experiment in corporate open source contribution, the Eclipse Foundation has, over the past decade evolved from a focused Java development project to a diverse community supporting a wide array of languages and needs. From browser based development to the Internet of Things, Eclipse is always willing to rethink its role and adapt itself to the fast changing industry around it."

Many happy returns of the day.

Posted by John K. Waters on 02/03/2014 at 9:42 AM0 comments


2014 Developer Opportunities and Challenges, Part II: UX Skills Gap, Crowdsourcing

See Also:

The coming year is fraught with challenges for enterprise developers, but it's also full of opportunities, say top industry analysts -- if you keep your eyes on a few key trends.

Eric Knipp, who manages Gartner's burgeoning Application Platform Strategies research team, puts the growing importance of the user experience (UX) at the top of his list of developer challenges for 2014.

"With so many device types out there, with the Web evolving, with the Internet of Things coming along, fast, we're getting constant questions from our clients about how to reduce the interactive friction experienced by both internal and external users of their software," Knipp told ADTmag. "And the fact is, inside most large enterprises most of the developers just don't have the chops to do that well."

This gap in UX skills in the enterprise is a fundamental problem, Knipp said, because the developers building the applications and designing the interaction patterns are typically not familiar enough with how the users of those apps interact with that software to do their jobs.

It's an old problem, he said, but a new opportunity.

"UX is one of the big things to get your arms around in 2014," he said. "It presents a great opportunity to outpace your competitors if you do, especially if you recognize that it isn't just important for consumers using your mobile app, it's also important to the productivity and satisfaction of your internal employees."

Another opportunity Knipp sees ahead for developers comes from what might for many be an unexpected place: crowdsourcing and hackathons.

"The enterprise is finally taking an interest in crowdsourcing AD," he said. "The question of how enterprises source apps -- do we build them ourselves, do we hire a third party outsourcer, do we buy them off the shelf -- now has a third answer. Most organizations don't get it yet, but they will."

Crowdsourcing also gives developers a chance to sharpen their skill sets, he said.

"Say, I'm doing boring old Java EE at my day job, which pays the rent very well, but I want to do Node.js," he said, "or native iOS development or Ruby on Rails with deployment into Heroku. The bank I work at doesn't provide opportunities for me to do that. You can sharpen your skills in contests in crowdsourcing forums where the applications that you're building are increasingly enterprise class. This is a great opportunity for individual developers. If you want to sharpen your tool box, this is a fantastic place to do it."

That crowdsourcing trend points, if indirectly, to another trend: the social organization of developers. Jeffrey S. Hammond, vice president and principal analyst at Forrester Research specializing in application development and delivery, calls it The Emergence of Social Development.

"The social consciousness of developers is changing their orbits, so to speak," Hammond said. "It used to be that developers clustered around big ISVs, such as Microsoft, IBM and Oracle. Those vendors were the planets around which the developers revolved. I believe those 'planets' are increasingly the open source communities, like Eclipse and Apache and Hadoop. GitHub is part of it, too. As the gravity shifts to communities from vendors -- and open source communities in particular -- they become much more important. And much more influential."

The implication here is that the developers themselves could well be calling more of the shots somewhere down the road.

"In a way, I see the software dev space going the way the music industry and the movie industry have gone, where the race for talent gives talent a much bigger say," he said.

Looking ahead to 2014, Hammond said it's simply a great time to be a developer.

"This is a big change from six or seven years ago, when people said all the development was going to China. I've seen that prediction completely turned on its head. That may be the case for maintaining existing systems, but for new systems, if you know this stuff -- if you know Node.js and you know how to use Bootstrap -- you can name your price these days."

Hammond is developing a new report on the shifting orbits of developers, and Knipp is working on a paper about developer crowdsourcing. I'll let you know when they're published.

Posted by John K. Waters on 01/28/2014 at 10:00 AM0 comments


2014 Developer Opportunities and Challenges, Part I: Embedded, APIs, Mobile Systems and More

See Also:

Now that the confetti has settled, I thought it would be a good time to talk with industry mavens about what lies ahead in the coming year for developers, both the challenges and the opportunities.

Not surprisingly, many of the industry watchers I spoke to agreed that machine-to-machine learning (M2M) and the Internet of Things (IoT) offered enormous opportunities for developers to get into the embedded space. "Having the Java people get involved will make it easier for those not familiar with this space," said Michael Azoff, principal analyst at Ovum, "but [coding for] real-time systems is a skill and requires some domain expertise. It's not a pure software space, but demand will be huge for the skills."

Azoff also observed that 2014 will continue to see niche languages like Erlang, Scala, and Lisp finding a place in more developers' tool boxes, as Java continues its evolution with lambda expressions, which are coming in Java 8. The ability to add functional programming features to Java "could be the next big programming paradigm shift," he said. "It's the one to watch."

IDC analyst Al Hilwa sees 2014 as "the age of the API redesign." "We are entering the golden age of APIs," he said in an email. "API design becomes a mainstream developer discipline. Organizations re-design APIs to new realities of mobile networks and devices. API marketplaces proliferate. API management will be essential." He also said that, despite its growing popularity and maturity, HTML5 will not replace native mobile app development through 2017. Instead, HTML5 will co-exist with native development, he said. APIs that enable Web pages to use the WebSocket protocol will see increased use in mobile development, for example.

2014 is rife with buzzwords, but a few -- mobile, cloud, big data, and social -- comprise what Gartner Group calls a "nexus of forces." Each has a separate impact, explained Gartner analyst Mark Driver, but together they're causing profound changes. The challenge for developers lies in creating a synergistic relationship among these forces, he said.

"Building a mobile system for example isn't just about building a mobile system," he said. "It's about all the things you have to do to feed that system and integrate with it. You start asking questions like, How will a mobile system cause me to have to go back and undo all of my existing Web architectures that I've had in place for ten years? How does big data affect how I do these things? What new frameworks and best practices to I have to bring in to build truly cloud-native applications? It really is a challenging time for developers."

Mobile is somewhat central to that challenge, Driver said. "Mobile is now assumed to be an element of virtually any new project in the enterprise," he said, "which is about much more than simply shrinking a screen down to fit on a smart phone. It's not only a separate set of tools and languages, it's a different way of building applications, like the reactive programming model, for example, which involves architectures for systems that deliver highly responsive user experiences with a real-time feel -- architectures that are relatively new to a lot of enterprise developers."

Mobility is key component of in a list of strategies that have created a new "business reality" for developers, said Dana Gardner, principal analyst at Interarbor Solutions. But 2014 could offer new leadership opportunities for developers as businesses acclimate to new mobile-first, cloud-first, and data-centric strategies. Developers, he said, are in a unique position to help move companies forward in ways that others in IT are struggling with.

"We're now adjusting to this new business reality," he said, "and software has never been more critical. Developers can advocate -- from their vantage -- on how best to define strategy and how to attain it. They could, for example, align with line of business managers on key business objectives and requirements, and then sell that together to the operations and IT leadership. They can, in effect, lead, thanks to disruption; they can put the horse firmly in front of the cart, where it belongs."

"So if developers or app dev groups can advocate now for what they believe is right to get to mobile-first using cloud-best to then produce and deliver the data and analysis where it does the most good," he added, "they will immensely help their company, while improving their credibility, standing and worth. They might even make it a better place to work at and thrive."

Posted by John K. Waters on 01/21/2014 at 11:53 AM0 comments


UPDATED: Our First Ever App Dev Trends Conference Set for December 2014

Update 3/4/13: We've rescheduled App Dev Trends 2014 to avoid conflicts. The conference will now take place December 8-11, 2014, at the Mandalay Bay Resort and Casino in Las Vegas. The new date allows us to extend the Call for Papers deadline to April 11. We've had a great response so far, and we're glad to be able to provide more time for speaker proposals, so please keep sending them in! -John

Maybe you've heard the rumors about a new technology conference focused on the makers and maintainers of the purpose-designed software that drives organizations in virtually every industry in the world -- in other words, the readers of Application Development Trends. I can now confirm those rumors.

Yup. We're finally putting on a show of our own.

App Dev Trends 2014, which is set to run Sept. 29-Oct. 3 at the Mirage Resort and Casino in Las Vegas, will throw a spotlight on the unique challenges faced by enterprise software professionals. This event is about cutting-edge intelligence on a wide range of trends, tools, and best practices that our readers gotta have to keep up with the ever-evolving demands of their organizations. It's about knowing what's next and preparing for it. It's about acquiring new skills and adapting existing skill sets. It's about boosting organizational efficiency, productivity and competitiveness. And it's about rubbing elbows with peers and pros facing the same challenges.

Also, there will be food. And drinks. And snacks.

This is our first ADTmag-branded conference, but the event is being organized by the experienced pros who put on the Visual Studio Live! and Live! 360 conferences. They're also the publishers of this site, our newsletters, as well as our sister pubs, Redmond magazine, Visual Studio Magazine, and more. So we're in good hands.

Today, we're issuing the official Call for Presentations (seems like there should be horns or something) for the following conference tracks:

  • Agile in the Enterprise: Best-Practices in the Real World
  • Enterprise Cloud Development: The Distributed Computing Challenge
  • Big Data: The Enterprise Taps a Gusher
  • Mobile: The Enterprise Game Changer
  • Beyond DevOps: A Whole-Enterprise Effort
  • M2M and the Internet-of-Things: The Enterprise Embedded
  • The Social Enterprise: New Workplace Connections

The Web site for submitting presentation proposals is up and running now, and the deadline for submissions is Feb. 21, 2014! That's just around the corner, so please submit your proposals soon. We're expecting a lot them, and I'm looking forward to reading every one.

Did I mention that I'm the conference chair? I'm pretty sure that means I have to bring the snacks.

Posted by John K. Waters on 01/08/2014 at 1:48 PM0 comments


IDC Study Counts the World's Developers: 11 Million Pros

Here's a question that vexes analysts and industry watchers: Exactly how many developers are there in the world? Apparently, codederos are a hard bunch to count. Leave it to the indefatigable Al Hilwa to get the job done -- well, Al and his fellow International Data Corporation (IDC) analysts.

IDC recently published its "2014 Worldwide Software Developer and ICT-Skilled Worker Estimates," and I got a peek at the report, which Hilwa authored. It's a country-by-country build-out of population estimates based on the analysis of granular occupation surveys and census data (where available), education enrollment and graduation data (where available), and other materials and correlations where those data were not available. It provides numbers for 90 countries and three world regions, including those you'd expect (U.S., China, India, Japan, Brazil, the U.K., Russia, Canada, etc.) and a few you might not (Nigeria, Qatar, Luxemborg, Jamaica, etc.) Together, these countries account for 97 percentof the world's GDP.

Sexy, right? Okay, maybe not, but this is information software and IT service providers could really use when it comes to things like resource allocation and investment decisions, which Hilwa "calls gating factors to addressable market calculations." (Come on, that's kinda sexy.)

One of the things I love about this 32-page report is that it counts professional coders and hobbyists. IDC defines "hobbyists" as coders building software in their spare time for their "personal entertainment," student developers, contributors to free and open-source software projects, and unfunded entrepreneurs. That group also includes part-timers putting in less than 10 hours a month and full-time knowledge workers called upon from time to time to write code for things like productivity apps.

IDC believes that there are approximately 18.5 million software developers in the world right now. Around 11 million of those are pros and 7.5 million are hobbyists. IDC also believes that there are 29 million ICT-skilled workers in the world, a number that includes pro devs and 18 million operations and management skilled workers.

Hilwa, who serves as program director in IDC's Application Development Software group (and who called this project "a labor of love"), says that he expects the overall number of developers and ICT-skilled workers to grow over the next couple of years, but thanks to the cloud and mobile trends, most of that growth will be within the developer population. He added that the "mobile revolution" is expected to boost the growth of the hobbyist developer population.

So, where are all these developers living? According to the report, 19 percent of the worldwide population of software developers (both pros and those coding for funzies) live in the U.S.; China has 10 percent ; and India has 9.8 percent . India has more pros than China, which has more hobbyists. Looking at the bigger picture, 36 percent of all software developers live in the Asia/Pacific region; 39 percent live in Europe, the Middle East, and Africa; and 30 percent live in the Americas.

The full IDC report is available now to IDC clients.

Posted by John K. Waters on 01/06/2014 at 3:35 PM0 comments


Possible Game Changer: IBM's Open Source Watson Cloud Platform

When IBM announced its decision last month to turn its Watson cognitive computing technology into an open software development platform, complete with APIs and (Big Blue hopes) a partner ecosystem, the news didn't exactly set the world on fire, but maybe it should have.

News of Watson's victory in 2011 over two human contestants on the Jeopardy game show did spark a mainstream media blaze, albeit a brief one, rife with facile quips about IBM's "Frankenstein of trivia," and repeats of übercontestant Ken Jennings' comment: "I, for one, welcome our computer overlords."

But that was Watson the Machine, a system specifically designed to compete on the game show, comprised 90 IBM Power 750 servers, each running 8 POWER7 cores (3.5 GHz) with 4 threads per core. The OS was SUSE Linux Enterprise Server 11; the software was written in Java, C++, and Prolog; and it used Apache's Hadoop and UIMA and IBM's DeepQA software. The resulting system was able to interpret queries in natural language and used statistical analysis, advanced analytics, and all that processing power to search millions of pages in seconds.

Now comes Watson the Platform -- or more precisely, the Watson Developer Cloud -- a cloud environment for the development of cognitive computing applications that use the big data and analytics capabilities that killed on Jeopardy. Developers will be able to embed a Watson capability into either an existing application or a new app. They'll access the cognitive computing mojo via the Watson Experience Manager, a portal app that provides access to a development sandbox. And each app will be defined on the Watson Developer Cloud by partners adding their own content from the Watson Content Store. The result will be apps labeled "Powered by IBM Watson."

For developers, the IBM Platform/Developer Cloud holds the promise of an entry ramp into the world of cognitive computing, which Big Blue reminds us often is likely to be new territory for most software makers. As IBM puts it, Watson brings to developers its ability "to help facilitate a dialogue, put content in context, maintain continuity of discussions, cull through millions of pages of data, return insights, identify patterns difficult to detect, and learn throughout the process." The ability to add these capabilities to applications "can truly be game-changing," the company says. And it's probably right.

IBM concedes that it will take an ecosystem for this ambition undertaking to succeed, and it's already forming strategic partnerships through the Watson Ecosystem Program. Among the first to sign up was Elance, provider of an online work marketplace. IBM has tapped Elance as its "inaugural talent partner," the company said in a statement, to build a Watson Talent Platform.

"Actually, we're the only talent partner," Elance CMO Rich Pearson told ADTmag.

Elance, which is headquartered in Mountain View, Calif., with offices in Oslo, Norway, matches businesses with freelancers online via a public "talent cloud" comprising about 3 million freelances in 170 countries. The system filled 1.2 million jobs this year, Pearson said.

Working with IBM, Elance developed a private talent cloud for the new Watson Ecosystem. The company's infrastructure, which tests skills and collects job performance data, will help to populate the ecosystem with top performers, Pearson said. The initial talent pool for Watson-enabled apps will likely include data scientists, mobile developers, and designers with experience in data visualization, he said. The company has partnered with Skilled Up to provide online courses for developers who want to ramp up their skillsets. Elance is also planning to apply Watson technology to its own business, Pearson said.

"We're in the business of matching talent and businesses," he said, "and we use our own algorithm for that process. The idea of using IBM Watson technology to improve that process is fascinating from our perspective. The Watson secret sauce is its ability to help with decision making, and we're looking at it as a way of providing richer data to help us make smarter and faster decisions."

IBM's nascent plan is still in the making, though the company has promised to start delivering on all of this in 2014. In the meantime, Watson and Watson-like technologies have already spread beyond that flashy game show debut, perhaps most notably to Memorial Sloan-Kettering Cancer Center. The Center and IBM are working together to combine Watson's supercomputing powers with Sloan-Kettering's clinical know-how—along with "existing molecular and genomic data and vast repository of cancer case histories"—to create a diagnostic and treatment system based on updated research.

Keep an eye on the IBM Watson Web site (and this blog) for further developments.

Posted by John K. Waters on 12/17/2013 at 12:09 PM0 comments


BSIMM-V: Free Software Security Insights from 67 Companies

Here's a provocative statistic: Within a group of leading companies that includes Microsoft, PayPal, Salesforce, Nokia, Sony Mobile, and Visa, the average ratio of full-time software security specialists to developers is 1.4/100. That's one of the findings in the recently published fifth edition of the software-security "measuring stick" known as the BSIMM (Building Security In Maturity Model).

A "maturity model" describes the capability of an organization's processes in a range of areas, from software engineering to personnel management. The Capability Maturity Model (CMM) is a well-known example from software engineering. The BSIMM (pronounced "bee-simm") is the first maturity model for security initiatives created entirely from real-world data.

The BSIMM was developed as a tool to help organizations evaluate their software security programs by comparing them to the programs of other companies. It's based on data collected by its authors through interviews and direct observations of the most successful large-scale software security programs. Although those programs use different methodologies and terminologies, they're described in a uniform way in the BSIMM via a framework, called the Software Security Framework, which provides a common vocabulary and allows for apples-to-apples comparisons.

So, is that ratio of software security pros to developers the right one? That's not a question the BSIMM was designed to answer, says one of its authors.

"The BSIMM is based on the study of real practices as they exist," explained Gary McGraw, CTO of security consulting firm Cigital and author of eight books on software security. "It describes those practices; it's not a prescriptive model. But it's real data, not hunches and guesses, so I can go to the board and say, here's you, and here are the other 26 firms that look like you that we've measured before. And I can say, it looks like you're the slowest zebra. And then we have a conversation about that."

BSIMM-V includes data from 67 participating companies, up from the 51 included in the fourth edition. The number of companies has grown every year since the first edition was published in 2008; that one was based on studies of nine software security initiatives. BSIMM-V describes the work of about 3,000 people, collectively, McGraw said.

As a measuring stick, the BSIMM allows an organization to compare and contrast its own software security efforts with those of its peers. As the report puts it, "You can then identify goals and objectives of your own and look to the BSIMM to determine which additional activities make sense for you."

BSIMM's authors argue that highly mature initiatives are well rounded, carrying out all of the 12 core practices described by the model, including: strategy and metrics, compliance and policy, architecture analysis, code review, security testing, penetration testing, and configuration management. The model also describes how mature software security initiatives evolve, change, and improve over time.

During the course of their investigation, the researchers have observed a total of 112 activities related to software security. These are actions carried out or facilitated by the software security group within an organization as part of a practice, and each activity is directly associated with an objective. The researchers added two new activities in the last edition of the BSIMM based on their observations in the field: simulate software crisis and automate malicious code detection. BSIMM-V adds another new activity: operate a bug bounty program.

Keep in mind that what the BSIMM is describing is security activities around software development, specifically. The computer security industry as a whole is growing fast, McGraw noted, at a rate of about 8.9% per year, generating between $20 and $40 billion in revenue annually. And while software security accounts for only 10% of that growth, he said, that segment is growing more than twice as fast: 20% per year, by some estimates. "I like to think of us finally as the pinky fingers on the two hands of computer security," McGraw said.

The BSIMM was originally developed by Cigital and Fortify Software (since acquired by HP). The two most recent editions of the study were authored by McGraw; Sammy Migues, Director of Knowledge Management and Training at Cigital, and Jacob West, CTO of Fortify Products in HP's  Enterprise Security group. The first three BSIMMs were authored by McGraw, Migues, and Brian Chess, distinguished technologist at HP (and co-founder and former chief scientist at Fortify).

BSIMM-V is available for download. It is distributed free under the Creative Commons license.

Posted on 12/12/2013 at 10:54 AM0 comments


Cascading: Open Source Java App Framework for Big Data

Enterprise interest in Big Data and associated analytics software has sparked intense interest in Apache Hadoop, the open source framework for running applications on large data clusters built on commodity hardware, and something of a flood of tools for developers working with it. But as an applications market emerges in this space, the next Big Thing for Big Data is likely to be app-oriented middleware.

That's an insight Tony Baer, principal analyst at Ovum, shared with me when I talked with him recently about Continuuity's recent Reactor 2.0 release, which the Java toolmaker billed as the first scale-out application server for Apache Hadoop.

"It is inevitable that applications will be developed that run against Big Data," Baer said, "and as that occurs, it will be necessary to have an application layer that allows developers with Java and other languages to develop apps that run against it."

Baer's prediction makes perfect sense, and it's one reason Java jocks might want to keep an eye on Concurrent, the company behind the open source Cascading project. Cascading is a Java application development framework for rich data analytics and data management apps running across "a variety of computing environments," with an emphasis on Hadoop and API compatible distributions.

"Big Data is moving to the next phase of maturity and it's all about the applications," the company says on its Web site. "The applications process the data and extract the value at scale and we believe that there must be a simple, reliable and consistent way to build, deploy, run and manage these data driven applications."

Great minds.

Concurrent characterizes Cascading as "a rich Java API for defining complex data flows and creating sophisticated data oriented frameworks," and it claims more than 110,000 user downloads a month. Its published user list includes Twitter, eBay, Square and Etsy, among others.

The San Francisco-based company recently announced Cascading 2.5 with new support for Hadoop 2 and YARN, the next-gen Hadoop data processing framework (sometimes called MapReduce 2.0).

Chris Wensel, Concurrent's founder and CTO, has argued that developing and building applications on Hadoop has proven to be difficult, despite the framework's rapid enterprise adoption. "With Hadoop 2, the community has addressed many concerns, paving a clearer path for enterprise users," he said in a statement. "At Concurrent, we're dedicated to forging a simpler path to mass Hadoop adoption by delivering a framework for building powerful and reliable data-oriented applications supporting data driven business models -- quickly and easily. Our support for Hadoop 2 was an easy decision, as we continue to be an integral part of the Hadoop and Big Data ecosystem, providing solutions that simplify application development and management for the enterprise."

As a Java-based framework, Cascading fits naturally into JVM-based languages, including Scala, Clojure, JRuby, Jython and Groovy. And the Cascading community has created scripting and query languages for many of these languages. The company's extensions page offers a growing list of user contributed code.

Cascading 2.5 is publicly available and freely licensable under the Apache 2.0 License Agreement.

Posted by John K. Waters on 12/04/2013 at 2:45 PM0 comments