VMware's recent announcement of an integration of its Spring Framework with Apache Hadoop is aimed at making life easier for enterprise Java developers who want to use the popular open-source platform for data-intensive distributed computing. The new Spring Hadoop is a lightweight framework that combines the capabilities of the Spring framework with Hadoop's ability to allow developers to build applications that scale from one server to thousands and deliver high availability through the software, rather than hardware.
By integrating the Hadoop Framework, a Java-based, open-source platform for the distributed processing of large data sets across clusters of computers using a simple programming model, with the Spring Java/J2EE application development framework, VMware has created a project that fits neatly under the Spring Data umbrella. The open-source Spring Data project comprises a group of sub-projects seeking to make it easier to develop apps that use a bunch of new data access technologies, such as non-relational databases, cloud-based data services and MapReduce frameworks like Hadoop.
In addition to Apache Hadoop, the list of Spring Data sub-projects includes, among others, the Spring Data JPA, which simplifies the development of Java Persistence API-based data access layers; VMware's GemFire distributed DB management platform; the Redis advanced key-store; and the MongoDB document-oriented database.
The new framework also supports comprehensive HDFS data access through such Java Virtual Machine (JVM) scripting languages as Groovy, JRuby, Jython and Rhino. HDFS (Hadoop Distributed File System) is designed to scale to petabytes of storage and to run on top of the file systems of the underlying OS.
The list of Spring Hadoop capabilities also includes: declarative configuration support for HBase; dedicated Spring Batch support for developing workflow solutions that incorporate HDFS operations and "all types of Hadoop jobs;" support for the use with Spring Integration "that provides easy access to a wide range of existing systems using an extensible event-driven pipes and filters architecture;" Hadoop configuration options and templating mechanism for client connections to Hadoop; and declarative and programmatic support for Hadoop Tools, including FsShell and DistCp.
Developer Costin Leau announced the integration on the SpringSource Community blog. "…Spring Hadoop stays true to the Spring philosophy offering a simplified programming model and addresses 'accidental complexity' caused by the infrastructure," he wrote. "Spring Hadoop, provides a powerful tool in the developer arsenal for dealing with big data volumes."
VMware has released Spring Hadoop under the open source Apache 2.0 license. It's available now as a free download.
Posted by John K. Waters on 03/13/2012 at 3:54 PM0 comments
The Multipurpose Internet Mail Extensions (MIME) specification that defines the way multimedia objects are labeled, compounded and encoded for transport over the Internet turns 20 this month. Ned Freed and Nathaniel Borenstein were the two primary authors of the spec. Borenstein, who worked at New Jersey-based Bellcore at the time, sent out the first real MIME message on March 11, 1992. That message included an audio clip of the Telephone Chords, an all-Bellcore barbershop quartet featuring John Lamb, David Braun, Michael Littman and Borenstein, singing about MIME to the tune of "Let Me Call You Sweetheart."
"Those of you not running MIME-compliant mail readers won't get a lot out of this," Borenstein wrote in that message.
Are there any non-MIME-compliant mail readers today?
Borenstein, who is today Chief Scientist for cloud-based email management company Mimecast, was in Silicon Valley recently to speak at the Cloud Connect conference. I grabbed a few minutes with him when he stopped in at the Computer History Museum in Mountain View. We got the two questions he's asked most often out of the way first.
"Everybody assumes I founded this company, but when I joined, it was six years old," he said. "When I first heard the name, before I ever thought about working there, I thought, they can't do that! But as I learned about the company, I found that I loved what they were doing and I liked the people a lot. And it certainly doesn't hurt them to have the author of MIME working at Mimecast."
Borenstein says most people also want to know if he ever thinks about how much money he would have made if he'd had some sort of financial stake in the now-ubiquitous Internet standard for multimedia data. "They ask me, 'Have you ever thought about what it would be like if you got a penny for every time MIME was used?' The answer is, yeah. It's hard to be precise, but I'd estimate that MIME is used about a trillion times a day. My current income would be roughly the GDP of Germany."
Borenstein joined Mimecast in 2010 after spending eight years at IBM as a distinguished engineer. His duties include long-term product planning, external writing and speaking, and patent strategy and submissions.
"When I joined the company, it had never filed a patent, because all the principals believe, as I do, that patents are deeply evil," he said. "Unfortunately, I had to point out that it's also true that deeply evil people can hurt you, and you really have a responsibility to protect yourself. Our patent strategy is primarily defensive."
Mimecast filed for a patent last year to support a cloud-enabled e-mail analytics system it is developing. The company's flagship service provides cloud-based e-mail management for Microsoft Exchange. That service includes e-mail archiving, continuity and security. It unifies "disparate and fragmented e-mail environments into one holistic solution that is always available from the cloud," the company says on its Web site.
"The cloud makes it possible for companies of a size that could never really contemplate it before to make practical and valuable use of big data and business analytics," Borenstein said. "They can take all that data and finally use it for something besides a dead repository."
In fact, Mimecast's new e-mail analytics system, which he called "proactive e-mail," takes on that very problem. If the demo he showed me is any indication, it could go a long way toward solving the so-called organizational memory problem.
"In any large organization, there's always someone who knows what you're trying to find out," he said, "and yet finding that information is almost always harder than rediscovering it. This is where I see the cloud going: supporting value-added apps that dig into those company archives and bring your own information back to you so that you can use it."
Borenstein is an energetic and positive guy, and he seems to like the work he's doing now at Mimecast very much. But he does miss the days when pure research labs like the one that spawned MIME weren't so uncommon.
"Labs like Bellcore, which was an institution of nearly pure research, are rare birds these days," he said. "And we all suffer for that rarity. After all, MIME grew out of a simple mandate to come up with something that would increase bandwidth usage."
"People would ask me," he added, "why are you working so hard on getting pictures into e-mail? And I'd say, someday I'm going to have grandchildren, and I want to be able to get pictures of them by e-mail. And they would laugh, because back in the 1980s that was too far-fetched."
Borenstein showed me the first photo sent to him in an email by his daughter of his twin granddaughters: an ultrasound image of a cluster of cells.
"The thing I had envisioned all those years ago was supposed to be much cuter," he said.
On March 5, ACS, the corporate successor to Bellcore, celebrated the twentieth anniversary of MIME at its New Jersey headquarters with, among other things, a reunion of the Telephone Chords. Borenstein said he was practicing "so I don't miss the notes this time." I couldn't make it to the event, but the original message featuring the Telephone Chord's singing their MIME song in four-part harmony is available on Borenstein's "MIME & Me" Web page.
Posted by John K. Waters on 03/09/2012 at 5:09 PM0 comments
The annual CloudConnect conference got under way this week in Santa Clara, Calif. There's a great speaker lineup, and lots of vendor news at this year's event. Among the more noteworthy vendor announcements was Nimbula's beta release of its Director 2.0 product. The company expects a general availability release in March.
Nimbula is a Menlo Park, Calif.-based provider of "cloud operating system technology" that was developed by the company's founders in Cape Town, South Africa. The company describes Nimbula Director as "a new class of cloud infrastructure and services system that uniquely combines the flexibility, scalability and operational efficiencies of the public cloud with the control, security and trust of today's most advanced data centers." Think of it as Amazon EC2-like services behind the firewall.
Nimbula Director is an extensible cloud platform designed to embrace augmentation by third parties. The custom logic of network, data, PaaS and other cloud services can be embedded in the cloud and run and managed as though Nimbula wrote it. Consequently, these services inherit Director's high availability, multi-tenancy, and network security functionality.
The latest release adds some nifty enhancements, including support for VMware's ESXi hypervisor. Version 2.0 will also support VMware's Cloud Foundry PaaS. Director 2.0 is among the first solutions to bring the Cloud OS model of an EC2-style cloud to VMware customers. (Nimbula is now a member of VMware's Technology Alliance Program.)
This version also extends its management "from the control plane up into the end user application space," the company says. In other words, users can now orchestrate the provisioning of applications and let the system monitor and manage them over their lifetimes.
And this release also "rounds out" Director's IaaS networking feature set (which already included DHCP, NAT, firewall and VLAN services) with DNS and VPN services. This gives Director a complete set of the networking services required for running real world apps.
Two other announcements caught my attention:
- Cloud-based hosting service AppFog has added Blitz.io and Iron.io to its recently introduced add-on program. Blitz provides application and Web site developers with "powerful yet simple capabilities including continuous monitoring, performance testing and remediation." Iron.io specializes in cloud queuing systems; IronMQ is its elastic message queue and IronWorker is its task queue.
Portland, Ore.-based AppFog was founded about a year ago by Web developer Lucas Carlson, co-author (with Leonard Richardson) of Ruby Cookbook: Recipes for Object Oriented Scripting. The company initially billed itself as a PHP-based Platform-as-a-Service (PaaS) provider, but quickly began supporting Ruby, Python, Perl, Node.js and Java. The company launched the add-on program in December, and just keeps adding vendors. The list also includes New Relic, MongoLab, MailGun and MongoHQ.
- The OpenStack experts at Mirantis have teamed up with cloud training organizer CloudCamp to deliver a series of training courses in OpenStack technology. The program aims to "expand the pool of skilled engineering talent and expertise that a growing number of enterprises require in order to create, deploy and maintain OpenStack solutions."
OpenStack is an open-source project made up of several interrelated projects focused on delivering various components for a cloud infrastructure solution. As the community Web site describes it, the project "aims to deliver solutions for all types of clouds by being simple to implement, massively scalable, and feature rich." More than 145 leading companies participate in the OpenStack project, including AMD, Cisco, Citrix, Dell, HP, Intel and Microsoft.
Posted by John K. Waters on 02/14/2012 at 12:25 PM1 comments
Back in October at JavaOne, representatives from the Java Community Process (JCP), the group that certifies Java specifications, talked about changes coming to the organization. First on the list was the "low-hanging fruit" of transparency, participation, agility and governance addressed in Java Specification Request (JSR) 348. Since then the community has been, in the words of JCP chair Patrick Curran, "revising the process through the process."
Now the JCP has gotten around to a moderately trickier adjustment: The promised merger of the two JCP Executive Committees: the SE/EE EC and the ME EC.
"JSR 355: JCP Executive Committee Merge" seeks to merge the two ECs and reduce the total number of committee members (there are 32 right now). The JSR includes a provision for maintaining the existing two-to-one ratio of ratified-to-elected seats, and a rule that neither Oracle nor any other member may hold more than one seat on the merged EC.
The ECs are charged with guiding "the evolution of Java," and it's not a small job. The ECs pick the JSRs that will be developed, approve draft specs and final specs, approve Technology Compatibility Kit (TCK) licenses, approve maintenance revisions and occasionally defer features to new JSRs, approve transfer of maintenance duties between members and provide guidance to the Program Management Office (PMO).
Curran is the spec lead on JSR 355, and the initial Expert Group membership consists of all members of the ME and the SE/EE Executive Committees -- including, among others, Mike Milinkovich, executive director of the Eclipse Foundation; Red Hat's Mark Little; brand new EC member Twitter's Attila Szegedi; Intel's Anil Kumar; and Google's Joshua Block.
The Expert Group has committed on the JCP Web site "to complete the JSR within about six months, thereby permitting the changes to be initiated during the 2012 elections," but also allows that the changes "may need be phased in over time."
The JCP also states its case for the consolidation:
Changes in the Java ME market, and the increasing maturity and consolidation of the Java market generally, suggests that some rebalancing between Java ME and the other platforms, together with a modest reduction in the total number of EC members, would be appropriate. Looking forward, the expected convergence between Java ME and Java SE is likely to render the current division into two separate ECs increasingly irrelevant. Since Java is One Platform, it ought to be overseen by a single Executive Committee.
Curran put it more succinctly during his aforementioned JavaOne session: "It seems like the right thing to do," he said, "that we should have a single executive committee which will deal with all of the three platforms -- because it is one platform with three flavors."
Let me know what you think about this merging of ECs in the comments!
Posted by John K. Waters on 02/07/2012 at 1:31 PM0 comments
If you were wondering whether Mac developers were also facing the pressure to become polyglot programmers so many industry watchers mentioned in their 2012 predictions (like this one and this one), consider the recent announcement from Zend Technologies. The creator and commercial maintainer of PHP announced the general availability of its Web app server, Zend Server 5.6, featuring new support for Macheads.
According the company's CEO and co-founder Andi Gutmans, Zend has been seeing an increased demand from "the community that develops on the Mac."
"We've seen this demand for Mac support from the Zend PHP developer community, at the annual ZendCon event, on Zend Forums and among user groups worldwide," Gutmans told me in an e-mail. "Now, Mac developers can leverage the agility of PHP for Web app development when they use Zend Server, with the best PHP runtime, powerful monitoring, diagnostics and performance optimization."
Okay, maybe it was his PR team. The point is, Zend does a consistent job of responding to trends surfaced by customers asking for new features. Not bleeding-edge fads, but real, late-in-the-hype-cycle trends. Last August, for example, the Cupertino, Calif.-based company released Zend Server 5.5, which essentially addressed the increasing unpredictability of traffic and workloads caused both by the growing popularity of the cloud and the proliferation of end user devices. Mobile and the cloud are not breaking news. Zend was responding directly to customer demand, Kent Mitchell, Zend's director of product management, told me at the time.
And now Zend is responding to Mac developers who, they say, want to have access to the full suite of Zend Server application development capabilities previously available only to Linux and Windows developers. Actually, that should be enterprise Mac developers. And my lead sentence should probably have read, "If you were wondering whether Mac is moving into the enterprise…"
Gutmans' PR team also sent an interesting list, entitled "Why Mac Matters to the Development World," which included:
- There are now nearly 60 million Mac users worldwide.
- Mac runs neck and neck with Linux based on Zend's 3-year history of Zend Studio IDE downloads
- Zend has seen steadily growing interest for Zend Server's enterprise-class capabilities on the Mac platform -- from within the Zend PHP developer community, at the annual ZendCon event, on Zend Forums, and among user groups worldwide.
As Mac use grows in the enterprise, the company reasons, so does PHP use.
More info on Zend Server is available on the company Web site. There's also video tour of the product.
Posted by John K. Waters on 02/01/2012 at 12:27 PM0 comments
The problem the company faced, providing a Web site that would be rich and dynamic on devices that weren't very powerful, is a common one in the enterprise, Galbraith said. Node solves that problem, he said, by allowing his team to deliver sophisticated features to mobile users on the client side. All front-end code in Wal-Mart's mobile app is now executed on the back end.
"We're excited to have a viable back end for that," Galbraith said. "That's why Node really excited us, and at Wal-Mart, we're doing a lot with that kind of architecture right now."
"We rely on services all over the world," Almaer added. "But we don't use all of those services. Node allows us to front all these services... and scale up very nicely. It's perfect for what we're doing in mobile."
The duo (and they are a team) said that they considered using HTML 5, but decided against it.
"We haven't seen people create what we want for retail in an HTML 5 app," Galbraith said. "For us, hybrid is more interesting in something like what the LinkedIn app has done. It's the same UI across all platforms, but it provides a native experience."
If you have any doubts about the potential impact of Wal-Mart's decision to embrace Node.js, consider its influence on RFID. In 2003 the company announced plans to roll out an RFID tagging program with its suppliers. As Mark Roberti put it in his recent analysis of enterprise RFID deployments, "RFID is woven into the fabric of [Wal-Mart's] IT systems." A number of big retailers have followed Wal-Mart's example, including Macy's, which, earlier this month, announced that it would require clothing suppliers to apply passive ultrahigh-frequency RFID tags to items that account for about 30 percent of the company's sales.
BTW: Almaer and Galbraith founded the Ajaxian.
Posted by John K. Waters on 01/30/2012 at 3:47 PM0 comments
Scanning the 2012 horizon with Martin Schneider, research manager at 451 Research, it's hard not to feel excited about the coming year. Schneider, who covers core business applications, app platforms, and next-gen data systems for the enterprise IT analyst firm, expects to see an acceleration of a number of trends that have been gaining momentum over the past few years. Chief among them: the rise of DevOps.
"We're starting to see a confluence and crunching together of systems," Schneider said. "Which is good, because it takes a lot of the work out of managing application system, and a lot of time out of building them. What we're seeing with the cloud, open source and SaaS (as it relates to the cloud) is a lot less that has to be bought from a component perspective, and a lot more freedom for developers."
But, he added, because of the cloud, the relationship of developers to the systems they build is evolving.
"When you build clouds and use cloud components," he said, "you don't just develop anymore; you develop and maintain. These things become living systems, and developers need to understand the use case a little better. They need to get comfortable with the idea of maintaining applications, rather than just building features and functionality. The operational aspect needs to be taken into consideration much more than ever before."
Schneider also agrees with other industry prognosticators that we're entering the age of the polyglot programmer.
"We've seen this already, but I think it's a trend that's going to come to a head," he said. "What sometimes gets missed is that developers going to be expected to know more arcane languages, such as Erlang, which has been popping up regularly over the past 24 months. As we get into highly asynchronous, cloud-based, distributed systems, where you need to write one platform that runs on your desktop, your mobile, and your tablet, you need to know these [kinds of languages], because they're more scalable and cloud-friendly. And not many people do. Right now, there are only a few thousand people in the world that you could call Erlang experts."
"It's smart for the developers to understand this trend now, because it means they'll be able to make more money," he added. "But the important thing to keep in mind is, the days of, Oh, I'm a Java guy, are over. I think 2012 is the beginning of the end of that."
Also, in 2012, developers are going to have to begin thinking like database administrators, Schneider said.
"The cloud changes the way we leverage data," he said. "It's distributed, it's highly scalable. The structured relational model doesn't work in the cloud. The licensing is too strict, it's too expensive, it's too inflexible, it's out of date. Developers need to think about data even more. We've seen that as you see people build for things like Twitter, but I think it's going to explode, because everybody is starting to need to scale the way Twitter does. Business want to get value out of these Terabytes of information that they before they kind of ignored. Developers are going to be asked to start building applications that understand, accept, consume and get value out of all these data."
Finally, the user experience: "Gen Y is starting to use business and consumer applications at a level that no other generation ever did. And they expect the experience to be consistent whether they're at work, at home, working on their laptops, their tablets, or their phones. And that creates a challenge for developers."
Schneider believes that developers are beginning to "get the user experience at a different level," he added. And one spark of a trend he expects to generate some heat in the enterprise in 2012 is the application of BtoC tools, such as loyalty programs and gamifcation (think Zynga).
"It's the idea of pulling people in immediately and helping them to unlock the power of the application, so they become the 'power user' inside an organization," he said. "You drive much more value and lifecycle out of your application doing it that way. We're seeing companies like Bunchball and Badgeville trying to bring this into the business world, usually embedded in things like Salesforce.com. But I think we might see a more generic use of that. "You can build the coolest app in the world, but if there isn't an incentive to use it, that's both natural, innate and around the business, people just won't use it."
Posted by John K. Waters on 01/17/2012 at 11:33 AM3 comments
Two weeks into 2012 and I'm already getting a sense of what the new year holds in store for enterprise software developers. I don't have a crystal ball, of course, but I've been talking with industry watchers who do -- at least I think that's how they do it. Or is it careful observation of industry trends and rigorous data analysis? How about Tarot cards?
I've been checking in with Jason Bloomberg to get the skinny on SOA and enterprise architecture almost since he and Ron Schmelzer founded ZapThink back in 2000, and more recently on architectural approaches to cloud computing. ZapThink became a division of Dovèl Technologies late last summer, and Bloomberg became the division's president. Schmelzer, who often described himself to me as a serial entrepreneur, is busy with his new company, Bizelo.
Bloomberg identified three key the Agile architecture trends that he believes are likely to affect developers:
HTML 5 Reaches a Tipping Point
"HTML 5 is maturing as the next generation of Web markup to the point where we see it as a replacement for Adobe Flash," Bloomberg said. "But also as a common mobile development platform. If you're building an app for an iPhone, you use Objective C; if you're building one for Android, you use Java. But with HTML 5, you can do almost everything you can do with the native application environment, but in a cross-platform way. It gives you much more flexibility and opens up a whole new realm of possibility for mobile and enterprise development."
He also points to HTML 5's WebSocket technology, which provides a way to establish a full duplex tunnel through HTTP for high volume bi-directional communication. "The classic example is real-time stock quotes," Bloomberg said, "but it's really anything that requires a multi-media conversation between clients and servers. It opens up all kinds of possibilities in terms of richer application capabilities in user interfaces, as well as interacting with server side capabilities."
The Next Generation of REST
"Most people don't really get what REST (Representational State Transfer) is all about," Bloomberg said, "even people who call themselves RESTifarians. There's this misconception that REST is about building APIs. What REST is really about is building distributed hypermedia applications. In particular, if you look at the four REST constraints, the most important is that hypermedia have to be the engine of application state. Why this is important is that it frees the middle tier from having to maintain application state, because it pushes application state to the client. The persistence tier still maintains what the REST world calls resource state. But if you can push application state to the client, you can now have a stateless middle tier. This is especially important if you are using the cloud, where you need a stateless middle tier to make that elastic. You can provision additional servers without having to worry about the state of some ongoing application."
"Once people start figuring out what this is all about," he added, "it turns how we do BPM (Business Process Management) on its head. We don't use some central process engine anymore; we rely on the client to maintain application state, and this is the only way we can move BPM into the cloud and take advantage of the elasticity benefits. From the developer perspective, they're already playing with REST, but there's this next step of hypermedia-based workflow based on REST. It just changes the way we do work flow.
Cloud Configuration Management
"There's no way we can do Agile development for the cloud unless we do automated configuration management," Bloomberg said. "We have to raise the bar on that. It's not good enough simply to have, say, a server image that we're going to deploy here and there and think of that as automation. We basically have to come up with a model-driven approach for provisioning everything, so that if we need to do a cache or do some additional virtual machine instances, it can all be done in an automated way."
"What that means is moving that configuration management back to the developers," he added. "It's part of the DevOps story. Developers essentially have to code how operations do their job, because operations should be fully automated. We don't want manual operations, because they can't keep up.
Bloomberg points to his favorite example: Amazon.com.
"The development lifecycle at Amazon.com is more than one deployment per minute, every minute, 24 hours a day," he said. "That's were development is going. How can they do that? It's because it's a fully automated cloud-based environment. It's not like they've gotten everything right yet, but that's the direction. If you look the enterprise, where they have six to nine month development cycles, that's the dark ages! Amazon and Google are generations ahead. That's where the technology is going, and enterprises, to be competitive, will have to get with the program. And part of your role as an enterprise developer is thinking about how you can take advantage of the cloud."
If you're not getting ZapFlash, Bloomberg's semi-regular newsletter, you can grab the RSS feed here.
More predictions for 2012 from savvy industry watchers in upcoming posts.
Posted by John K. Waters on 01/09/2012 at 8:18 AM0 comments
There aren't many of us who will be looking back on 2011 with a wistful sigh. But if my conversations with industry watchers this week are any indication, enterprise software developers have a lot to look forward to in 2012 -- and a couple of things to prepare for.
Gartner analyst Eric Knipp shared a few of his firm's beyond-2012 predictions in an e-mail:
- By 2015, at least 20 percent of the Global 2000 organizations will use Ruby in opportunistic application development projects.
- By 2014, 75 percent of the Fortune 1000 will offer public Web APIs.
- Through 2014, 60 percent of the value of PaaS functionality will be delivered and recognized as SaaS revenue.
- By 2015, aPaaS providers that do not offer differentiated SaaS will not be profitable.
"aPaaS" is Gartner's acronym for Application Platform-as-a-Service, which Knipp and fellow analyst Yefim Natis defined in a report as a "highly productive, easy to learn and use development environment that delivers business applications that are customizable, changeable, capable of implementing serious business functionality and, when deployed, offered with massive scalability, high-end enterprise-class (and beyond) performance and reliability, supporting massive amounts of data, all at SMB prices."
The implications of these prognostications for developers this year, Knipp said, are fairly straightforward and not to be ignored:
- aPaaS will continue to provide a broader range of services. Expect to see serious players acquire context brokerage capabilities to expand the portfolio of services interesting to mobile app developers, for example.
- Expect to "build the APIs first" in an increasing number of projects. You will start from the middle out, rather than designing the presentation or database layer first.
- More polyglot programmers. We will start to value the jack of all trades more than the master of one.
- Bonus -- Not in my predictions but others at Gartner spoke of it this year: We're All Developers Now. Citizen developer activities will accelerate.
To this list, he added another bit of advice: "When you're choosing aPaaS to build your app on, put a premium on ecosystem and ecosystem-oriented features (e.g. multi-tenancy management, billing and subscription management, marketplace, and so on). The last aPaaS men standing will be those who build the biggest portfolio of ISV offerings running SaaS in the platform."
(BTW: If you're not reading Knipp's blog, you should be.)
More predictions for 2012 from savvy industry watchers in upcoming posts.
Posted by John K. Waters on 01/06/2012 at 7:46 AM2 comments