In-Depth

Business Intelligence Through a Clouded Crystal

• By Stephen Swoyer
• July 1, 2005

Talking Points
CASTLE OF THE DARK CRYSTAL

• Crystal faces competition from Microsoft's successful SQL Server 2000 Reporting
  Services offering and the Eclipse Foundation's Business Intelligence Reporting Tool.=
• Some versions of Crystal are missing key features such as a centralized server
  or report repository, report lifecycle management capabilities and an enduser-
  oriented Report Designer.

A reporting tool is a reporting tool is a reporting tool. From the perspective of many enterprise programmers, it’s a truism. That’s because the market for developer-oriented reporting tools has been dominated—perhaps for as long as a decade—by Crystal Reports.

Things have changed a lot, however, and the Crystal of today isn’t necessarily the 800-pound gorilla of developer-oriented reporting tools. It faces growing competition on at least two fronts, principally from Microsoft’s successful SQL Server 2000 Reporting Services offering, and from a nascent open-source initiative, the Eclipse Foundation’s Business Intelligence Reporting Tool. The former has already started to elbow its way into Crystal's bread-and-butter Visual Studio .NET reporting business, and the latter—which just became available in version 1.0 in June—threatens to do the same for J2EE, via the hugely popular Eclipse IDE.

With a revamped Reporting Services on tap in SQL Server 2005, and with BIRT development picking up steam, what’s in the cards for Crystal?

Crystal is promiscuous and ubiquitous
Like many codejockeys, Peter Tyler, a database programming consultant with The Lutheran World Federation, started using Crystal Reports because it came bundled with one of his shop's core business applications—ACCPAC, a shrink-wrapped accounting application published by Best Software. Like many organizations, Tyler’s organization expanded its use of Crystal over time so that it’s now the outfit’s only software-based reporting tool.

“If we operate a database—apart from data stored in Excel, and there is a lot of data quietly stored in Excel—we use Crystal Reports 8.5 to provide reports,” he says. “We feel that it is an extremely rich reporting tool. It can report off a multitude of different data sources. It can be very fast, too, if one knows how to tweak all the options. I can write reports in [Crystal] off our accounting system…which return data much faster than the application itself.”

Crystal Reports is one of the most prolifically bundled products in software history. It has shipped with Visual Basic since version 3.0—released in 1993—and is bundled with development tools from Borland Software, BEASystems, IBM and others. It’s been bundled with many BI offerings (such as Hyperion Solution’s Essbase), and is extensively used as a default tool for operational reporting (going directly against relational repositories) by a large number of customers.

It’s bundled with a bevy of other, non-BI applications, too, such as ACCPAC. In fact, Crystal is so ubiquitous as to be downright promiscuous: Just about every organization uses it somewhere, in some capacity. “Somewhere, in someone’s toolset, you’re likely to come across Crystal Reports,” says veteran industry-watcher Mike Schiff, VP of data warehousing and business intelligence with consultancy Current Analysis.

One upshot of this is almost a de facto lock-in. Many organizations have enormous investments in Crystal, Schiff says, in terms of reports that can’t easily be migrated over to other platforms, as well as in Crystal programming expertise. Developers have been working with Crystal (in some cases) for a decade or more, and many business analysts are also familiar with its use.

Why would an organization heavily invested in Crystal want to switch to another tool anyway? The chief reason is cost: Crystal Reports gives developers an easy way to incorporate simple reporting capabilities into their apps, but in its free, embedded form, isn’t designed to support large numbers of users, nor to be used by non-developers, such as the business manager who wants to create (or customize) her own report.

There’s a reason for this. Crystal’s strategy has been to seed the market with Crystal Reports (through prolific bundling arrangements), encourage adoption of and investment in Crystal and create opportunities to up-sell additional Crystal products and services, says Business Objects’ James Thomas, product marketing manager.

Thomas didn’t—and probably wouldn’t—say as much, but one way in which first Crystal, and now Business Objects, has tried to do this is by leaving developers wanting more. To that end, the version of Crystal that’s embedded with Visual Studio and other development tools is missing key features, such as a centralized server or report repository, report lifecycle management capabilities and an end-user-oriented Report Designer.

“The report design components that live within the IDEs essentially are a lot of the objects for formatting, for data access, for parameters, for formula languages,” Thomas says. “But some functionality, including more of the complex templates and sophisticated design elements, aren’t included in the IDEs.”

Business Objects sells them separately, however. The company provides a standalone Crystal Reports Designer component, which is aimed at business analysts and managers. Effective this year, the company also sells a Crystal Reports Server bundle, which provides a centralized repository for report data, along with report lifecycle management capabilities. Then there’s Business Objects Enterprise (nee Crystal Enterprise), the company’s high-end offering, which delivers fault tolerance (via clustering), support for scheduling and other goodies.

The issue is that these up-sell products are available at prices some IT organizations can’t afford. According to one prominent analyst, who declined to be named, Microsoft’s free Reporting Services offering has forced Business Objects to become more competitive on the low end (witness the new Crystal Reports Server, available only for individual servers with up to four CPUs), but that hasn’t affected the company’s high-end pricing.

As a result, some programmers are forced to make do with plain-vanilla Crystal Reports—and its limitations. Such constraints include a limit on the number of concurrent users (some codejockeys report that performance issues crop up with as few as 15 concurrent users) and strict adherence to tested best practices, such as eschewing the use of subqueries, using OLEDB providers whenever possible, optimizing queries and filtering results.

Until recently, this was simply the status quo. However, Reporting Services has changed that. And for some organizations that aren’t completely locked into Crystal, or who aren’t users of Business Objects BI stack, Microsoft’s enterprise reporting offering has emerged as a compelling alternative.

Reporting services rising
In the 18 months since it was first released, Reporting Services has already won over some longtime Crystal developers, including Juan Arellano, a programmer with a Latin American ISV and services firm who’s versed in Crystal. “I think the strength of Crystal relies on the fact that it’s been the only reporting tool in the market for so many years,” Arellano says. “Thousands of companies have adopted it, and in many cases they aren’t willing to assume the cost of changing to another.”

These days, Arellano says, he prefers working with Reporting Services whenever he can. At the same time, he allows, that’s not always possible because of customer resistance. “When customers let me choose the reporting tool, I go for Reporting Services,” he says. “But that’s not always the case—many customers are already comfortable with Crystal, and they are reluctant to change.” That’s the crux of the problem for Microsoft, which in a rare role reversal, is trying to elbow its way into a mature market that’s dominated by a popular entrenched player. Microsoft has played to its strengths, however, and made some progress, thanks to the ubiquity of Visual Studio and the popularity of its SQL Server 2000 database.

From the get-go, Microsoft has been careful not to position Reporting Services as a competitor to Crystal, Brio and others on the BI reporting front, but as a no-brainer solution for developers working with Visual Studio .NET.

“We’re saying, ‘Let’s just leverage the best development tool out there. Let’s just integrate with Visual Studio so that they can easily build reports,” says Tom Rizzo, group product manager for SQLServer, during the run-up to the Reporting Services launch. “They can use the debugging that they’re used to in Visual Studio, the data tools that they’re used to in Visual Studio, so if you’re a Visual Studio customer, shame on you if you don’t use [Reporting Services].”

There’s some evidence this strategy has been moderately successful. For starters, says Microsoft, Reporting Services has been downloaded more than 140,000 times. Even if only a fraction of the users who download it are using and deploying it in production environments, this still translates into a significant rate of uptake.

There’s anecdotal evidence, too, that many experienced Crystal hands like developer Arellano are at least open to the possibility of using Reporting Services. Microsoft has published several prominent case studies, and its Reporting Services USENET group typically buzzes with accounts of come-to-Reporting-Services experiences.

Take Brandon Osborne, a senior permanent consultant with integrator Encore Consulting. In the on-the-beach, off-the-beach world of consulting, says Osborne, he’s had an opportunity to work “with a little bit of everything.” And even though Reporting Services still has its share of shortcomings compared to Crystal, Osborne thinks Visual Studio developers should think about getting hip to it.

“It may be in its infancy now, but when SQL Server 2005 comes out [you’ll] be considerably ahead of the curve of other developers that thought Reporting Services wasn’t going anywhere,” he argues, noting Encore already has assisted customers in several successful Reporting Services rollouts. “One of our clients in the financial sector utilizes C# and Reporting Services to render the multitude of reports required in their application,” Osborne says. “Considering that the bundled version of Crystal Reports only provides five user licenses and Reporting Services doesn’t cause any licensing constraints to those with a SQL Server license, Reporting Services was a no-brainer.”

Ironically, Osborne says, one of Reporting Services purported strong points—integration with Visual Studio .NET—is actually one of its weakest features, at least in comparison to Crystal. “I like [Crystal’s] level of integration with Visual Studio .NET,” he says. “If Reporting Services had this level of integration, I’d be much more likely to recommend it,” he says while, noting that Reporting Services doesn’t offer an easy way to embed reports in .NET applications. “There should be a way to embed reports in a .NETsolution without exporting the report to HTML format and displaying it in a Web browser control,” Osborne says.

This will change in SQL Server 2005, Microsoft’s Rizzo says, while suggesting a native Visual Studio control will make it easier for developers to embed reporting capabilities—independent of Reporting Services’ SQL Server-based underpinnings—in their applications. One upshot of this, Rizzo suggests, is a thriving OEM market similar to Crystal’s own seeding strategy.

There’s reason to believe Rizzo isn’t telling tales out of school. According to Encore’s Osborne, SQL Server 2005 Reporting Services looks like a more serious competitor to Crystal. It boasts support for multidimensional reporting, and also ships with a new WYSIWYG report design tool called Report Builder. On top of that, the revamped SQLServer 2005 Reporting Services retains the report lifecycle management features and the zero price tag of its SQL Server predecessor.

“Reporting Services for [SQL Server] 2000 leaves a lot to be desired, but SQL Server 2005 hits the nail right on the head,” Osborne concludes. “It’s considerably more advanced than the preceding version and one of the key benefits is that it has been developed to be much more user friendly than Crystal Reports.” The enterprise application development landscape is far removed from .NET everywhere. Java and J2EE have an enormous and growing presence. Yet, the J2EE IDE favored by the majority of developers has long been ill-served on the reporting front. It’s still early, but that’s changing, too.

BIRTh of a Java reporting tool
For a project as ambitious as Eclipse, the absence of an integrated reporting facility seems like a serious oversight. After all, from the time IBM jumpstarted Eclipse in late 2001, almost 3 years elapsed before reporting specialist Actuate and the Eclipse Foundation announced the creation of a new top-level Business Intelligence Reporting Tool (BIRT) project.

Meanwhile, Java reporting tools abound. There’s Jinfonet Software’s JReports, JasperSoft’s Jasper, JFree and StyleReports, to name just a few. Given the availability of these, and other free or pay-for-use open-source tools, Eclipse’s lack of a canned reporting environment might not seem like such a serious liability. To some extent, it hasn’t been.

According to an online poll conducted by the Eclipse Foundation, for example, most J2EE developers prefer to build their own reporting facilities. Even BIRT boosters acknowledge as much. “Basically, 85 percent of them are doing roll-your-own [solutions],” says Mike Thoma, VP of marketing with Actuate and Eclipse BIRTproject leader. “They’re using Java and J2SE, and basically they have two recourses. Their default now is to code [a custom reporting tool] in Java. The second recourse is to buy commercial products. There’s two kinds of those, the 100 percent pure Java ones and those that aren’t.”

From the perspective of many enterprise J2EE developers and the Eclipse Foundation, the lack of a native Eclipse reporting facility is a critical shortcoming. First, pay-for-use Java IDEs like IBM’s Rational Application Developer for WebSphere Software, Borland’s JBuilder, and BEA’s WebLogic Workshop all ship with Business Objects’ Crystal Reports.

Second, it seems illogical: Eclipse is positioned as the most extensible of frameworks, such that a plug-in or extension for almost any conceivable language, feature, or capability can easily be incorporated into it. That a plug-in for such an essential service—reporting—has long been MIA strikes some Eclipse codejockeys as inconceivable.

“I agree 100 percent…that embedded reporting is a key J2EE application development pain-point,” wrote one Eclipse user, a programmer with a government agency, in a post to the Eclipse BIRT newsgroup. Like many codejockeys, this developer says he stumbled upon BIRT while searching vainly for a functional Java-based reporting solution. His problem? None of the open-source tools worked with the complex forms that are de rigueur in the government sector.

In just 72 hours, however, a BIRT community contributor (and Actuate technologist) responded with BIRTdesign code custom tailored for this programmer’s problem. “Given what you’ve done…[and BIRT’s] support for pixel accuracy in reproducing government forms, I now understand that BIRTwill support such forms even though it is not its primary focus,” he wrote in response.

Talk about build to order. That’s the promise of the Eclipse project and of open-source software, in general, and even though the Eclipse Foundation announced BIRT1.0 (or release status 1.0) only in June, a number of programmers are excited about its long-term prospects.

“Our first tests of BIRT have returned good results, basically because it seems to be a good solution to our current report tool problems,” says Martin Miguel López, a Java programmer with TopGroup, an Argentine consulting and software development firm. TopGroup is currently using JasperSoft’s Jasper, says López, but has been frustrated by that product’s shortcomings, which include a steep learning curve for junior report developers, as well as performance issues.

“We have a strong open-source orientation because the economic issues of our country make our enterprise less competitive if we choose commercial solutions,” López says. “So, this means our report tool choices are minimal.” Giving BIRT a look-see was a no-brainer in this instance, he says. “We use Eclipse as our default IDE in Java projects, and we feel comfortable with its perspective-based point of view,” he explains, noting that BIRT, which has a report designer component, can be effectively used by both hardcore Java coders as well as junior report developers.

In a lot of ways, López suggests, BIRT is already superior to most of the existing Java reporting tools he’s looked at. For example, its Report Designer GUI tool generates an XMLfile format that’s proven to be more stable from version to version—a problem with Jasper. There are other differentiators, too. “[R]eports are more flexible and it seems to be so much easier to create complex mixed reports with graphs and texts than with other tools,” López says. “The only thing we found to be a big drawback of BIRT is that one of the most [important] features [for] our customers…is the possibility to export reports to Microsoft’s Excel format, and BIRTdoesn’t seems to have this feature.”

López concedes BIRTisn’t a perfect solution. He says TopGroup would seriously consider Crystal Reports, if that product was open source, “We feel very optimistic about BIRT, as it has become very mature in a very short period since it was proposed as an Eclipse project,” he says.

Three’s company
When you look at the big players in the market for developer-oriented reporting solutions, it’s tempting to picture things in terms of a pincer movement, with Microsoft on one flank and BIRT (or another Java reporting solution) on the other, slowly squeezing Crystal in the middle. To do that would be unrealistic, however. (See related story, “Different and divergent expectations for Crystal.”)

After all, as Business Objects’ Thomas points out, Crystal natively bridges both the .NET and J2EE worlds, something that neither Reporting Services nor BIRT can claim to do. More to the point, he says, Crystal will still be bundled with Visual Studio .NET 2005, along with the three most popular commercial J2EE IDEs.

“If you talk about reporting, you need to talk about any data source and any development language, so it has to be both Java and .NET. It has to work with whatever application server you have, not just WebSphere or Microsoft,” he argues. “We still think there’s a need for an independent, platform-agnostic, Java and .NET reporting platform. This is where you hit the wall with a pure-Microsoft or a pure-Java approach. You have to have both.”

Of course, there’s some doubt among users about just how platform-agnostic the new, Business Objects-stewarded Crystal actually is, but few users would dispute that Crystal offers best-in-class reporting features.

Thomas’ point regarding J2EE and .NET support is a legitimate one, says Current Analysis’ Schiff. Simply by virtue of its huge installed base, he argues, Crystal isn’t going anywhere.

What’s more, there’s plenty of room for all three players. “Alot of people are still trained in Crystal, a lot of companies have huge investments in Crystal reports, and Crystal works with both [J2EE and .NET],” Schiff concludes. “So a more likely scenario is that all three will coexist.”

More on ADTmag.com
BI tool keeps Independence Air competitive
By Lana Gates

Eclipse brings open source to BI market
By ADT Staff

The SQL Server 2005 paradigm shift
By Stephen Swoyer

PHOTO BY BRUCE AYRES