No Rest for the Query
- By Alan R. Earls
- January 1, 2006
The Big Idea
- Data from the Web and apps such as Microsoft Office is making search engines ubiquitous. Orgs want better search capabilities to enhance the value of existing apps and minimize the frustrations of end users accustomed to the simplicity of search engines such as Google.
- What most orgs really need is a universal search capability that can extend beyond any one app. Getting there will require a thorough needs assessment, APIs you can work with, good engineering and patience.
- Search is a hot research area for suppliers. Emerging search strategies designed to streamline the hunt for information include conceptual searching, indexing, clustering, classification, taxonomy, profiling and summarization.
According to Monica Sharp, product manager
at Siemens Medical Solutions, her INVISION
health information system customers must provide
daily information to non-technical end users
within their healthcare organizations to support their workflow
and business practices.
"This places a burden on the IT staffs," she explains, who
must search for the right information and then build, schedule
and distribute these reports. This inefficient process can
cause delays and often results in incomplete or redundant
Today more than half of those customers use Siemens Express
Query to get the info they need. Express Query (with search engine
capability provided by Progress Software subsidiary
EasyAsk) was developed in 2001 to provide
a user-friendly reporting tool.
Using a browser-based user interface,
Sharp says, end users quickly, easily and
securely access their info. "With numerous
navigation options, they may type in
business questions in ordinary English,
build queries through drop-down boxes
and a point-and-click database tree, or
run pre-saved reports with the click of
their mouse," she says.
Answers are presented in several output
options—the most popular is a
spreadsheet—and end users can easily
save search results on a workstation or
distribute reports over a network. End
users can teach their language to the
EasyAsk dictionary, the heart of the app,
further increasing the conversational
ability of the search tool.
Sharp says the decision to implement
EasyAsk was not difficult. The
ability to develop a Siemens offering
using EasyAsk, in conjunction with any
ODBC compliant data source, was essential,
however. Siemens uses many
different database management platforms
and needs to port Express Query
across product lines.
Other software the company evaluated
lacked comparable features and
functionality. With 90 percent of
Siemens' IS customers running Express
Query, "it was a matter of designing an
implementation of EasyAsk that supported
our internal [ASP] requirements,"
Discovery or search?
Keyword search is not dead, but its limitations have vendors hotly pursuing alternatives.
The problem with keyword search, according to Delphi’s 2005 report, “New Perspectives on Enterprise Search,” is that it is focused on retrieving a specific reference and not on “discovery,” which is actually finding what users want. As the universe of searchable data expands, keyword searches will become progressively less useful.
Fortunately, several researchers are harnessing ideas about information to produce search alternatives:
- Conceptual searching produces results by having the search engine know the meaning of specific phrases, if necessary deriving that meaning from other words or phrases in close proximity. The “knowing” can be provided by human input or through machine-based inferences.
- Unified indexing implies that external and internal content has been indexed so well that users can rely on it for finding anything.
- Thematic indexing means related items have been carefully linked under a single, manageable topic.
- Clustering makes the relationship between documents explicit, using themes, key terms or other shared linguistic patterns to group documents on the basis of semantic attributes.
- Classification harnesses semantics in an attempt to accurately categorize the meaning of individual words, sentences and paragraphs.
- Taxonomy is a hierarchy that defines “type-of” or “part-of” relationships.
- Profiling supports the creation of user- or group-specific profiles that map stated interests described as subjects, sources, keywords or content samples. Profiles can be generated on the basis of rules or through automated inferences.
- Summarization focuses on the key elements in a document to simplify searching.
—Alan R. Earls
Siemens isn't the only organization to
add search capability to an app. Everyone
these days is looking for search capabilities
that can enhance the value of
existing apps, power new apps and
minimize the frustration of end users
accustomed to the simplicity of search
engines such as Google. Vendors, some
new, some familiar, are obliging with an
array of products.
Seybold Group analyst Susan Aldrich
believes search engines have become
ubiquitous over the last decade, driven
in part by the growth in unstructured
data generated by the Web and by
Microsoft Office applications.
"Suddenly the impetus to get at that
data has increased, and the capability to
do so has increased, so now application
developers have started to ask why they
need to have users go elsewhere to invoke
a search capability," she says.
Human factors also play a role in the
quest for better search engines. "When
you watch user behavior on a Web site,
you often find people getting lost after
a search or running into ambiguities,"
says Guy Creese, an analyst at Ballardvale
Buy vs. build
Matt Kaiser, product manager at Ceridian,
an HR information services company,
says his customers wanted more flexibility
to slice and access data. The
Ceridian IT team looked at all the data
users wanted to access and decided to
plunge into building its own search and
"At first we thought we would be able
to build it, but then we grew up," Kaiser
recalls. After realizing that a buy would
be preferable to a build, Ceridian began
looking at various vendors. Like Siemens,
they were drawn to the search capabilities
provided by EasyAsk. The search engine
is easy to use and can accommodate
sophisticated and casual users.
Although Ceridian has not done any
formal ROI, Kaiser is certain EasyAsk
was the right choice. The company now
looks to outsiders as suppliers of code
Monster style staffing
Medical device manufacturer Boston
Scientific also needed better search
capability for an internal app. Personnel
recruiters had created a staffing
automation environment for job candidates'
resumes that spanned the
company's 12 domestic hiring locations
and 5 international locations. However,
admits David Pantano, national recruiting
manager, "Everything we did
in recruiting was a manual process. It
was slow, with duplicated efforts and it
To streamline the process, Pantano
and his colleagues issued an RFP that
attracted 22 respondents. The company
selected Recruitmax, a workforce management
software provider. Akey feature
of Recruitmax is the ability to sift
through resumes using an embedded
search engine called HireReasoning,
developed by Engenium.
Recruitmax does not depend on
Boolean expressions; it uses its own algorithms
and learning. The result is
that a recruiter can cut and paste a job
description, say for a Java developer,
into a search box, and Recruitmax will
not only generate a selection of job
candidates but also search on its own
using related concepts such as JDK, J++
and JCK, in addition to Java.
Today embedded search simply needs to
be more effective, asserts Whit Andrews,
a research VP at Gartner. "What
we have here is a world where our employees
are now essentially Google addicts,"
he says. People want to interact
with apps in a Google style, especially
younger people, who have always had
Encarta, Yahoo or Google to help them.
Search rather than structure is essential
to their view of the world.
When Contact Network wanted to strengthen its flagship application’s search capabilities, the company quickly put aside build-your-own notions in favor of an open-source solution. Today Contact’s app, designed to uncover existing professional relationships between employees and potential clients, gets much of its muscle from Apache Lucene, an open-source, fulltext search app written in Java.
“Our product is built on SQL Server,” CIO Geoff Hyatt explains. “We switched from the built-in search feature in SQL Server to Lucene 3 years after our product’s introduction because it has more power and functionality, and [it] has the ability to deliver more accurate results.”
The open-source search add-on has proven to be a low-maintenance and low-cost option. “We would have paid for it because it is worth it,” Hyatt says.
—Alan R. Earls
Even the industry's iconic apps are
feeling the heat. Outlook has had search
capability for several years, but Microsoft
is now being pressed to improve it and
make it more intuitive, says Andrews.
Of course, not everything is built to
be searched Google style. Databases, for
example, are built to be parameterized,
Andrews notes, because that is inherent
in the database structure. Orgs must decide
whether to conform to an app's inherent
search structure or to trump it
with new search capability.
Googling the enterprise
The real need in most organizations, whether it is expressed formally or not, usually boils down to a universal search capability that can extend beyond any one app, says Whit Andrews, a research VP at Gartner.
Several suppliers offer enterprise-oriented search apps—FAST, Autonomy, Endeca, IBM, Hummingbird and Convera. “[Apache] Lucene is a well-known open-source offering that is getting a lot of use as an embedded search capability,” says Andrews.
With OEM or embedded search apps, orgs should bear in mind a few key thoughts, Andrews says. Look at the search engine provider’s contractual relations. “Are they going to offer generous terms that will allow you to actually make money off of the search capability?” he asks.
Harder than it looks
Think through the challenges that search add-ons may create. Most important, Andrews says, is getting APIs you can work with. “The ability to integrate into a variety of content repositories—things like ODBC compliance and specific brand relationships such as Informix or Oracle,” he says, are central to getting value from embedded search apps.
Looking at the APIs is crucial, but it isn’t enough, asserts Guy Creese, an analyst at Ballardvale Research. “You need to really see what you are getting and whether you will be able to work at a level that is granular enough to accomplish your search goals,” he says.
Dan Keldsen, an analyst at the Delphi Group, cautions: How easy it is to add a search engine and how well it will work still depend on a thorough needs assessment and a good engineering approach. “The effort and the results can range all over the place,” he warns. If you have different search engines already within apps, portals and so on, and you try to bridge that, it may be hard to get good at searching for any one thing, he adds.
To do search well is still a hefty task. “You can put together a quick-anddirty search solution, but if someone spends more time, you can start to peel back the functionality you really need,” Keldsen says. Eventually, you may get to the search capability that really provides what you and your users are looking for.
—Alan R. Earls
People are no longer willing to accept
limitations, and they don't see
why they should have to use multiple
search points to get at business data,
Andrews says. Because Google seems
to give them everything they want—even though it is not able to access a
great deal of information—people
want to have a similar experience with
their business searches, "and that's a
tall order," he says.
ILLUSTRATION BY ROBERT PASTRANA