Knowledge portals unify data streams
- By Colleen Frye
- June 12, 2001
Over the years, different types of software tools have tried, with limited
success, to address this issue, from groupware and collaborative tools
to document management, to information search and retrieval to knowledge
management. The rise of the Internet/intranet/extranet has only exacerbated
the problem, bringing even more disparate and far-flung data to the corporate
network. Turning both structured and unstructured data into information,
or knowledge, to better accomplish business objectives is still a quixotic
quest in many ways.
But the trend toward Internet portals such as Yahoo!, which organize,
categorize and even personalize information, and that also provide a single
point of access to things like online shopping, news feeds and stock trading,
is beginning to be reflected in the corporate world with the emergence
of corporate portals.
According to the Delphi Group, a corporate portal sits on top of the
intranet architecture, providing a single point of access to internal
as well as external data. "Corporate portals go far beyond integrating
information spread across a company's intranet only," said Larry Hawes,
senior analyst at Delphi Group, Boston. "Portals also provide a unified
view of information from the Internet and from internal legacy systems,
including ERP and mainframe systems."
While very early in the market, corporate portals promise to deliver
on helping to turn data into information by organizing and categorizing
it, as well as putting some meaning around it and then getting it to the
people who need it to do their jobs. "Information is data put together
in revealing ways," said Ian Black, head of communications and public
affairs for British Aerospace, Farnborough, Hampshire, England.
Corporate portals represent the coming together of various "knowledge"
technologies. "I see the corporate portal as an evolution from document
management, encompassing document management and knowledge management,"
said Jim Wessely, senior consultant, Central Research & Development
at DuPont, Wilmington, Del. "Document management never did hit the mainstream,
and knowledge management has been vague at best. The concepts embodied
in those discrete market areas are encompassed in a portal. To make centralized
information available, and to receive the information you need when you
need it, you've got to have document management, data mining, text mining,
knowledge sharing and acquisition, and access to knowledge capital. It
means the portal
really is the encompassing body."
A corporate portal's primary function is quite straightforward, sums
up Derek Binney, marketing director of the corporate knowledge program,
Computer Sciences Corp., El Segundo, Calif. "A corporate portal is aimed
at integrating knowledge management with doing real work; you don't have
to go somewhere else to do the 'knowledge' thing," noted Binney.
Searchers on target
The Delphi Group projects that the cororate portal software market will
reach $740 million in 2001. Today, there are about 40 ISVs with applications
or tools that provide one or more elements of portal functionality, according
to Delphi. These ISVs come from a variety of established product categories:
collaboration, search, knowledge management, information aggregation and
publishing, Internet, document management, business intelligence and ERP.
Each brings something different to the table. In addition, a new category
of software, the Enterprise Information Portal (EIP) is emerging, with
new players such as Columbia, Md.-based Sequoia Software, which offers
an XML-based framework.
Three companies with a heritage in information search and retrieval
that are targeting the corporate portal market are Verity Inc., Semio
Corp. and Autonomy Corp. Where general purpose Internet search engines
go broad, often turning up an overwhelming number of "hits," these information-retrieval
products go deep, turning up more targeted and meaningful "hits." Delphi
also puts Dataware, Cambridge, Mass., Excalibur Technologies Corp., Vienna,
Va., and Toronto-based Hummingbird Communications Ltd. in this category.
As companies develop more sophisticated intranets, they are demanding more
from their retrieval products. "Two years ago, customers were poorly educated
about the power of retrieval technology," said Ronald Weissman, vice president
strategy, corporate marketing at Verity
Inc., Sunnyvale, Calif. "The Web has done a great job of educating people
in the basics.
"Everybody knows how to do a search in Lycos," he continued. "But in
the corporate setting, where you're dealing with legacy data and problems
of classification and security, that's a much more sophisticated [search]
than popping up Web pages. Most customers don't understand linguistics
or semantics, but they have a clear sense of what they want in a business
perspective. The corporate portal has redefined that territory."
What businesses want is a way to bring order and usability to chaos.
"We have tons of data all over the company in discrete islands, and in
different formats and technologies, from DB2 to Notes to document management
systems to individual file systems to paper archives," said DuPont's Wessely. "A lot of this stuff
isn't doing a heck of a lot of good; the trick is to get it out where
it can be used."
In response to this demand for more sophistication, search-and-retrieval
technology vendors like Verity, Semio and Autonomy have "moved well beyond
simple keyword indexing and searching to offer advanced full-text capabilities,
including automatic information categorization and concept-based analysis
of documents," said Delphi's Hawes. "They can classify single pieces of
information and then relate them to other information based on concepts
articulated in their content. The value that these products bring to the
corporate portal environment is their ability to automatically and transparently
analyze, organize and reveal structured and unstructured information from
multiple internal and external sources."
Each of these vendors takes a very different approach to search and
retrieval. Verity's heritage is in keyword searching and Boolean logic.
Semio uses a pattern recognition approach to match semantically related
terms. Autonomy, which has its roots in neural networking technology and
was spun out of Neurodynamics in 1996, uses adaptive pattern recognition
to find related ideas. And while Verity and Autonomy sell their software
solutions, Semio's model is one of a service provider.
"One of the key differentiators between Autonomy, Semio and Verity is
the level of user involvement needed to construct the initial taxonomy
that drives categorization and concept clustering," explained Hawes. "Semio
is zero-based; it automatically generates a taxonomy without user input.
Verity requires an administrator to develop an extensive taxonomy before
it can group information. Autonomy occupies a middle ground in that it
can start with a simple pre-built taxonomy and create a more elaborate
one as it analyzes document collections."
A critical differentiator, noted Hawes, "is the completeness of portal
functionality that each vendor supplies in its product. Autonomy's Portal-in-a-Box
and Verity's Knowledge Organizer provide much of the functionality portal
implementers need to begin to support a portal project. Semio's products
form a less complete toolkit and are not integrated into an out-of-the
box portal solution."
Verity is the most established of these three players, while Autonomy
and Semio are relative newcomers. Verity's Weissman said the company's
philosophy behind its "topics" technology is a combination of human and
computer skills. "Our customers find that some human-authored business
rules give you a more precise determination of what you're looking for,"
he noted. Said Delphi's Hawes, "Verity's 'topics' approach allows for
highly accurate control of the categorization process and the application
of human library science."
Verity's knowledge retrieval suite of products includes the following
pieces: Information Server, Intranet Spider, Agent Server, Knowledge Organizer,
CD Web Publisher, Verity HTML Expert and KeyView Pro.
At Computer Sciences Corp. (CSC),Verity technology plays a key role
in the knowledge environment, which consists of the knowledge base, collaborative
technologies and communities of interest, and a corporate portal. CSC
supports 55,000 users on IBM Netfinity equipment, hosted in four major
centers around the world. CSC's Binney said the company wanted an engine
that provided a universal search of CSC's knowledge base without needing
to know where the data was. At CSC, that knowledge-base data resides in
Lotus Notes databases and Domino-enabled Web content, relational databases,
the company intranet and outside news feeds. In addition, Verity "drives
one of the major taps off the portal." Binney said CSC used Cold Fusion
to develop the portal, and has plugged in best-of-breed technology.
CSC is also in the process of negotiating an alliance agreement with
Verity, which would give CSC access to Verity development and research.
While CSC uses Verity to enhance internal users' experience, New York
City-based Chase Manhattan uses Verity technology to enhance its external
customers' experience. Chase recently relaunched its Web site to be more
customer-centric. Chase liked Verity because of its functionality out
of the box, said Jennifer Baliotti, product manager for the search engine
"We only had four months to gather the bank-wide requirements, select
the product and implement. We didn't have a lot of time to customize,
and we had specific needs," she said. "For example, we've broken the site
into avenues that speak to different customer segments." These segments
are personal, small business, and corporate and institutional. "When customers
use the search engine, we want to provide them with results specific to
their avenue of interest."
A feature unique to Verity among the products Chase evaluated was the
ability to highlight the words in the search results. "So every time a
customer searched for 'loan,' the word 'loan' would be highlighted each
time it appeared on a result page; you would have had to customize other
apps to get that feature," said Baliotti.
For users who know what they are looking for, such as loan information, text-retrieval
technology such as Verity's enables "you to retrieve as much as you can about
what you know is there," said DuPont's Wessely. For those who wish to discover
which information is relevant, Redwood Shores, Calif.-based Semio
fits the bill, said Wessely.
For the past five years, Wessely has been researching ways to mine textual
information, and to show primary, secondary and even tertiary relationships
among that information, as DuPont seeks to become a knowledge company
internally. Wessely said he has researched neural network-type technology,
AI rules-based natural language technology, semantic analysis and more,
filling a file cabinet with enough technical esoterica "to choke a horse."
For DuPont's purposes, Wessely "found two approaches that can make sense:
some of the neural [network technology] and what Semio is doing.
"Semio takes into account how the language is used," he said, "as opposed
to most neurals, which rely upon the proximity relationships of terms
to determine association. Both are valid approaches."
Wessely's first use of Semio was as an add-on to a departmental virtual
library, which he developed several years ago for the business development
group. His challenge at the time was to corral all the different sources
feeding into the company's library for this discipline, from commercial
reports and databases, to FoxPro and Excel files to books on shelves,
and to "put it into one coherent repository that was easily approached"
so people could gain insight without physically going to the library.
The virtual library provided numerous ways to access information, from
a table of contents, to a search to various links. Wessely chose the Verity
search engine, he said, because of its robustness, scalability and flexibility.
Then he discovered SemioMap, which creates visual, navigable representations
of text-based content, and added it to the virtual library so people could
"get info they didn't know about.
"When you have a search engine, you know what you're looking for and
what you want extracted," said Wessely. "On the other side of the coin,
you have a body of information you want to turn into knowledge; that's
where mining and a discovery engine come in." SemioMap shows the relationships
between ideas or phrases so "I'm able to 'connect the dots,'" he added.
Semio has since rolled out a taxonomy-generation tool, which it is providing
as part of a monthly service. According to President and CEO Roger Ferguson,
Semio consultants will go on-site and work with project leaders to define
where the data is coming from and what the company wants the taxonomy
to do. Semio will then build the taxonomy and update and maintain it over
"What we produce looks like a Yahoo!-style directory," explained Ferguson.
"You would look for things by browsing the directory, which goes from
general to specific. The difference between us and a search engine is
that the user generates his own notion of what he's looking at. We present
you with a list of concepts found in the documents, and you select the
concepts meaningful to you. You can start with not knowing very much and
quickly hone in on what's meaningful."
He added, "A unique feature is that you can define and build multiple
taxonomies for the same set of documents. For example, the HR department
might want to look at a document in a different way than engineering."
While Semio will rent its taxonomy-generation software, Ferguson said
they do not encourage it because "there's still art involved" with building
and maintaining a taxonomy.
DuPont's Wessely said Semio's service model is a good one, because "Semio
understands what it takes to generate a taxonomy. In general, most people
will be stumped. However, there will be an occasion [for some companies]
where nobody external to the company is allowed to see the data, whether
because it's private or a matter of national security, for example. At
that point, Semio will have to give out the tool. If I were them, I would
not target those types of application areas right off the bat."
As a Fortune 10 company, though, DuPont is working closely with Semio
and learning how to generate a taxonomy, said Wessely. He expects this
feature to play a role in the company's planned corporate portal, for
which he is the principal architect. While the portal is under development
and subject to change, he said it will be aimed at internal employees
and appropriate contractors. DuPont is developing a portal prototype because
"we need to empower the knowledge worker. Without the enabling technology
to do that, it will never happen."
Even with the technology, moving users to a knowledge mode and getting
them to take advantage of things like corporate portals takes some education
and training, as well as a cultural shift, suggests British Aerospace's
Black. "The single biggest issue with all this technology is behavioral,
not IT. It doesn't matter how good the technology is built to understand
the user if the user refuses to take advantage." Getting users to understand
how to interact with all this information coming back through such a small
window takes time, he said. "The technology is scalable, but the users
British Aerospace is using San Francisco-based Autonomy's
Portal-in-a-Box as the customer-facing portal in its intranet, as well as
a portal to the company's virtual university. The company moved toward portals
because "we were looking at an intelligent way to make sense of the huge amount
of unstructured data that sat on our own networks, and that we could also [use
to] make sense of structured data as well," said Black. "Last year, we started
as a way to find a more intelligent process for searching, but now it's more about the management of information and
data; it's much more than searching."
The goal is simple, he said -- to get the right information to the right
person, at the right time. A unique feature of Autonomy's technology is
that registered users can have their own Web page, with information tailored
to them. Users create agents that go out and bring back information, and
are then automatically retrained as the users' search becomes more refined.
The tool "brings information to someone about something they didn't even
know they wanted to know," said Black.
Black selected Autonomy's product, which he considers second-generation
technology, after looking at first-generation search-and-text tools. He
said the cost of operation for Portal-in-a-Box is low, and it is easy
to set up and use. "It has been written well, very tight, and it hasn't
suffered years of bits and bolts being added to it. It literally comes
out of the box and runs." However, he added, "it would add a lot of configuration
[time] if you really want to get it bolted into your organization" by
integrating it with applications.
That is what the Department of Defense is doing, said First Lieutenant
George Hellstern with the Chief Technical Task Force at the DoD in Crystal
City, Va. The DoD is integrating Autonomy's product with the Joint Reserve
Intelligence Support System. "For our application, we're doing a lot of
customized things. For general use, it's not something you need to customize
The goal is twofold: to develop a list of all IT skills throughout the
DoD, and to have a knowledge management tool for the department. Hellstern
said when a user fills out a resume on the Web site, it will produce a
portal tailored to the individual's needs or what they do. Then each user
will have agents, as well as the ability to add new agents, which will
be constantly looking for the latest news on a particular topic, say aircraft
Portal-in-a-Box gives users control over what gets returned to them,
which reduces the chance of getting information back that may not relate.
However, Hellstern did say that a downside is that the user has to point
at what he or she wants to look at. "You have to have an idea of where
you want to look. Say if I was searching for colleges and didn't put in
.edu, then nothing would get searched at .edu."
Personal and critical
Hellstern, however, praises the personalization features of Autonomy,
which Verity officials said the company is working on and Semio said it
"Personalization is a critical element of both knowledge management
and corporate portal applications," said Delphi's Hawes. "End users need
to be exposed to relevant information only, rather than wading through
the overwhelming amount of information available to them. The rise of
e-business demands that knowledge workers be able to control their computing
interface so they can quickly access relevant information and participate
in the business processes germane to their role in the organization. An
interface is not a portal unless it can be personalized manually by the
user and/or automatically by leveraging attributes contained in a user
Added Michael Lynch, Autonomy CEO, managing director and founder, "With
older technology, you can only put in a query. With new technology, it
can read whole pages. While you're working, the system reads what's on
your [personal] page and pops up with information that might help you
with what you're working on. It brings that work to your work."
And getting work done more efficiently and effectively is what corporate
portals and knowledge management are all about. But different companies
have different needs, and information-retrieval tools like Autonomy, Verity
and Semio all bring different features to the table. Some companies, like
DuPont, may end up using two or more of these products together as part
of their knowledge architecture.
Delphi's Hawes adds that "the search-and-retrieval tool is only one
of many necessary to provide a comprehensive portal application. Specifically,
in addition to categorization and search capabilities, a portal will need
integration with other enterprise applications, collaborative facilities,
publishing and distribution features, and business process support."
He also warns that few portal products at this early stage are complete.
"A company must choose a specific product as a base for a corporate portal
development project and integrate the functionality of other software
offerings to complete the picture."
DuPont's Wessely said his company will use open technology and take
a best-of-breed approach, believing that no one product can be everything
to everybody, especially a Fortune 10 company like DuPont. "If someone
has a complete portal to plug in, I know it can't do everything I have
to do," he said. It will come down to a business decision for many companies,
he believes: "Doing it right vs. just getting by."
In the meantime, as companies begin building their corporate portals,
they are hoping to achieve the promise that "knowledge is power."