Shopbots: A Syntactic
Present, A Semantic Future
T
he Internet’s immense growth has led to dra-matic changes in business-to-consumer com-merce (B2C). E-comcom-merce sites abound, offering users a wide range of services and products. People can now shop from their homes for every-thing from flowers to cars; the traditional shopping trip has been transformed into a virtual one.Although users have product and services information at their fingertips, they increasingly require assistance, guidance, and support to navi-gate the Internet’s vast information space and make informed purchasing decisions. Shopbotsare specialized agents that help users sift through this information by
• finding product specifications and reviews; • comparing products, vendors, and services
according to user-defined criteria;
• finding the best-value products or services; • monitoring online shops for product
availabil-ity or special offers and discounts and sending user alerts;
• recommending services and products; and • identifying new products of potential interest.
Once a shopbot receives a product request from a user, it queries vendors and providers. It then retrieves and compares — or differentiates and evaluates, according to Yun Wan and colleagues1—
relevant information such as price, delivery options, and warranty. By performing such
com-parisons for the users, shopbots ease the purchas-ing process and enhance users’ shopppurchas-ing experi-ence, while also saving them time and money.
Despite the initial hype,2 however, shopbots
have yet to fulfill their immense potential. This is primarily due to technical limitations: shopbots don’t yet “understand” the information that they retrieve and (moreover) can’t yet dynamically dis-cover and query vendors. Here, I discuss current shopbot limitations and how using the Semantic Web and Web services can solve these problems.
The Online Shopping Experience
At the onset of e-commerce, users were wary of online shopping and reluctant to provide personal and financial details to online vendors for fear of credit-card fraud and loss of privacy. In recent years, vendors have better addressed security and privacy issues, and users have become more famil-iar with the technology. As a result, online shop-ping sales have increased massively.Existing Approaches
Users typically look for products online by visiting familiar sites or by identifying potential vendors using standard search engines and keyword re-trieval. Users then visit each site and search for the product, its price, and other related information.
This basic approach has several shortcomings.3
First, hundreds of vendors might sell the same or similar products. Unless users are aware of
partic-Despite high expectations, shopbots have yet to significantly facilitate a richer, more satisfying online shopping experience for users. By taking advantage of Semantic Web and Web services technologies, however, researchers can overcome current technological limitations and finally realize the shopbot’s significant potential.
ular vendors, they must decide which and how many sites to visit to acquire the necessary infor-mation. Although visiting multiple vendors re-quires considerable time, not visiting enough can produce suboptimal decisions.
Second, identifying vendors through search engines might simply return the largest vendors, not necessarily those with the best prices. Third, if users need several items that aren’t typically sold togeth-er, the search time increases for each new product category. Finally, each time users visit new vendors, they must acquaint themselves with new interfaces. This not only further increases search time but also unavoidably hinders impulse shopping.
To help customers stay informed, vendors sometimes let users sign up for product alerts, which might inform customers about price changes, for example, or about the availability of a forthcoming or out-of-stock product. Although these alert services aren’t personalized, users must often complete long surveys that sometimes request personal data. Furthermore, submitting an email address for notifications in itself reveals an aspect of the user’s identity and thus weakens his or her privacy.
Shopbot Benefits
Delegating product research, comparison, and evaluation to shopbots can clearly offer users sig-nificant benefits. First, shopbots can save users time by searching numerous vendors quickly. By querying multiple vendors — including those that users are unfamiliar with — shopbots are more like-ly to get the best deals for their users. Shopbots can also uncover special deals that users would be oth-erwise unaware of.
The second potential benefit is in psychologi-cal burden shifting.4 Users often feel uncertain
about buying products but, by using shopbots, users can shift some of the psychological costs of decision making onto the agent. If a decision turns out to be less than ideal, the user can blame the shopbot and thus minimize his or her own psy-chological risk. (Indeed, because they’re using shopbots, users might remain unaware that a rec-ommendation is suboptimal, further minimizing the psychological burden.)
Finally, shopbots can create marketplaces that are more efficient. Shopbots let users easily com-pare vendor prices and services and thereby increase marketplace competition. Users also gain access to smaller vendors offering competitive prices and thus reduce larger vendor monopolies.
Shopbot Technology
When shopbots appeared in the mid ’90s, tech-nology analysts predicted they’d have a huge impact on vendors and on business conduct in the years to come.2However, shopbots have yet to
meet this potential.
Characteristic Shopbots
The first shopbot for price comparisons was Ander-sen Consulting’s BargainFinder,5which compared
prices on music CDs from online stores. However, because BargainFinder simply evaluated vendors based on their prices — ignoring all other site fea-tures — many music retailers started blocking access to the agent. Eventually, BargainFinder stopped operating.
PersonaLogic6helped users make decisions on
products and services based on their needs and preferences. Given a user profile, PersonaLogic could identify products with features the user val-ued. However, vendors had to provide an interface that explicitly disclosed product features in a way that matched the agent’s user profiles. In 1998, AOL acquired PersonaLogic and withdrew it from the market.
ShopBot7 submitted queries to e-commerce
sites and parsed the results to extract product information. The agent exploited regularities in e-commerce sites, using an automatic process for building wrappers to parse semistructured HTML documents. It then extracted product features from the documents, such as product description and price. Shopbot, which was later renamed Jango, avoided the vendor-blocking problem that BargainFinder faced by sending product informa-tion requests from the user’s browser, rather than the agent’s server. In 1997, Excite acquired Jango and incorporated it into its search technology. Excite has since evolved into a vendor-biased comparison site.
A more recent shopbot example is IntelliShop-per,3which observes users’ actions and
unobtru-sively attempts to learn their preferences and help them with their shopping. IntelliShopper also mon-itors various vendors for products that might match user needs and preferences. The agent also preserves user privacy: it conceals users’ IP addresses by passing requests through one or more anonymizer servers. Moreover, it uses shopping personas to hide all user information from the IntelliShopper server. The system stores logic about the different target vendors in modules. These modules specify how the agent can query vendors,
interpret the results, and extract the information users require from the returned HTML pages.
Technical Limitations
Shopbots operate like meta search engines, using a form of screen-scraping8to retrieve information
from different vendors. The agents typically let users sort product and vendor information along some desired dimension, such as price.
Essentially, shopbots interact with vendors through HTML pages that were originally designed and generated for humans to read and understand. Shopbots submit queries to vendors and attempt to process the resulting HTML pages by parsing them and searching for the target item’s name. They then search the nearest set of characters with a currency sign (presumably, the item’s price). To retrieve such information, shopbots rely on the fol-lowing regularities across e-commerce Web sites:7
• Navigation — sites make it easy for users to navigate and find what they need.
• Uniformity — Web pages of a particular site look similar.
• Regular vertical separation— sites use spaces to separate products, and each new product description starts on a new line.
In fact, the vendor’s database has the information that shopbots need in a machine-processable and well-structured format. However, as Figure 1 shows, the shopbot can’t interact with the data-base: it has access only to the Web pages that are generated from the database. Consequently, developers must implement clever heuristics that can extract the original, well-structured data from the HTML pages’ implicit information. Such heuristics are inevitably ad hoc, difficult, and time-consuming to develop. They’re also error prone and can retrieve only limited information. Inevitably, developers must update the shopbot’s heuristics whenever the layout of the vendor’s site changes. This makes shopbot development cumbersome, and the resulting systems inflexi-ble and vendor-specific. Shopbots therefore can’t discover and query new vendors at runtime: the shopbot’s developers must first identify vendors, then tailor methods for retrieving information from their Web pages.
Also, because current techniques rely on syn-tax, shopbots can retrieve only limited informa-tion from vendors, ignoring harder to retrieve attributes, such as warranty and shipping options.
This is a severe limitation. Although price is often the most important factor in purchasing decisions, other attributes also factor in, including delivery and payment options and the vendor’s reputation. Such value-added services can be the deciding fac-tor when several vendors offer a product at the same price. Also, vendors naturally object to a comparison of price alone because it fails to reflect the full range of vendor services and value. Con-sequently, many vendors block shopbot price requests. Shopbots are thereby restricted in their ability to compare multiple vendors and help users find the best deal around.
Finally, syntax-dependent shopbots can inad-vertently retrieve inaccurate pricing information, creating discrepancies between the prices the shop-bots report and those listed at the vendor’s site. The reason is that shopbots don’t understand the retrieved Web pages’ content. Consequently, they can’t distinguish between such things as vendors that include shipping charges and taxes in their prices and those that don’t. These technological limitations have led to the demise of vendor-independent shopbots. They’ve also contributed to the emergence of comparison sites (see the “Com-parison Shopping Sites” sidebar).
Shopbots and the Semantic Web
Despite high expectations of shopbots’ impact on retail markets, their true potential hasn’t been real-ized. This is mainly because of inherent limitations in the shopbots’ information extraction technique — that is, screen scrapping. Web services offer a highly promising alternative to this approach.Figure 1. Shopbots at work. Shopbots use vendor-specific wrapper programs to wrap up and submit a user’s request for product information to multiple vendors through HTML pages. The wrappers parse the returned HTML pages and extract the required product information.
Task Shopbot Users Wrapper1 HTML pages Vendor 1 Database Wrapper2 HTML pages Vendor 2 Database Wrapper3 HTML pages Vendor 3 Database
Web Services Overview
A Web service is a collection of functions that a developer packages as a single entity and publish-es on a network for use by other programs.9
Devel-opers can describe Web services’ programmatic interfaces using the XML-based Web Services Description Language (WSDL; www.w3.org/TR/ wsdl). Service descriptions include the input and output parameters and their data types, the format for message exchange, and the service’s opera-tions, network address (the Uniform Resource Iden-tifier, or URI), and protocol bindings.
To facilitate a service’s discovery by requesters, developers can describe and register it in direc-tories. Universal Description, Discovery, and In-tegration (UDDI; www.uddi.org) describes a mechanism for registering and locating services. Once a requester knows the service’s URI, it can use SOAP (www.w3.org/TR/soap) to invoke the ser-vice with its parameters. Web serser-vices’ power lies in their ability to interact seamlessly and trans-parently to achieve combined functionality and produce the required result.9
Vendor Gateways
Because Web services can retrieve information directly from the vendor in machine-processable form, shopbots can use them as gateways to ven-dor sites. For example, Amazon (www.amazon. com) offers Web services that let programs per-form operations such as retrieving product information and adding items to a shopping cart. Agents can thus employ Web services to acquire, aggregate, and compare information on products, services, and vendors. This greatly enhances shopbots’ potential for comparisons based not only on price, but on various other attributes as well.
Shopbots can potentially access numerous ven-dors, provided that they use programmatic inter-faces to facilitate information retrieval. However, this process could be hindered if agents don’t understand references to product or service attrib-utes. Unless an agent understands that “warranty” and “guarantee” mean the same thing to different vendors, for example, it won’t be able to correctly retrieve information and present and compare it in a uniform or meaningful way.
To facilitate this common understanding among agents, underlying domain ontologies are required. Such ontologies can be used by
• developers to describe Web services, and
• shopbots to understand and process the retrieved information and present it to users. Therefore, to take advantage of Web services, developers must endow shopbots with the ability to use and reason about ontologies.
Limitations of Existing Standards
In an increasingly interconnected world, hundreds of providers might offer a particular service or product. Consequently, shopbots need some means of locating providers. One obvious solution here is to use a UDDI registry to discover providers and their services. This would give shopbots a service description and location, and instructions for invoking the service. However, current XML-based standards for describing Web services — such as SOAP and WSDL — are designed to provide descriptions of transport formats and mechanisms, address binding, and the service’s interface, but not descriptions of a service’s capabilities.
Similarly, UDDI fails to represent service capa-bilities and thus can’t facilitate automatic Web services location based on capability specifica-tions.10,11Thus, while shopbots could search a UDDI
registry for US vendors, for example, they could-n’t search for vendors that ship outside the US. Searching for services with such specific capabili-ties is extremely difficult. UDDI registries also lack an explicit semantics: although two XML service descriptions might look exactly the same, they might mean different things in different contexts.
Enriching Service Descriptions
For shopbots to achieve their full potential, the matching between a service request and potential providers must take into account semantics and capability specifications and must be done in con-text. Hence, we need both richer Web services descriptions and a way to process those descriptions. The DARPA Agent Markup Language (DAML; www.daml.org)10and its most recent incarnation,
the Ontology Web Language for Services (OWL-S; www.w3.org/Submission/OWL-S), provide a more suitable framework for describing services capa-bilities than SOAP or WSDL.
In OWL-S, developers describe a Web service using three interrelated sub-ontologies:
• ServiceProfiledescribes a service’s capability specification, which is essential in determining whether or not it meets the requester’s needs. • ServiceModeldescribes how a requester can use
the service, how to ask for it, and what happens when it’s executed.
• ServiceGroundingspecifies the communication protocol, message formats, and other service-specific details, such as the port numbers used to contact the service.
Selecting providers that use these enhanced descriptions would require a registry that lets agents search it using semantics that UDDI doesn’t currently offer.11Alternatively, middle or broker
agents could act as ontology-based search engines and take into account semantics.12In this case,
specialized middle agents that store, maintain, and provide connection information would match shopbot service requests with one or more pro-viders. Vendors would publish and register their Web services with these middle agents, and shop-bots would query the middle agents when receiv-ing user requests for products.
Realizing the Vision
Figure 2 shows how shopbots could use the Semantic Web and Web services:
• Vendors create Web services that allow access to their sites, and they publish and register these services with middle agents.
• Users delegate the task of finding and compar-ing products to shopbots.
• Shopbots contact middle agents who identify and return a set of providers offering the ser-vice along with information on how to contact and interact with the service.
Given a user’s product query, the shopbot invokes the appropriate service and retrieves the required information. In this setting, shopbots could dynamically retrieve information from vari-ous vendors without having to depend on static (and sometimes unreliable) links. They could also cache Web services’ locations for common requests, thereby improving efficiency. The end result would be easily digestible information that users could consider to make their final purchasing decisions.
If shopbots are to retrieve semantic informa-tion on vendors’ products and services, researchers must address several outstanding challenges, including the development of
• efficient algorithms for matching requests with Web services;
• efficient ontology services, including ontology consolidation services; and
• reasoning mechanisms that let shopbots per-form true comparison shopping on the basis of user preferences, the retrieved information, and the underlying ontologies.
Although standardizing ontologies to describe
cer-Figure 2. How shopbots could use the Semantic Web and Web services. Upon receiving a user’s product query, the shopbot contacts one or more middle agents and requests information on matching vendors that have registered their Web services. Shopbots, middle agents, and developers all consult ontologies to facilitate common understanding and information exchange.
Middle agent
Vendors Web
services Ontologies
Service invocation (SOAP) Publish Register Request Task Shopbot Users
tain retail sectors would be ideal, ontology stan-dardization is a general problem. Consequently, shopbots and middle agents must be able to rea-son with and compare multiple ontologies.
So far, work on shopbots that can use Web ser-vices to retrieve vendor information has been rather limited. One recent approach, the Intelligent Product Information Search (PIS) system, offers meta search and comparison for products.13IPIS
assumes that shopping malls offer Web services to describe their available products. Upon receiving a user request, IPIS configures appropriate vendor-specific queries using an ontology and submits them to the vendors’ Web services.
The IPIS system follows some of the principles described here, but has two important limitations. First, it can retrieve information only from a pre-defined set of vendors whose programmatic inter-faces (that is, Web services) are known in advance. There are no directory services or middle agents to let shopbots discover and query vendors at run-time. If none of the predefined vendors provides the requested product, IPIS cannot search for alter-native vendors.
Second, while IPIS uses an ontology, it is a cen-tralized ontology containing the predefined vendors’ vocabularies. To enable dynamic product informa-tion retrieval from multiple vendors and present it consistently to users, agents must have access to and reason with multiple distributed ontologies.
Despite its limitations, IPIS is a step in the right direction and illustrates that the use of the Seman-tic Web and Web Services can greatly enhance shopbot functionality.
Toward a Semantic Future
Transitioning shopbots to the Semantic Web and Web services would alleviate existing problems with shopbots and comparison shopping sites in several ways.
Expanding the Marketplace
Through the use of Web services, Shopbots could retrieve various product attributes well beyond price. Expanding the comparison attributes could in turn motivate more vendors to provide pro-grammatic interfaces to their sites, thus increas-ing their accessibility to shopbots and their visibility to customers. Although this inevitably increases competition, it also improves overall market efficiency.
Smaller vendors could also increase their visi-bility and benefit from the large traffic volumes offered through shopbots.14Larger vendors with a
bigger, more loyal customer base might have less incentive to join or allow shopbot access. Still, some will choose to do so to further increase their customer base. In any case, shopbots and compar-ison sites would be liberated from dependencies on specific vendors and would thus be better able to offer impartial advice.
Protecting Users
Shopbots can facilitate Web anonymity by proc-essing product information requests without revealing users’ identities. This would make it dif-ficult for vendors to run price discrimination schemes. Such schemes depend on the vendors being able to identify users and build profiles which they can subsequently use to measure a product's desirability to specific users.
Shopbots can also use reputation systems to fil-ter out unreliable vendors. Such systems collate information on vendors by aggregating feedback on past transactions from users (or agents), and then use various means to measure vendor trust-worthiness.15Shopbots might be able to steer users
Comparison Shopping Sites
C
omparison shopping sites don’t typically use agent technology. Usu-ally, they either rely on vendors to provide required information on catalogues and products or operate as meta search engines and search vendor sites. Sites like MySimon (www.mysimon.com) and DealTime (www. dealtime.com), for example, collate product catalogues of vendors who pay for site listings.Meta search engine sites, such as Kayak (www.kayak.com) and Side-Step (www.sidestep.com), can have formal or informal vendor relation-ships. Vendors can simply let such sites access and retrieve information from their Web sites. Or, they might form explicit partnerships and pay a commission either for each hit resulting from the listing or for sales result-ing from clickthrough purchases.Vendors can also place advertisements on comparison shopping sites.
Froogle (http://froogle.google.com), Google’s product search engine, indexes products from multiple e-commerce sites in two ways:
• sellers can submit product information directly; or
• Froogle uses Google Web searches to identify Web pages that offer the target products and services.
Clearly, comparison shopping sites might not offer impartial advice or the best deals. In most cases, comparison shopping sites don’t retrieve and evaluate attributes such as delivery options, warranty, or other value-added services; users have to access the online stores directly to retrieve and compare such information.
away from dubious vendors and help tackle market fraud. Finally, users could indirectly interact with shopbots through personal agents. For example, users could delegate product-finding tasks to their personal agents, who could in turn contact shop-bots and request information. Personal agents could then filter the information returned by shop-bots even further by considering additional user preferences or other restrictions.
Assisting Business-to-Business Transactions
In addition to benefiting B2C users, shopbots can assist organizations in business-to-business (B2B) transactions. A shopbot can perform searches and comparisons for suppliers and components on behalf of businesses and organizations, such as those in a supply chain. Shopbots can locate the cheapest suppliers or those that satisfy certain con-ditions, such as delivery deadlines. B2B users would receive the same benefits from the Web ser-vices infrastructure, including reduced costs and improved market efficiency.
M
y colleagues and I are currently working on two shopbot-related problems. We are apply-ing graph network analysis to the problem of auto-matic Web-service discovery and composition. We are also exploring how we can use the OWL-S ser-vice profile description to enhance semantic matchmaking. In future work, we aim to investi-gate the problem of reasoning with multiple ontologies and ontology consolidation services.In the years to come, using the Semantic Web and Web services will lead to the development of increasingly sophisticated shopbots. In addition, given that mobile devices such as smart phones and PDAs have become indispensable to many people’s daily lives, the next step will be to devel-op shdevel-opbots that can devel-operate on these Web-enabled devices.
Acknowledgments
The discussion on the problems associated with current shop-bots is derived from my book, Agent Technology for E-Commerce (forthcoming from Wiley & Sons, 2007). Reproduced with permission.
References
1. Y. Wan, S. Menon, and A. Ramaprasad, “A Classification of Product Comparison Agents,” Proc. 5th Int’l Conf. Elec-tronic Commerce(ICEC 03), ACM Press, 2003, pp. 498–504. 2. J. Edwards, “Is That Your Best Offer? Shopbots Search the
Web for Bargains,” CIO Magazine, 2000; www.cio.com/ archive/110100/et.html.
3. F. Menczer, W.N. Street, and A.E. Monge, “Adaptive Assis-tants for Customized E-Shopping,” IEEE Intelligent Sys-tems, vol. 17, no. 6, 2002, pp. 12–19.
4. V. Rajiv and P. Aggarwal, “The Impact of Shopping Agents on Small Business E-Commerce Strategy,” J. Small Busi-ness Strategy, vol. 13, no. 1, 2002, pp. 62–79.
5. B.T. Krulwich, “The BargainFinder Agent: Comparing Price Shopping on the Internet,” Bots and other Internet Beasties, J. Williams, ed., Macmillan, 1996, pp. 258–263. 6. A.G. Guttman, R.H. Moukas, and P. Maes, “Agent-Mediated
Integrative Negotiation for Retail Electronic Commerce,” Proc. 1st Int’l Workshop on Agent Mediated Electronic Trad-ing(AMET 98), LNCS 1571, Springer, Berlin, 1999, pp. 70–90. 7. R.B. Doorenbos, O. Etzioni, and D.S.Weld, “A Scalable
Com-parison-Shopping Agent for the World-Wide Web,” Proc. 1st Int’l Conf. Autonomous Agents (Agents 97), ACM Press, 1997, pp. 39–48.
8. M.P. Singh, “The Service Web,” IEEE Internet Computing, vol. 4, no. 4, 2000, pp. 4–5.
9. M.P. Singh and M.N. Huhns, Service-Oriented Computing: Semantics, Processes, Agents, John Wiley and Sons, 2005. 10. M. Paolucci and K. Sycara, “Autonomous Semantic Web Ser-vices,” IEEE Internet Computing, vol. 7, no. 5, 2003, pp. 34–41. 11. K. Verma et al., “METEOR-S WSDI: A Scalable P2P Infra-structure of Registries for Semantic Publication and Dis-covery of Web Services,” Information Technology and Management, vol. 6, no. 1, 2005, pp. 17–39.
12. H.C. Wong and K. Sycara, “A Taxonomy of Middle-Agents for the Internet,” Proc. 4th Int’l Conf. MultiAgent Systems (ICMAS-00), IEEE CS Press, 2000, pp. 465–466.
13. W. Kim et al., “Development of a Meta Product Search Engine with Web Services,” Proc. 2nd Asia Information Retrieval Symp. (AIRS 2005), LNCS 3689, Springer-Verlag, 2005, pp. 571–576.
14. G. Iyer and A. Pazgal, “Internet Shopping Agents: Virtual Co-Location and Competition,” Marketing Science, vol. 22, no. 1, 2003, pp. 85–106.
15. P. Resnick et al., “Reputation Systems,” Comm. ACM, vol. 43, no. 12, 2000, pp. 45–48.
Maria Fasliis a lecturer in the Department of Computer Science at the University of Essex. Her research interests are in agents and multiagent systems — particularly formal theo-ries for reasoning agents, group formation, and social order — as well as in applying agent technology in e-commerce. Fasli has a PhD in computer science from Essex Universi-ty. She is the author of Agent Technology for E-Commerce (forthcoming from Wiley and Sons, 2007) and was award-ed a Higher Education Academy National Teaching Fel-lowship in 2005 for her innovative approaches to learning and teaching. Contact her at [email protected].