WebCrawler

The WebCrawler Project began as Brian Pinkerton's research project at the Department of Computer Science and Engineering at the University of Washington in Seattle. The project has since moved to America Online, where it continues to operate as a freely available Internet search tool. A short history of the WebCrawler tells the story in more detail.

The aim of the WebCrawler Project is to provide a high-quality, fast, and free Internet search service. To carry out this mission, our team is involved in a wide range of activities, from operating the service itself to publishing research in Internet resource discovery to helping build Internet standards. As we build the infrastructure for the next-generation of search tools, we remain committed to offering the highest quality service possible.

A Brief History of WebCrawler

The WebCrawler's history is a story about how the rapid rise of the Internet transformed a University of Washington research project into a successful commercial product. Since its inception as a research tool, the Internet has seen many similar stories whose collective message is clear: research and development activities can have a profound effect on the direction our society takes.

In early 1994, students and faculty in the Department of Computer Science and Engineering gathered in an informal seminar to discuss the early popularity of the Internet and the World-Wide Web. Students typically try out their ideas in small projects in these seminars, and several interesting projects were started. The WebCrawler was Brian Pinkerton's project, and began as a small single-user application to find information on the Web.

Fellow students persuaded Pinkerton to build the Web interface to the WebCrawler that became widely usable. In that first release on April 20, 1994, the WebCrawler's database contained documents from just over 6000 different servers on the Web. The WebCrawler quickly became an Internet favorite, receiving an average of 15,000 queries per day in October, 1994 when Pinkerton delivered a paper describing the WebCrawler.

The Web continued its exponential growth, and before long the WebCrawler needed real resources to keep up. Dealernet, a local Seattle company, became WebCrawler's charter sponsor by donating a new server to the effort. In return, their logo was displayed on the WebCrawler's "Sponsor" page as a supporter of the WebCrawler research effort. The WebCrawler moved to Sieg Hall.

At nearly the same time, Starwave, a departmental affliate, became the second sponsor by donating a second machine and by funding Pinkerton to focus his time on the WebCrawler effort. In doing so, Starwave was strengthening its ties to the department and funding work that had increasing relevance to their business. During the WebCrawler's early development, Pinkerton had been working as a research assistant in the Department of Molecular Biotechnology and supporting the WebCrawler at night and on weekends. With the Starwave funding came the understanding that Pinkerton would focus on the WebCrawler as the basis for his thesis.

Growth is generally considered a good thing. But too much of a good thing can be difficult to manage. In January, 1995, WebCrawler became the heaviest consistent user of CSE's network. In February, major changes to the service were necessary to accommodate the new load. In March, the daytime load was so high that the service was usable only at night. The necessity of funding this effort with real money was obvious; the difficultly was timing: WebCrawler needed resources soon, or the increasing load would kill it.

The WebCrawler project was run by a grad student with just enough faculty supervision to ensure that the work could support a thesis. Who would fund a service whose goal was providing a search service to a million Internet users throughout the world? If outside sponsors funded the effort in exchange for what amounted to advertisements, could the service continue operating at the University? In the end, the service was sold to America Online, Inc. to become a commercially operated and supported engine. It moved from the University to its new home in San Francisco on March 29, 1995.

WebCrawler was the first full-text search engine on the Internet. Several competitors emerged within a year of WebCrawler's debut: Lycos, InfoSeek, and OpenText. They all improved on WebCrawler's basic functionality, though they did nothing revolutionary. WebCrawler's early success made their entry into the market easier, and legitimized businesses that today constitute a small industry in Web resource discovery.

In November of 1996, WebCrawler was acquired from America Online by Excite, Inc. WebCrawler is now one of two flagship brands of the Excite Network. The Excite Network consists of four leading Web brands, Excite, WebCrawler, City.net, and Magellan.

Today WebCrawler is one of the most popular search services on the Internet. WebCrawler was the first search engine and WebCrawler is still the best search engine for consumers. With the recent launch of WebCrawler Shortcuts, WebCrawler continues to be an innovator in the competitive search engine space. WebCrawler will always be the place to go to find just what you're looking for on the Web.