Lumiere Project: Bayesian Reasoning for Automated Assistance
Decision Theory & Adaptive Systems Group
Redmond, Washington 98052-6399
The Lumiere project at Microsoft Research was initiated in 1993 with the goal of developing methods and an architecture for reasoning about the goals and needs of software users as they work with software. At the heart of Lumiere are Bayesian models that capture the uncertain relationships between the goals and needs of a user and observations about program state, sequences of actions over time, and words in a user's query (when such a query has been made). Ancestors of Lumiere include our earlier research on probabilistic models of user goals to support the task of custom-tailoring information displayed to pilots of commercial aircraft, and related work on user modeling for the decision-theoretic control of displays that led to systems that modulate data displayed to flight engineers at the NASA Mission Control Center. These projects were undertaken while several of us were based at Stanford University.
The first Lumiere prototype was completed in late 1993 and became a demonstration system for communicating with program managers and developers in the Microsoft product groups. Lumiere development has continued. Later versions explored a variety of extensions including richer user profiling and autonomous actions.
Early on in the Lumiere project, studies were performed in the Microsoft usability labs to investigate key issues in determining how best to assist a user as they worked. The studies were aimed at exploring how experts in specific software applications worked to understand problems that users might be having with software from the user's behaviors. We also sought to identify the evidential distinctions that experts appeared to take advantage of in their reasoning about the best way to assist a user. Such information was partly obtained from protocol analysis of videotapes and transcripts of the thoughts verbalized by users as well and of the experts that were trying to assist these users in a number of Wizard of Oz studies.
A cognitive psychologist running a study at the Microsoft usability labs. Usability studies have played a significant role in the Lumiere project. More generally, over 25,000 hours of usability studies were invested in Office '97.
The Lumiere Wizard of Oz studies helped to elucidate important distinctions that were later woven into Bayesian networks. Later, usability studies were employed to test the actual performance of the Office Assistant, and user reactions to different versions of the interface.
The Lumiere prototypes have explored the combination of a Bayesian perspective on integrating information from user background, user actions, and program state, along with a Bayesian analysis of the words in a user's query. A Bayesian methodology for considering the likelihoods of alternative concepts given a query was developed at Microsoft Research, in collaboration with the Office product group in 1993. This Bayesian information-retrieval component of Lumiere was shipped in all of the Office '95 products as the Office Answer Wizard.
The earliest Lumiere prototypes combined an analysis of actions and words when they are available. Lumiere continues to monitor events with a event system that combines atomic actions into higher-level modeled events. The modeled events are variables in a Bayesian model. An event language was developed for building modeled event filters. As a user works, a probability distribution is generated over areas that the user may need assistance with. A probability that the user would not mind being bothered with assistance is also computed. The figure below shows a snapshot of Lumiere's reasoning about a users needs.
A snapshot of the Lumiere system showing some of the instrumentation behind the scenes. The multicolored bargraph represents a probability distribution about the needs of the user. The large, red bar indicates the system's computation of the likelihood that the user would like being notified with some assistance.
Folding in the consideration of words in a user's query. The probability distribution inferred about a user's needs is revised with the consideration of words in the user's query.
During the course of Lumiere research, we have considered alternative user-interface metaphors for acquiring information from users and for sharing the results of Bayesian inference with users. Beyond experimenting with embedded actions and traditional windowing and dialog boxes, we have been interested in character-based interfaces as a way to provide a natural way to centralize assistance services. We built some animations in the early days of Lumiere to demonstrate basic interactions with a Bayesian-minded character (you can view some frames from an early demonstration animation...).
The first phase of porting Lumiere to the real world occured with the completion of the Office '97 product suite. Office committed to a character-based assistant. Users can choose one of several assistants each of whom have a variety of behavioral patterns--all of whom draw their intelligence from Bayesian user models.
One of several "assistants" available in Office '97. The Office assistants provide a focus for event and query based interaction centering on assistance with the use of the software. This character is named "Genius."
Lumiere technology goes live in Office '97. The Office Assistant in Office '97 monitors user behaviors and assists users. Models were built for each of the Office applications.