My Humble Abode

Wizard of Oz studies are a fun and valuable type of user experience (UX) research you can do to get a sense of how your idea may work with the target audience before having an actual prototype, especially for more complex technology (e.g., voice assistants, chat bots, or adaptive systems). The main concept behind conducting a Wizard of Oz study is that, while the participant believes they are interacting with a fully automated system, there is actually a “wizard” behind the scenes controlling the responses or actions of the system. This allows you to observe how users will interact with your prototype, and to how it will feel real to them, without having to spend time and money creating a fully functional prototype.

What Is a Wizard of Oz Study?

The Wizard of Oz study is a usability test in which an observable interface appears as if it is independently functioning, however, the participant is really interfacing with a researcher that is controlling the predictive function of the system behind-the-scenes.

The Wizard of Oz method can be particularly beneficial for testing early stage concepts related to artificial intelligence (AI), natural language processing (NLP), or any complex automated solution that has not yet been fully developed.

The name Wizard of Oz describes this type of testing because, like the character behind the curtain in the movie The Wizard of Oz, the user experiences the prototype as though it were an independent functioning prototype rather than just a researcher using the system as a proxy for the functionality of the prototype they are testing.

Why Use This Method?

Validate Your Idea Early

• Determine whether your idea is desirable before investing time and resources into developing the engineering aspect of it.

Test Natural User Behavior

• Observe how users naturally request things, what kind of feedback they give the system, and what their reactions are to limitations within the system.

Iterate Quickly

• Change the way that the wizard behaves or responds on the fly to try out alternative design strategies.

Examples Using Wizard of Oz

1. Research on the Voice User Interface

Researchers have developed an experimental system to study the natural language people use when giving a voice command. This involved creating a fictional system that provided responses through a wizard instead of using a speech recognition engine. Users were given tasks such as making calendar appointments or requesting directions to test out how they gave their commands. The results show that users’ spoken commands can vary significantly from what would otherwise be considered a more formally scripted command. The results of this study are intended to help guide the design of the real speech interface.

2. Chatbot Communication

In another study, researchers utilized a wizard-of-oz technique to simulate an automated customer service chatbot and have people interact with it via text, believing it to be fully automated. Although there was a wizard behind the scenes creating responses to user inputs, the results show that people think chatbot messages have personality and intent. Therefore, when creating automated messages for use in customer service chatbots, designers must use an appropriate tone and communicate clearly.

3. Adaptive Navigation Interfaces

Researchers used the WoZ methodology to simulate how an adaptive navigation interface inside a vehicle would generate new driving directions based on updated traffic conditions and because of the driver’s actions or behavior. The study provided insight into how drivers respond to unexpectedly alternate directions and what type of driving directions or suggestions seem the most intuitive to them.

Final Thoughts

Using the Wizard-of-Oz method enables designers to obtain early and/or rich data regarding the behaviour of users interacting with a non-existent system. The approach enables a designer to learn the importance of certain features and functionality before making an investment in designing or developing automated processes since a human acts as an automated process.

The Wizard-of-Oz approach is also applicable for testing a voice interface, chatbot or adaptive system, and acts as a bridge between an imagined experience and an actual experience when using a specific interface or system.