A Framework to Conduct and Analyse User Tests in VR Using the Wizard-of-Oz Methodology

Abstract

It bis often challenging to measure participants' reactions during user tests where a Wizard-of-Oz (WoZ) method is applied. This method is applied by the observers to validate functionalities of concepts that are difficult to build. In this study, a new technique is developed using virtual reality (VR) to improve the measurement of the participant's reactions during such a test. In this system, called ExperienceDNA, VR user tests can be monitored and controlled through a desktop interface. In addition, physiological trackers (eye tracking and heart rate monitoring) are used to measure what the participant is looking at and to gauge their preferences. Moreover, the use of VR allows for quick adaptations to the virtual environment the participant is confronted with. In this way, highly versatile tests can be conducted while minimising the initial setup effort. Our approach has been validated by performing a pilot test on a predefined use case. The qualitative feedback collected from analysing batches of data from the pilot test is presented in the results section. In conclusion, this paper covers the development, description and evaluation of the ExperienceDNA framework, as well as some ideas for future improvements to this framework.