Predicting the efficacy of simulator-based training using a perceptual judgment task versus questionnaire-based measures of presence

Abstract

The quality of a virtual environment, as characterized by factors such as presence and fidelity, is of interest to developers and users of simulators for many reasons, not least because both factors have been linked to improved outcomes in training as well as a reduced incidence of simulator sickness. Until recently, most approaches to measuring these factors have been based on subjective, postexposure questioning. This approach has, however, been criticized because of the shortcomings of self-report and the need to delay feedback or interrupt activity. To combat these problems, recent papers on the topic have proposed the use of behavioral measures to assess simulators and predict training outcomes. Following their lead, this paper makes use of a simple perceptual task in which users are asked to estimate their simulated speed within the environment. A longitudinal study of training outcomes using two of the simulators revealed systematic differences in task performance that matched differences measured using the perceptual task in a separate group of control subjects. A separate analysis of two standard presence questionnaires revealed that they were able to predict learning outcomes on a per individual basis, but that they were insensitive to the differences between the two simulators. The paper concludes by explaining how behavioral measures of the type proposed here can complement questionnaire-based studies, helping to motivate design aspects of new simulators, prompting changes to existing systems, and constraining training scenarios to maximize their efficacy.