Introduction

Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR), or inclusive as Extended Reality (XR), describe immersive technologies that can merge the physical and virtual worlds. Physiological measures provide a quantitative evaluation of user response to certain stimuli in these computer- generated synthetic environments, or “virtual environments”, and can provide a feedback loop to significantly improve user experience and performance in such environments.

This presentation will explore how biofeedback technologies and approaches are being used within AI enabled XR systems for training and/or educational applications. The biosignals explored provide insights into physiological and/or emotional processes in users/participants. The presentation will discuss how biometric feedback is used in XR technologies, with a focus on specific biofeedback sensor use in the context of simulation training and education, concluding with recommendations about how biometrics can appropriately be used in XR synthetic training systems.

Method

A systematic review was conducted following the PRISMA guidelines looking at ‘What biofeedback technologies and approaches are being used within AI enabled XR systems for training and/or educational applications?’. An initial scoping search of existing research revealed many theoretical and untested-on- human concepts and approaches. From this scoping search, the key inclusion criteria for this literature review were defined. Of importance is that the included biofeedback enabled systems are validated on real human participants and therefore provide actual experiment results. A total of 803 studies were identified for screening and post evaluation, 48 met the search criteria and were included for analysis.

Results:

A total of 11 different biosignals were captured across the different studies. When considering the use of biosignals by primary measure, stress made use of the most individual biosignal types. Electrodermal activity, also referred to as galvanic skin response (GSR), was the most widely used biosignal, with most application in the measurement of stress, cognitive load, and emotions. Eye tracking was also frequently used in the measurement of stress, cognitive load, and attention.

A total of 42 different biosensor devices were identified in the resultant studies. These sensors covered a range of different implementation approaches, from purpose-built integrations, experimental lab setups, and implementations using commercial off-the-shelf equipment.

Each of the biosignals in the studies enables biofeedback mechanisms and can form the basis of AI/machine learning approaches to remove artifacts, process signals into usable data, and/or identify patterns. The biosignals themselves provide insights into physiological and/or emotional processes in users/participants.

Conclusion & recommendations

This paper summarises current research implementing XR technologies in combination with biofeedback and AI approaches, with a focus on the specific biofeedback sensor use in the context of simulation training or education contexts. Several important recommendations emerge from the research, including:

A minimum of two biosignals should be captured where possible, and thus devices that capture multiple biosignals are preferred.

The use of simpler sensor technologies and associated measures is preferred to limit the impact of movement artifacts and maximise reliability of data. Wristband devices present as particularly useful devices for biofeedback implementations in XR simulation training applications.

Given the dynamic innovation occurring in biosensing technology, implementations of biofeedback enabled XR synthetic training systems should focus on identifying appropriate biosignals and actuation of biofeedback in virtual environments and tasks. A “plug and play” approach to sensing technology is recommended, allowing sensing technologies to be updated/upgraded overtime while the fundamental benefits of the biofeedback implementation are retained. As such, details regarding integration of biosensing technologies with synthetic environment development tools (ie. game engines) should be a focus of development approaches.