Machines that understand how humans are feeling

Fraunhofer scientists have developed a diagnostic tool that recognises the state of the user in real time and communicates this information to machines. The technology is useful for situations where a machine is designed to negate the poor human performance or error. In order for it to intervene appropriately, it needs to have capacity to understand what is going on with the user.

Jessica Schwarz, from the Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE in Wachtberg, has undertaken a doctoral thesis that examines the question of how to precisely determine user states. In addition, she looked at what influence a person’s state may have on incorrect behaviour and how automated systems can make use of this information.

“For complex applications it is not sufficient to focus on only one impact factor,” said Schwarz. For instance, an increased heart rate does not automatically mean that a person is stressed, as this can have various causes. Schwarz therefore examined which factors specifically impact human performance and created a holistic model that provides a detailed view on user states and their causes.

In her model, she differentiates between six dimensions of user state that impact human performance: workload, motivation, situation awareness, attention, fatigue and the emotional state. She uses physiological and behavioural measures to detect these states and combines them with external factors such as task, environmental factors, current level of automation and time of day, as well as individual factors such as experience.

“This allows us to assess the user’s state in more detail and also identify causes for critical states,” Schwarz said.

The doctoral student also verified her theoretical findings in experiments. Test subjects were required to assume the role of an air traffic controller and steer simulated aircraft safely through a virtual airspace. As stress factors, the number of aircraft was increased, the instructions of the ‘controllers’ were ignored and background noise was added in some conditions. EEG sensors on the head, an eye tracker and an ECG chest strap recorded physiological changes in the test subjects.

“We had previously conducted intensive interviews with real air traffic controllers to enable us to reproduce their challenges with man-machine interfaces as accurately as possible,” said Schwarz.

A diagnosis interface was then created that detects in real time when individual impact factors become critical and communicates this to the machine, allowing it to react accordingly.

The research project is due to be completed by the end of the year, with the technology now close to application. Potential fields of application include all highly automated tasks where critical user states can be a safety issue.

Image credit: ©iStockphoto.com/Vladislav Ociacia