March 2017
By Matthias Peissner and David Blank

Dr.-Ing., Dipl.Psych. Matthias Peissner is head of the Competence Center "Human-Computer Interaction" at the Fraunhofer IAO in Stuttgart, Germany. His inter-disciplinary team is composed of engineers, computer scientists, designers and psychologists works on new technologies, methods and design ideas for human-machine interactions.



David Blank (MA) is research assistant at the Competence Center "Human-Computer Interaction" at the Fraunhofer IAO. He holds a degree in Communication Design and is enthusiastic about human-centered design of HMIs, great design concepts and innovative ideas.

Joint project HMI 4.0

In a trend study, the Fraunhofer IAO identified and analyzed the most important fields of action for successful human-machine interaction in production of the future. The focus of the study was on questions around ergonomic HMI design as well as the integration of new technologies such as social media, interaction and recognition technologies.

We started the joint project "HMI 4.0" to implement insights from this study in practice. Manufacturing companies, machine and system producers, system integrators as well as technology and software producers jointly researched and formulated the topic "HMI 4.0" under the scientific project management of the Fraunhofer IAO, starting with the topic "user-defined operating help". The project was supported by users and experts of Daimler, Broetje-Automation, Oerlikon, Copa-Data, Getac and Siemens. It helped to consider the topic from various angles and to develop a system that supported users in successfully processing a malfunction and making knowledge accessible and useful for others.



From users to users

A research project of the Fraunhofer IAO in Stuttgart, Germany, is studying how a user can independently create user assistance. This should reduce malfunctions in machines and systems. How is this user assistance created and how can it supplement technical documentation?

The sooner malfunctions can be recognized and handled, the more productive machines and systems can operate. The challenge lies in the large number and range of possible malfunctions. Instructions for resolving and avoiding such malfunctions usually only covers standard situations. However, many malfunctions can be resolved only by very experienced employees or users. The “human sensor” is unbeatable. This is also made clear by the exchange in the consortium of the joint project HMI 4.0. Experience shows: Users hear whether a tool is worn out or wrongly set, and they can execute tasks without errors only after year-long training with a lot of intuition. They know “their” machine and usually know very quickly what is happening and where to start looking for the error.

A lot remains undigitalized

Documentation of errors and transfer protocols continue to exist in old calendar books. Employees talk with one another about what has occurred and how the problem can be resolved in their opinion. In shift operations this leads to some being well informed, while others are ill informed. Moreover, an evaluation of this data is practically impossible.

User records the solution

In the future, the user of a machine or system who has successfully solved a malfunction will document it and describe the way to the solution. The entries and relevant sensor data shall be collected and consolidated in a self-learning system. The data, documentation and instruction are used as preventive measures to avoid standstill. If a known or a similar malfunction still occurs, the documentation generated by the users serves as help for the operation and is provided to other users.

Optimal operating help

Malfunctions can be resolved quickly only with well-designed operating information. The structure of instructions as well as the media used play a major role and can be adapted to the operations and situation. To obtain initial insights, we asked test persons at the Fraunhofer IAO to follow the instructions of a repair manual for a smartphone. Some basic structural elements could be identified through this experiment, which were included in the operating help and its creation.

From the correct perspective

To ensure that operating help doesn’t miss its purpose, it is important to address the user from the user’s perspective and always from the actual situation to the result or, even better, to an intermediate result. The view of the result serves as a control. However, many users can also conclude the steps from a target situation and do not need any instructions to get there. If the images are created from another perspective, the user must make a transfer. For example, our experiment showed that the participants always turned to the instructions in the image perspective, to make the correlation of the instructions to reality simpler. The smartphone was turned again to the position that worked best for implementing the step.

Instructions that cannot be skipped

Safety instructions, particularly when marked as such, appear to animate the user into skipping them. It is essential to have a mechanism in place for such important instructions, so that the information cannot be skipped without further action.

There is a logical structure for operating help (Fig. 1). It also applies to its creation. For example, the structure aims to avoid typical situations such as forgetting spare parts or tools. The user should be able to check at the start, whether the operating help really describes his problem and which alternative causes of error are possible for selection (Fig. 2).

Figure 1: General presentation of the instruction structure.
Source: David Blank and Matthias Peissner


There can be more reasons for an error. The system operating help guides the user from the most probable to the most improbable cause of the error. For example, there is a 60 percent chance that the cause for a machine failure is due to a sensor signal and a 40 percent chance that it is due to a loose sensor cable.

Figure 2: The steps for action within the structure of the instructions.
Source: David Blank and Matthias Peissner


Cultural differences

In western countries, it is helpful to communicate the reason for the error and the target to the user. Then he can check whether the system is suggesting the right solution or whether he can think of a better and perhaps quicker solution. In turn, he can document this new solution. After several studies, it is possible to state that a particularly detailed representation makes no sense in other countries, and the instructions can be leaner.

Modular structure

The Fraunhofer IAO has compiled, analyzed and categorized modules for instructions from various sources and reviewed and enhanced them jointly in the project consortium. This also includes possible variants of layouts and processes for such an application with the respective pros and cons of input and output. Other information includes the various interaction technologies, when they should be used, what must be considered for specific types of users and naturally the various media that can be used. There are now over 50 modules that have been described together with the sources.

Use of media

Is an animation better than a photo or a line drawing? Is a video eminently suitable? Are icons better than text? The use of media can be argued ad nauseum. Various perspectives need to be considered. Photographs, although a problematic task due to reasons of security and data protection, are not an unsolvable issue. It could be much more difficult technically. Which device should be used to take the photographs and how can they be incorporated in the system quickly and easily? Mobile end devices are recommended for that.

Assuming that mobile devices or existing operating panels of the machine are used, the following media can be put to use:

  • Interactive 3D models or renderings for marking the error location
  • Icons for categorization and documentation
  • Animations for highlighting mechanical machine functions for safety
  • Photos (or video) with optional manual marking for the documentation
  • Text as annotation or for instructions

This does not mean that these elements cannot be used differently or do not make sense otherwise. Anyone who wants to use animations, should do so. However, it is then necessary to overcome the challenge that the user will view the animation multiple times, to understand it. That is the case when the animation is repeated automatically. The user will view the animation multiple times, so that he can grasp the start and end point and its meaning. However, this takes time. Animations are most useful, when the machine has moving parts, for instance. It can also be about blinking LEDs, so that the user makes a direct connection.

When using video, it is necessary to consider that it is divided in individual short steps and it is easy to navigate between them, e.g. to repeat a step. The creation of such a video with the corresponding markings is more difficult than just taking photographs. However, it is easier to work with a video, when the user cannot interrupt the sequence of the solution for the malfunction. In contrast, purely audio recordings are unsuitable for operating help.

Creating operating help

The effort for creating operating help must be as low as possible. It must be possible to compile the most important information with a few actions. Drag-and-drop, multi-modal inputs through gestures or speech for instance, graphical documentation forms such as CAD objects and the use of sensors for partially automated creation are just a few options. Moreover, it is necessary to note which information is preset. If the user uses a dropdown field with the preset categorization “Other errors” for instance, this category is registered surprisingly often.

A key component of the operating help is the location of the error (Fig. 3). All information, e.g. time, sensor data and the instructions are linked to it. The location of the error is the core of the problem and is marked by the user. A simple solution places a dot on a photograph of the machine. A better solution, and therefore to be preferred and technically possible is to select the component in an interactive 3D model and fixing the error to the malfunctioning object. The necessary sensor data can be automatically retrieved, saved and analyzed. This process of using the location as the first key point is a part of some new applications for instructions and documentation.

Figure 3: The user marks the location of the error.
Source: David Blank and Matthias Peissner


After the user has defined the location, he is guided step by step through the necessary details. The system developed by the Fraunhofer IAO allows itself to be adapted to requirements and rules. If no photographs are allowed, for instance, this module can be deactivated and the user is not asked for it. This modularity should offer an adequate solution without violating security guidelines.

Scalability for further scenarios

The developed system is scalable and thus functions for users from the most varied fields. We have taken care to design the system such that its use is conceivable even outside machines and systems. A complete new design is not required, an adaptation can be implemented with already available and industrially implementable means. The application for operating help can be used on a panel, but also on a smartphone or tablet.

Increase motivation

The success of user-generated operating help greatly depends on the motivation of the users to create operating help. We could not confirm any issues with regards to users not wanting to share their knowledge to secure their job. The users showed interest in this system and considered the advantages for themselves in the forefront. At several places our studies showed that such a system for exchange of knowledge is motivation enough, when it is designed to be responsive and simple.  The summary of the statement of surveyed users say: "Excellent, then we can help each other mutually with it" and "Wow, this would be great in case I don’t know how to move on". Such an aid fulfils the need for safety and competence particularly well.

The documentation section is mandatory (Fig. 4) and the instruction section is voluntary, to increase motivation and overcome possible hurdles. The mandatory section is designed such that only few inputs are required and everything can be done through drag and drop.

Figure 4: The user must complete a mandatory section, but can provide action steps voluntarily.
Source: David Blank and Matthias Peissner


Games as role model

Means such as "Gamified Design" or even "Gamification" are great motivators for the voluntary section of instructions. The method uses gaming elements or other measures that promote motivation. We tested gamified design principles using a prototype and a survey at an automobile supplier and received very positive feedback.

The tested prototype allowed the user to compile instructions, for which he obtains a reward in the form of a specific number of screwdrivers. There were other rewards in appropriate context for especially good instructions. The user could tune his car in a virtual garage, purchase a better engine or other rims for the points. Everyone could identify the performance of a colleague based on the car. But it can also be small things to increase motivation or satisfaction. So, there was a small function that enabled thanking an employee by virtually giving him a headlamp flasher.

Play mechanisms can be transferred to any context. The topic “Car and Garage” became a garden in the final draft (Fig. 5). Every motive can be used basically. The fact that people are ready to overcome the unknown or the unpleasant was visible from the example of PokemonGo. People, who hardly leave their homes, were suddenly seen in the woods on a ten-kilometer march to hatch eggs.

It remains to be said that the objective is not to develop a game to be played during work. Rather, known positive effects are used. Collecting points for fulfilling tasks or learning takes place during work. The use of points, to design the car or garden for instance can easily take place during free time on the private smartphone.

Figure 5: A user is rewarded for his engagement. The result is also visible to the rest of the participants.
Source David Blank and Matthias Peissner


Cultural differences

Motivation, work and duty are perceived differently in different cultural groups. Some studies, mainly for the Asian region, were analyzed and the most important facts were compiled for the consortium. For example, an intrinsic motivation can be worked on, financial incentives can be offered or things can simply be ordered, i.e. made mandatory depending on the cultural group.

Future user groups

Communication is fun and easy, when it is simple and well-designed. This can be observed from messenger services such as Telegram, Threema or WhatsApp. Many people, who had major difficulties in operating their smartphone earlier, suddenly require mobile Internet and have apparently learnt to operate their device without problems overnight. To have another facet to life, mainly when it is easy, is a sensational motivation. For this reason, the system that has emerged in the consortium is broadly designed like a messenger (Fig. 6). The user communicates very easily with the machine and his colleagues. The machine can even answer, when it is asked for its status.

Figure 6: The status of the machine can be queried, the information on error messages can be entered and forwarded using a smartphone.
Source: David Blank and Matthias Peissner


Chance for knowledge management

The system should enable users to share their knowledge on specific error situations without great effort and thus contribute to user generated operating help. Moreover, it should ensure the scope and accuracy as well as the didactic quality of the operating help and motivate employees to share their knowledge. But who would be responsible for maintaining such a system consistently and ensuring high quality?

The advantage of such a service results mainly from a volume of information. Conclusions can be drawn only after a certain amount of sensor data, documentation or instructions are available. Either a company possesses many machines of the same type or it must share the information or parts of it with others, who have the same or similar machines. Confidential data such as special machine settings and recipes are the greatest problem here, immediately following the security while networking an entire production system. The advantages outweigh in this case, so that means and ways are found to develop a database that only uses the necessary data anonymously and is secure at the same time.

One of the options is the reconciliation of data from time to time through a special computer. One alternative is a service that is not directly connected to the production plant, but received information over another path. This is a very exciting topic, which must be defined more exactly and discussed, especially because it concerns all parties, from the employees operating the machines or systems to the works council, the trade unions, to the board and across company boundaries the competition as well.


Make the invisible visible

No one should fear starting this exchange, rather the focus should be on ensuring that it functions and does not become too complicated. The technology was one of the major hurdles till now. Technologies are becoming more economical, sensors are installed almost everywhere, a digital network is usually part of standard.

A "mixed reality" was a dream at the beginning of the joint project. Somewhere down the line it became reality, but required high amount of computing and was expensive. With the HoloLens, a device entered the market, which enabled marking information on a 3D model as well as on the real object itself, and without major developmental effort at that. That was the future.

Cloud services and developments such as the internet of things enable even small operations to upload their data and get it analyzed. This does not require expensive and complex infrastructure requiring maintenance anymore. Such services can soon be offered and used by anyone. The computing services of neuronal networks and the capability to learn will help to identify errors in advance. Humans will remain as sensors and creative decision-makers, but will also take up another role.

The project continues

The project "HMI 4.0" continues and the first achievements are being developed further. The topic of instructions and digital documentation will be present at several places in the Future Work Lab at the Fraunhofer and in the Arena 2036, e.g. with the HoloLens of Microsoft. There are many interesting topics, the next of which will start soon.