Digital Engineering Building blocks for Augmented Reality
Using a novel software made in Bochum, paper-based work instructions can be easily translated into Augmented Reality applications. No prior knowledge necessary.
Following the hype surrounding the mobile game Pokémon Go, most people have become aware of Augmented Reality (AR). AR applications enhance reality as it is viewed through a camera lens, for example on a phone display. The camera image is overlaid with additional digital information that do not exist in the real world. While it is only a bit of fun in the case of Pokémon Go, AR might be used to render many applications in everyday life much simpler. In mechanical engineering, for example, thick maintenance and repair manuals for complex equipment could become a thing of the past.
Our goal is to render Augmented Reality usable for everyone.
Mario Wolf
At present, however, it takes an expert to create the complex AR applications. “Our goal is to render Augmented Reality usable for everyone,” says Dr. Mario Wolf from the Chair of Digital Engineering at RUB. Together with Jan Luca Siewert, he has developed a tool that users without any prior knowledge can utilise to generate their own Augmented Reality apps. Siewert had paved the way for this technology in his Master’s thesis: “Mechanical engineering students must operate complex equipment and they are supplied with thick manuals for this purpose,” he explains. “More often than not, they stand in front of the unit and don’t know where to start. Our intention was to provide the supervisors with a tool for translating paper-based work instructions into AR manuals that explain step-by-step how to proceed.”
Adding instructions via tablet
Lecturers don’t have to know anything about AR, image processing and similar techniques in order to create their first digital manual. They start creating the manual using building blocks on a PC or a tablet. “One way to do it is to add an arrow in a certain spot and a hazard symbol in a different spot, in order to indicate that this is where the machine gets very hot and so on,” elaborates Jan Luca Siewert.
The Augmented Reality manual is created in two easy steps: the groundwork is laid using a web application. While the supervisor translates the hardcopy into digital instructions, the software suggests various icons and text boxes as well as the option to upload photos and PDF files. Subsequently, the supervisor can walk with a tablet in camera mode to the respective unit and indicate where an instruction is to be placed by tapping on the display.
As a result, students are supplied with a tablet rather than a thick paper manual. As they view the equipment through the tablet’s camera, the software automatically displays what they have to do in which order. An application like this is useful not only for teaching purposes, but also everywhere where complicated hardcopy manuals are necessary to operate complex machinery.
Tablet navigates automatically
Throughout the entire process, the tablet navigates the space automatically. All the user has to do is define a zero point at the beginning. To this end, he prints an image marker on a DIN-A4 sheet of paper and puts it up somewhere near the unit. The software recognises that marker automatically to be the zero point as soon as the camera points at it. Subsequently, the user can move freely across the room in order to compile his AR manual. Via motion sensors installed in the tablet, the program knows its position in the room automatically – even if the image marker is no longer visible to the camera. In addition, image processing algorithms are deployed to recognise and track distinctive points within the environment.
Problems occur only if the room is dimly lit or if smooth surfaces generate strong reflexions. Large, monochrome surfaces, too, interfere with orientation, as they don’t provide any distinctive markers for the algorithms to navigate by. “Under real-world conditions, our software can position information with an accuracy of a few centimetres,” says Mario Wolf. “This is enough for our purposes.”
Based on existing algorithms
Jan Luca Siewert and Mario Wolf have created the app based on a software development kit for AR applications. Such software packages provide algorithms and methods for identifying and entrenching objects in real life, for example for the purpose of navigating a space or for image recognition.
Currently, the Bochum-made software is running on the Apple I-Pad. This product offers the best image recognition algorithms and navigation features, according to the researchers. “The drawback is that we cannot distribute the app easily. Interested users have to visit us and we have to transfer the program to their device via cable,” points out Mario Wolf. In order to be easily distributed, the app must first be approved by the Apple App Store – but it is still in development.
Usability study
In the next step, the team at the Chair of Digital Engineering is conducting a usability study together with the chair of Materials Discovery and Interfaces under direction of Professor Alfred Ludwig, in order to find out which additional features are still required and to continue optimising the software. Mechanical engineering students as well as students from other faculties will be taking part in the test. Today, the app is being used at various RUB institutes to create operation manuals for complicated research equipment. The objective is always to facilitate application. “The trend in Augmented Reality is to move away from complex 3D animations that look flashy but serve only little purpose,” says Mario Wolf. Rather, AR should be rendered usable for the general public; the Bochum team has already taken the first steps in that direction – and has been contacted by interested potential users from the industry.