MANTIS AR/VR showcases at Finnish industrial events

The virtual reality and augmented reality market is developing fast both in Finland and on a global level. These technologies are emerging markets, especially on the consumer side, and are likely to affect maintenance related work in one form or another. These technologies also have a lot of innovation potential. Virtual reality and augmented reality technologies has been used in the advanced HMI approaches for the Finnish conventional energy production use case. Operation and Maintenance team of the Lapland University of Applied Sciences (LUAS) has made a technical implementation as a part of that use case. Mantis project and the VR/AR demo will be on display at the two different kind of industrial business events in Finland.

Industrial events in Kemi and Oulu during spring 2018

AR and VR demonstrators related to the Fortum use case will be presented at the “Rikasta Pohjoista 2018” (Rich North 2018) event and “Northern Industry 2018” events for industry professionals. “Rikasta Pohjoista 2018” seminar will be held in Kemi on 18th and 19th of April 2018. “Rikasta Pohjoista 2018” is the Operation & Maintenance and Mining industry event, where the theme changes every year. The theme of 2018 is “Reliability & International Mining Industry and Recycling Economy”. The event will bring together over 100 industrial professionals from mining, steel, forest, energy and recycling industry from Finnish companies. The two-day event has been arranged since 2015 at the Lapland University of Applied Sciences. Before that, the event was a one-day event, and only centered on maintenance.

Northern Industry 2018” is the largest and only industrial trade fair in northern part of Finland. It will be held in Oulu at the 23rd and 24th of May 2018.  Event gathers 5000 professionals from mining, steel and forest industry as well as from energy sector, chemical industry and from numerous service providers and equipment manufacturers. The event organizer is Expomark Oy.



Virtual Reality, Augmented Reality as Human Machine Interface

The distinction between industrial maintenance related usage of VR and AR approaches can be roughly defined between factory-floor and back-office usage, where AR is more applicable for factory-floor and on the field maintenance tasks and guidance and as well as the use in maintenance monitoring. VR is inherently more suited for back-office and other office activities such as training and planning. VR is not so handy and applicable being mobile device at the factory floor. Both AR and VR solutions could be utilized as a part of collaborative decision-making. Experts around the world could for example communicate with each other using avatars in a virtual space. AR could be used, for instance locally to observe machinery status. It could allow for new business opportunities in maintenance related support and collaboration.

LUAS first made an AR approach to the use case due to it being more suitable for use in maintenance monitoring on the field and factory floor. The AR approach was done using the Google Tango platform that consisted of a comprehensive Unity compatible AR SDK and a special hardware platform that comprised of an IR dot matrix projector and a special camera capable of measuring the time-of-flight of the independent dots projected onto a shape. The combination of the SDK and the hardware platform was used to mitigate inherent drift in any solely IMU (Inertial Measurement Unit) based positioning solution.

The AR application was named AHMI (Advanced HMI) and it would enable users to create a virtual representation of the flue gas recirculation blower at Fortum’s Järvenpää power plant and retrieve real-world measurement data onto the measurement points attached to the virtual, 3D model. It also supports adding virtual measurement points to real-world objects using real-world measurement data retrieved from MIMOSA database. The measurement data comes from Nome’s sensors that are stored in the common MIMOSA database. The measurement data is retrieved through a REST interface developed by LUAS. Figure 1 presents the 3D model of flue gas recirculation blower placed at a meeting room table.

3D model of flue gas recirculation blower placed at meeting room table
3D model of flue gas recirculation blower placed at meeting room table

The VR application was built solely on the HTC Vive platform, however it could be transferred to other VR hardware platforms such as Oculus, possibly even the Microsoft mixed reality platform. HTC Vive controllers were used at first, but they were replaced with Leap Motion controls, which is a structured light-based IR projection camera system intended for recognizing hand positions and gestures. This was incorporated into the VR demo system to replace the hindrance of the controllers to allow for an immediately more intuitive approaching using the user’s bare hands as a control tool. The VR version creates a virtual world that consists of the same elements used in the AR approach with added VR only features. The VR application could be used to monitor the measurement data and as a training tool for maintenance professionals for example to guide in machine disassembly.

Figure 2 shows the windows displaying real-world vibration data in the VR application.  In the virtual world, the users can open and reposition these windows to their own preferences. They also have a snap functionality that allows for neat alignment of all windows. Users can manipulate these windows naturally using their own hands rendered in VR with the Leap Motion.

Screen capture of the VR demonstrator displaying real measurement data
Screen capture of the VR demonstrator displaying real measurement data


VR/AR -world offer new kind of experience to the operation and maintenance work and to the people who work in the factory-floor or back office. We at LUAS believe we have a good opportunity to introduce this Fortum’s Järvenpää use case demo and VR/AR technology to industry professionals in events at Kemi and Oulu. It also gives a fantastic opportunity to disseminate what we have learned during the Mantis project to a wider audience.