The AI-supported Augmented Reality creator platform making immersive tours possible
Using a smartphone or tablet to return to the past of memorial sites or to get a glimpse of urban planning of the future: this can now be easily realised thanks to the MediaTech Hub startup ZAUBAR’s AR platform “We are telling location-based stories with our AR app. Wherever there is a limited physical location that can be digitally enhanced,” is how COO Anne-Sophie Panzer describes the application. She founded ZAUBAR together with CEO Stefan Marx in 2019. What originally started as a storytelling tool for journalists at MIZ Babelsberg has now developed into an app used by cultural institutions, tourism companies, factories, football stadiums and public transport systems.
ZAUBAR not only uses immersive technology to create precisely locatable AR experiences in project-specific collaborations, but also offers individuals with no prior technical knowledge the opportunity to create their own AR tours on the creator platform. Until now, Augmented Reality experiences had been distributed via apps or technically inferior web AR. This year, however, has seen ZAUBAR developing a new solution via App Clips and Instant Apps, a mixture of app and web combining the best of both worlds: instant experiences without compromising on quality.
A digital look back into the past
The project “The Liberation”, which ZAUBAR developed together with Bayerischer Rundfunk, tells visitors coming to the Dachau Concentration Camp Memorial Site about the day when the camp was liberated by the US Army on 29 April, 1945. The AR experience is primarily aimed at young people. When visiting the camp, they can experience certain sections directly on site and simultaneously in digital form to see personal stories of the inmates. The barracks at the Dachau camp - of which only some are now preserved and still standing at the Memorial Site - housed around 200,000 prisoners, many close to starvation and with 32,000 perishing. What was happening there? What did it look like back then? In the app, life-size black-and-white images captured by a photographer on the day of liberation are superimposed onto the surroundings. This creates a different and direct way of communicating history.
The collaboration with the team of the Memorial Site and the client Bayerischer Rundfunk was handled with particular sensitivity. The “The Liberation” app, which appeared in 2020 to mark the liberation’s 75th anniversary, was also nominated for the Grimme Online Award.
Another project with a historical background not only received a nomination, but then took home the Grimme Online Award in the “Knowledge and Education” category in 2023: ZAUBAR joined forces with WDR on the concept and realisation of the “Stumbling Stones” project in North Rhine-Westphalia. The 16,000 or so stumbling stones in the federal state are now registered online. They remember the last residences of people who were persecuted and deported during the National Socialist era. Anyone searching the streets with a mobile phone can use AR in front of the marked houses to find out more about the deported residents from that time.
Digital guide with an entertainment factor
ZAUBAR is currently working with Hamburg’s Hochbahn transport company on a fun tourist guide. Hamburg has more day visitors than Paris. The pilot project aims to help Hamburg’s day visitors find the right way out of the S-Bahn and U-Bahn network. It’s easy to get lost at the Jungfernstieg station next to the Inner Alster Lake with its 14 different exits. “To put it rather bluntly, we are transforming the Jungfernstieg into a labyrinth. The entertainment factor aims to appeal particularly to the younger generation who are interested in AR and VR. Enter Wayment - Jungfernstieg - is the name given to the project with a play on the words entertainment and wayfinding,” says CEO Stefan Marx. There are also playful elements enabling one to collect points with the app in addition to the simple navigation elements. Hamburg’s Hochbahn will be making the AR application available for downloading from 6 December.
The technology behind the AR experiences is location-based. The ZAUBAR team visits the locations in person on several occasions. “It’s important for us to identify with the site and space,” says Anne-Sophie Panzer. “The transition from the location - the physical space - to the overlay of digital content - should be seamless.”
The social component is also important. While one can completely disappear into the digital world with VR, AR makes it possible to interact with others as well as to look at and experience something together. One is challenged to make the move to leave one’s home and go outside. All you need for this interaction is a mobile phone that everybody is now carrying with them. In future, increasing use will be made of Vision Pro and other hardware, but the focus at the moment is on providing a lean and very user-friendly technology.
Localisation via VPS or markers with QR codes
ZAUBAR uses VPS Visual Position System technology in public places. If you scan a certain side of a street, the system will recognise it and can visualise where someone is standing right down to the nearest centimetre. The team works with markers and QR codes at locations like the Dachau Concentration Camp Memorial Site and private properties. In the case of an exhibition at the Kunstpalast Düsseldorf, the paintings themselves became markers in the app. Another project with the Hohenschönhausen Memorial Site shows AR in the permanent exhibition in the cellars - the former place of work for the women imprisoned there. There should be as little interference as possible with such a historical site, and that’s why volumetric capture of contemporary witnesses can also be accessed via markers on the walls.
As an alternative, ZAUBAR recommends its customers to use the simpler option of filming with green screen, which anyone can do themselves these days using a green background and image editing tool. Volumetric capture may be very realistic in its 3D scanning of people, but even a two-dimensional representation of contemporary witnesses can be fitting to tell a story for VR. This means that everyone can create their own virtual reality tour in just a few easy steps - as the Berlin city guide Jeff Mannes did. He uses ZAUBAR for his themed tours such as the “History of LGBTQ Berlin” which he runs in the district around Nollendorfplatz. Participants are able to use AR to call up images and portraits of the most important protagonists of times gone by. AR is used to replace the organic supermarket on the corner with an image of the El Dorado club, the Berghain of the 1920s, which stood on the same spot at that time.
Regular meetups for the XR community in the capital region
Content creation is often time-consuming and expensive. There will be greater deployment of AI in the future. This is somewhere where the ZAUBAR team sees a great opportunity in becoming automated in the creation of worlds and content in 3D. As Anne-Sophie Panzer notes: “We are experimenting a lot with generative AI although the terms for AR and VR are becoming increasingly blurred. And we are working on developing standard technologies, training them for us and integrating them into our CMS.”
ZAUBAR is working closely with Deutsche Telekom to give a boost to the local community in Berlin and help build up a network around Mixed Reality. They organise regular meetups in Telekom’s Hubraum Space in Schöneberg. Here people have the chance to try out Vision Pro, test products and exchange ideas with other XR developers and enthusiasts.
All those interested are very welcome to attend. Information about the next meetup can be found at meetups@zaubar.com.
About MTH Blog
The media technologies of the future are already being used today – not only in the entertainment sector, but also in a wide variety of industries. Christine Lentz meets up with tech enthusiasts, established companies and researchers for our monthly MediaTech Hub Potsdam blog to tell the stories behind the innovative business models.