Social Virtual Reality and Cultural Heritage

MediaScape XR places the remote visitor in the virtual Sound and Vision museum. The experience is tailor made for one specific cultural artifact: the costume that Jerney Kaagman, lead singer of the rock band Earth and Fire, wore in the music program Toppop in 1979. In the virtual environment, visitors can freely interact with a high-quality 3D model of the costume, enjoy a curated tour through this model and recreate the Toppop show with other remote visitors.

The whole experience is divided into two main parts: touring the museum and reliving the history. In the virtual museum, visitors could freely observe the 3D costume from different perspectives and scales, and read the anecdotal stories. After that, they would take a “time travel” to the costume’s history, that is, the TopPop show in 1979. Immersed in the historical setting, visitors could try on the shiny blue suit and recreate the fantastic musical show in their own style.

Pablo Cesar: “What we bring is a Social VR experience. Basically we teleport people in virtual reality. The idea is we have a highly realistic representation of yourself. We put you in a virtual environment and you can do activities with family and friends who are in remote locations.”

The MediaScape XR experience was designed with a user-centric approach, collaborating with curators from Sound and Vision as well as museum lovers. We conducted focus group sessions and co-creative workshops with them to understand the stakeholders’ motivations, gather design requirements and collect novel ideas.

By interviewing curators, it was decided to focus on one specific artefact, the costume. This artefact is firmly connected to the country’s cultural memory and hence brought more popularity and better storytelling. Moreover, it allows for more immersive and fun experiences in VR, such as trying on the suit. A list of design requirements was identified from the research results, while also the interaction and the user experience flow were decided. Following our findings, the team created the digital 3D reproductions of the artefact, museum, TopPop musical hall, as well as other exhibit elements. In addition, it was decided what kind of guide information and educational content will be shared with the visitors. We wanted to make sure visitors understand the story behind the artefact, technology and the context of this experience.

The MediaScape XR project was successfully demonstrated during VRDays 2021, showcasing it to hundreds of users. The specific setup we used to showcase the system consisted of two users. Each user was captured by three 3D sensors and was wearing an HTC VIVE PRO headset with controllers assisting them to navigate and interact in the virtual space. Although for the demo two users were placed side by side, it is not a local network communication, meaning that the setup can also work from 2 remote locations (for example one user in Paris and the other in London).

The Mediascape XR SocialVR application is built on top of the VR2Gather platform that allows us to connect multiple users in the same virtual space. The application, developed in Unity3D engine, allows this multi-user setup. To capture the users in real time, we use the cwipc framework. This framework consists of a set of libraries and tools capable of generating a full pipeline for volumetric communication between multiple clients. It performs capturing, encoding, transmission, decoding and rendering of point clouds in real time.

This new museum experience allows visitors to relive the history of museum pieces through the new medium of Social Virtual Reality. It illustrates how the traditional model of a museum experience as a passive observation is decisively shifting to active, interpretive engagement. Museums are rethinking and reworking their spaces to promote deeper understanding of their collections and mission, greater interactivity with their audiences, a fuller range of activities, and a more advanced usage of immersive technologies. The ongoing effort of digitizing museum collections has focused on the creation of digital surrogates as part of the preservation plan. MediaScape XR facilitates novel manners to enjoy and experience the artifacts or interact with them in meaningful and socially engaging ways.

Johan Oomen (Sound and Vision): “The future of the museum will be a mix of a regular museum where you can take a look at an exhibition and VR world that brings those objects to life.”

The project received international recognition in October 2022, when it was awarded the Best Demo Award at the renowned ACM International Conference on Multimedia. The DIS group and Sound and Vision continue collaborating in a number of projects, TRANSMIXR and 5D Culture, exploring how novel technologies can help cultural heritage organizations. Moreover, we are together involved in the Creative Industries Immersive Impact Coalition, a national innovation program coordinated by CLICKNL.

Groups:

• Sound & Vision
• DIS group

Projects:

• TRANSMIXR project
• MediaScape XR project
• Creative Industries Immersive Impact Coalition
• 5Dculture

Videos:

• MediaScape XR video
• Cake VR demonstration video

Scientific outputs:

• Best Paper award
• VR2Gather scientific paper
• DIS contributions to the book “Immersive Video Technologies”