Understanding users to optimise 3D experiences

When we think of the concept of ‘living in the future’, there is perhaps no technology that encapsulates this more than virtual reality (VR) and other immersive media. We are now able to bridge the gap between the 2D and the 3D, getting immersed in new digital worlds, whether for entertainment, gaming, education, or even communication. But as is the case with many new technologies, there are great challenges still to overcome before it can be more universally adopted. “My focus is mainly on understanding how users interact with immersive content, because with this kind of content it's not any more like you're watching a TV screen, where what you see is decided by a director,” says Silvia Rossi, a postdoctoral researcher in the Distributed and Interactive Systems (DIS) group at Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands. “Now with immersive media, you are the director, the viewer is the one that is deciding where to put the focus. This interaction is what makes this technology very fascinating, but at the same time makes it challenging.” 

Quality of Experience

It’s challenging because immersive media like virtual reality (VR), augmented reality (AR), mixed reality (MR) and 360° video, have heavy technological demands. A VR headset needs to respond in real time to every tilt of the user’s head or movement of their hands. Compared to traditional media, these immersive environments therefore require very low latency (the response time between a user’s action and the content being served) and very high bandwidth (data transfer capacity). Silvia is working to “anticipate or understand” the types of movement and interactions users have inside different types of content, to optimise the system. “If we know in advance where the user will look, we can transmit that part of the content at high quality to ensure a good quality of experience,” she explains.

Unique technology

To solve these issues, Silvia is working on two distinct areas of understanding user interactions in immersive media. Building on the work from her prize-winning PhD in London, she is using machine learning techniques to develop algorithms that can cluster users, and anticipate their movements within a piece of immersive content, to then serve up the right bit of content at the right time - reducing latency and easing the burden on bandwidth. Along with this, she works with a multidisciplinary group on human computer interaction within immersive media. “We did some experiments to try to understand the response that users were having in VR, to see, for example, if how we behave in the real world is also the same type of behaviour we have in VR,” she says. They looked at whether users respond to “social density” in VR the same way they do in the real world, finding that just like you might avoid a small space densely packed with other people in reality, users also felt uncomfortable “moving” around a virtual reality space with too many people in it. ”What makes this so interesting, is that in VR we are not really physically there, but we still have our preconceptions and beliefs of the real world and end up following them.”

Silvia first worked with CWI when she did an internship during the last year of her PhD, finding the multidisciplinary environment allowed her to get feedback on her work from “different perspectives that sometimes are super useful”. While she enjoys tapping into the diverse range of expertise within her group, she’s particularly excited by her lab’s possession of unique technology. It is one of just a handful with the specialised capability to live stream point clouds, which are data points plotted in 3D space, she says. Essentially, they can capture live depth camera footage of a person and transform them into a 3D digital entity. “Before it was more theory, my study during my PhD. Now at CWI we have the technology to do real time streaming with volumetric content, so step by step, I'm increasing the level of immersive media that I'm working with.” 

Working and living in Amsterdam

She also highlights how well connected DIS is with industry in the country, with the group having partnerships with companies in areas including creative industries, smart cities, and automotive. “For the moment I'm more interested in academia, but I have many colleagues that, thanks to the contact that we have with industry, they easily found their jobs after the PhD.” The transition to living in Amsterdam following five years in London has also been impactful for Silvia. “Amsterdam still gives you the possibility to have a lot of cultural experiences with museums and music, but at the same time it is smaller, and you can really go around with your bike…The cycling bit was what made me fall in love with Amsterdam.”