Internet is used for communication via video conferencing, voice calls, chatting, photo sharing, online gaming and virtual reality. These technologies are changing our daily lives and the way we interact with each other. Highly realistic reconstructed 3D representations that combine real and synthetic contents in a virtual world will be part of this online interaction more and more. This kind of communication poses many challenges to the existing Internet infrastructure however.
A large part of the challenge is due to the volume of reconstructed 3D data. End-to-End Internet connections are bandlimited and currently cannot support real-time end-to-end transmission of uncompressed 3D point cloud or mesh scans. Therefore methods for efficient compression and transmission need to be developed, taking different software and devices such as browsers, desktop applications, mobile applications or server side applications into account.
PhD Student Rufael Mekuria from CWI has developed strategies for compression and transmission of reconstructed 3D data in internet infrastructures. His research focuses on networking issues for 3D media content, specifically real-time 3D-Tele immersive content. It envisions that end-users can use various 3D capturing devices to create 3D representations of themselves and transmit them in real-time. Representations include triangle meshes and point clouds reconstructed from multiple depth images and spatial audio reconstructed with microphone arrays. These representations are currently not well supported by existing media codecs and transmission schemes and will enable new virtual reality and 3D tele-porting capabilities.
The new strategies for compression and transmission of reconstructed 3D data in internet infrastructures are 3 different approaches for the compression of 3D meshes and a codec for time varying 3 D point clouds. The newly developed strategies support real-time transmission and synchronization of 3D representations in the internet. Applications are in 3D-Tele Immersion, immersive and virtual environments. The strategies will make it possible to mix gaming (3D) content and realistic content (real people, video) more easily. Part of the implementation is open source, and work has led to the start of a consumer industry standardization activity in MPEG on point cloud compression.
Mekuria will defend his thesis on this topic 'Network Streaming and Compression for Mixed Reality Tele-Immersion' on Monday 23 January at VU University (VU). He carried out his research at the Distributed and Interactive Systems (DIS) group at CWI, supervised by Dick Bulterman (promotor) and Pablo César (co-promotor and head of the DIS group at CWI). The research was funded by the EU FP7 REverie project.
Paper: Design, Implementation and Evaluation of a Point Cloud codec for Tele-Immersive Video
