The topic is addressed via the help of a novel RGBD (color and depth) format and covers the full end-to-end system pipeline: capture, processing, transmission and rendering.
Video conferencing has become a commonly used tool for both private and professional remote communication. Despised yet immensely popular, current video conferencing and telepresence solutions have severe drawbacks in the provided quality of communication (including decreased creativity, engagement, social cues, and "zoom fatigue"). This creates the need for new communication technology that allows more natural communication with greater social presence. One way to potentially increase the communication quality is to extend video conferencing towards immersive 3D communication (also called holographic communication or Social Extended Reality, XR). Creating Social XR systems comes with multiple challenges in both technology developments and impact to the user experience.
One of the main question the thesis tries to answer is: “What are the technical challenges and limitations in extending existing video conferencing solutions to support photorealistic 3D immersive communication with RGBD (color and depth) video streams?” To address this question the dissertation is split into 3 main parts.
1.) A modular capture and processing application and detailed in-depth performance evaluations to record and reconstruct users via a novel 3D RGBD-based data format.
2.) A web-based system and system evaluation the measure the impact on network performance.
3.) A novel central composition component (including its evaluation) to optimize transmission and allow a number of simultaneous users similar to existing video conferencing solutions.
Furthermore, the dissertation presents three prototype demonstrators to show the applicability of our solutions under different use cases.
Finally, the results presented in the dissertation can be seen as a new reference framework for the general building blocks of Social XR systems and a point of reference for performance evaluation comparisons.
Simon N.B. Gunkel is a Senior R&D Scientist at TNO, the Netherlands Organization for Applied Scientific Research in The Hague. Simon received a master’s degree in computer science in 2012, both from the Technical University Berlin, Germany, and KAIST, Daejeon, Republic of Korea. With over 12 years of experience in multimedia R&D, Simon’s focus is on topics relating to Extended Reality (AR, VR, MR, XR), (photo-realistic) immersive communication systems, Web 4.0 and the Metaverse. Simon is pursuing a PhD on the topic of Social XR at the VU Amsterdam in collaboration with TNO and the Centrum Wiskunde & Informatica (CWI).
More information
PhD Defense Simon Gunkel: Tuesday December 3rd 2024, 1.45-3.00 PM at the VU University Amsterdam (Auditorium)
Promotors: Prof. dr. D.C.A. Bulterman, Prof. dr. J. van Ossenbruggen
Co-Promotor: Prof. dr. P.S. César Garcia
The dissertation is available online at the VU Library:
https://doi.org/10.5463/thesis.906
Relevant Publications:
- Simon NB Gunkel, Hans M Stokking, Martin J. Prins, Omar Niamut, Ernestasia Siahaan, and Pablo Cesar. Experiencing Virtual Reality Together: Social VR Use Case Study. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video, pages 233-238, 2018.
- Simon NB Gunkel, Hans M Stokking, Martin J Prins, Nanda van der Stap, Frank B ter Haar, and Omar A Niamut. Virtual reality conferencing: Multi-user immersive vr experiences on the web. In Proceedings of the ACM Multimedia Systems Conference, pages 498–501, 2018
- Simon NB Gunkel, Rick Hindriks, KarimMEl Assal, HansMStokking, Sylvie Dijkstra-Soudarissanane, Frank ter Haar, and Omar Niamut. Vrcomm: an end-to-end web system for real-time photorealistic social vr communication. In Proceedings of the ACM Multimedia Systems Conference, pages 65–79, 2021
- Sylvie Dijkstra-Soudarissanane, Simon NB Gunkel, and Véronne Reinders. Virtual visits: life-size immersive communication. In Proceedings of the ACM Multimedia Systems Conference, pages 310–314, 2022
- Simon N.B. Gunkel, Sylvie Dijkstra-Soudarissanane, Hans M. Stokking, and Omar Niamut. From 2d to 3d video conferencing: Modular rgb-d capture and reconstruction for interactive natural user representations in immersive extended reality (xr) communication. Frontiers in Signal Processing, 3, 2023
- Simon NB Gunkel, Sylvie Dijkstra-Soudarissanane, and Omar Niamut. "you ar’ right in front of me": Rgbd-based capture and rendering for remote training. In Proceedings of the ACM Multimedia Systems Conference pages 307–311, 2023.
- Simon N.B. Gunkel, Rick Hindriks, Yonatan Shiferaw, Sylvie Dijkstra-Soudarissanane, and Omar Niamut. Vp9 bitstream-based tiled multipoint control unit: Scaling simultaneous rgbd user streams in an immersive 3d communication system. In Proceedings of the ACM Multimedia Systems Conference, pages 23–33, 2024.
