Exploration of Social Engagement During Interaction in Metaverse with an Enhanced Avatar amidst Self-Driving
Keywords:
social engagement, HoloLens 2, driver involvement, mixed reality, human-machine interactionAbstract
This research explores the impact of utilizing Microsoft HoloLens 2 in a collaborative mixed reality setting in the Metaverse, focusing on the driver's social engagement with an autonomous driving system. In (semi-) autonomous vehicles, the driver acts as the system monitor, with driving relegated to a secondary task. The study leverages Microsoft Mesh XR technology, enabling immersion in shared mixed reality environments. A comparative user study assessed social engagement in two scenarios: baseline (communication via Skype/Meet on a mobile tablet) and mixed reality collaboration (using Microsoft Mesh on HoloLens 2 with augmented 3D avatars). Participants performed social interaction tasks and a remote tic-tac-toe game while monitoring the vehicle. Social engagement was measured using the Harms and Biocca questionnaire, revealing significant disparities in Co-presence, Perceived Emotional Interdependence, and Perceived Behavioral Interdependence. Participants found ease in interacting with avatars in the mixed reality scenario. The proposed methodology could extend to evaluate driver performance during handover procedures, especially in critical situations faced by autonomous driving systems.
References
Wang, Y.; Su, Z.; Zhang, N.; Xing, R.; Liu, D.; Luan, T.H.; Shen, X. A survey on metaverse: Fundamentals, security, and privacy.
IEEE Commun. Surv. Tutor. 2022. [CrossRef]
Jin, L.; Guo, B.; Jiang, Y.; Wang, F.; Xie, X.; Gao, M. Study on the impact degrees of several driving behaviors when driving while performing secondary tasks. IEEE Access 2018, 6, 65772–65782. [CrossRef]
Ebadi, Y.; Fisher, D.L.; Roberts, S.C. Impact of cognitive distractions on drivers’ hazard anticipation behavior in complex scenarios.
Transp. Res. Rec. 2019, 2673, 440–451. [CrossRef]
Ranney, T.A. Models of driving behavior: A review of their evolution. Accid. Anal. Prev. 1994, 26, 733–750. [CrossRef]
Luke, R.; Heyns, G.J. Reducing risky driver behaviour through the implementation of a driver risk management system. J. Transp.
Supply Chain. Manag. 2014, 8, 1–10. [CrossRef]
Boer, E.R.; Hoedemaeker, M. Modeling driver behavior with different degrees of automation: A hierarchical decision framework of interacting mental models. In Proceedings of the 17th European Annual Conference on Human Decision Making and Manual
Control, Valenciennes, France, 14–16 December 1998; pp. 63–72.
International, S. Available online: https://www.sae.org/standards/content/j3016_202104/ (accessed on 21 October 2022).
Naujoks, F.; Forster, Y.; Wiedemann, K.; Neukum, A. Improving usefulness of automated driving by lowering primary task interference through HMI design. J. Adv. Transp. 2017, 2017, 6105087. [CrossRef]
Technology, N. Available online:
https://www.nissan-global.com/EN/TECHNOLOGY/OVERVIEW/i2v.html (accessed on 21
October 2022)
Biocca, F.; Harms, C.; Burgoon, J.K. Toward a more robust theory and measure of social presence: Review and suggested criteria.
Presence Teleoper. Virtual Environ. 2003, 12, 456–480. [CrossRef]
Jo, D.; Kim, K.-H.; Kim, G.J. Effects of avatar and background types on users’ co-presence and trust for mixed reality-based teleconference systems. In Proceedings of the 30th Conference on Computer Animation and Social Agents, Seoul, Republic of
Korea, 22–24 May 2017; pp. 27–36.
Yoon, B.; Kim, H.-i.; Lee, G.A.; Billinghurst, M.; Woo, W. The effect of avatar appearance on social presence in an augmented reality remote collaboration. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka,
Japan, 23–27 March 2019; pp. 547–556.
Williams, K.J.; Peters, J.C.; Breazeal, C.L. Towards leveraging the driver’s mobile device for an intelligent, sociable in-car robotic assistant. In Proceedings of the 2013 IEEE intelligent vehicles symposium (IV), Gold Coast, Australia, 23–26 June 2013; pp.
–376.
Wang, M.; Lee, S.C.; Kamalesh Sanghavi, H.; Eskew, M.; Zhou, B.; Jeon, M. In-vehicle intelligent agents in fully autonomous driving: The effects of speech style and embodiment together and separately. In Proceedings of the 13th International Conference
on Automotive User Interfaces and Interactive Vehicular Applications, Leeds, UK, 9–14 September 2021; pp. 247–254.
Medeiros, D.; McGill, M.; Ng, A.; McDermid, R.; Pantidi, N.; Williamson, J.; Brewster, S. From Shielding to Avoidance: Passenger
Augmented Reality and the Layout of Virtual Displays for Productivity in Shared Transit. IEEE Trans. Vis. Comput. Graph. 2022,
, 3640–3650. [CrossRef]
Billinghurst, M.; Kato, H. Collaborative mixed reality. In Proceedings of the First International Symposium on Mixed Reality, Yokohama, Japan, 9–11 March 1999; pp. 261–284.
Cadet, L.B.; Chainay, H. Memory of virtual experiences: Role of immersion, emotion and sense of presence. Int. J. Hum.-Comput.
Stud. 2020, 144, 102506. [CrossRef]
Blade, V. Available online: https://www.vuzix.com/products/vuzix-blade-2-smart-glasses (accessed on 21 October 2022).
Leap, M. Available online: https://www.magicleap.com/magic-leap-2 (accessed on 21 October 2022).
Xi, N.; Chen, J.; Gama, F.; Riar, M.; Hamari, J. The challenges of entering the metaverse: An experiment on the effect of extended reality on workload. Inf. Syst. Front. 2022, 1–22. [CrossRef]
Popescu, G.H.; Ciurlău, C.F.; Stan, C.I. Virtual Workplaces in the Metaverse: Immersive Remote Collaboration Tools, Behavioral Predictive Analytics, and Extended Reality Technologies. Psychosociol. Issues Hum. Resour. Manag. 2022, 10, 21–34.
Cárdenas-Robledo, L.A.; Hernández-Uribe, Ó.; Reta, C.; Cantoral-Ceballos, J.A. Extended reality applications in industry 4.0.—A
systematic literature review. Telemat. Inform. 2022, 73, 101863. [CrossRef]
Vasarainen, M.; Paavola, S.; Vetoshkina, L. A systematic literature review on extended reality: Virtual, augmented and mixed
reality in working life. Int. J. Virtual Real. 2021, 21, 1–28. [CrossRef]
Dai, F.; Olorunfemi, A.; Peng, W.; Cao, D.; Luo, X. Can mixed reality enhance safety communication on construction sites? An industry perspective. Saf. Sci. 2021, 133, 105009. [CrossRef]
Cheng, J.C.; Chen, K.; Chen, W. State-of-the-art review on mixed reality applications in the AECO industry. J. Constr. Eng. Manag.
, 146, 03119009. [CrossRef]
Stothard, P.; Squelch, A.; Stone, R.; Van Wyk, E. Towards sustainable mixed reality simulation for the mining industry. Min.
Technol. 2019, 128, 246–254. [CrossRef]
Chai, J.J.; O’Sullivan, C.; Gowen, A.A.; Rooney, B.; Xu, J.-L. Augmented/mixed reality technologies for food: A review. Trends Food Sci. Technol. 2022, 124, 182–194. [CrossRef]
Gsaxner, C.; Li, J.; Pepe, A.; Jin, Y.; Kleesiek, J.; Schmalstieg, D.; Egger, J. The HoloLens in Medicine: A systematic Review and Taxonomy. arXiv 2022, arXiv:2209.03245.
Fernandez, P. Facebook, Meta, the metaverse and libraries. Libr. Hi Tech News 2022, 39, 1–5. [CrossRef]
Kolesnichenko, A.; McVeigh-Schultz, J.; Isbister, K. Understanding emerging design practices for avatar systems in the commercial social vr ecology. In Proceedings of the 2019 on Designing Interactive Systems Conference, San Diego, CA, USA, 23–28 June 2019;
pp. 241–252.
Avatars, U. Available online: https://unionavatars.com/ (accessed on 9 November 2022).
Kim, J.I.; Ha, T.; Woo, W.; Shi, C.-K. Enhancing social presence in augmented reality-based telecommunication system. In Proceedings of the International Conference on Virtual, Augmented and Mixed Reality, Las Vegas, NV, USA, 24 July 2013; pp.
–367.
Pejsa, T.; Kantor, J.; Benko, H.; Ofek, E.; Wilson, A. Room2room: Enabling life-size telepresence in a projected augmented reality environment. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San
Francisco, CA, USA, 27 February–2 March 2016; pp. 1716–1725.
Orts-Escolano, S.; Rhemann, C.; Fanello, S.; Chang, W.; Kowdle, A.; Degtyarev, Y.; Kim, D.; Davidson, P.L.; Khamis, S.; Dou, M.
Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, 16–19 October 2016; pp. 741–754.
Concept, H.M. Available online: https://www.hyundai.news/eu/articles/press-releases/hyundai-shares-vision-of-newmetamobility-concept-through-robotics-and-Metaverse-at-CES-2022.html (accessed on 21 October 2022).
Group, B. Available online:
https://www.press.bmwgroup.com/global/article/detail/T0329569EN/bmw-group-and-nvidiatake-virtual-factory-planning-to-the-next-level?language=en (accessed on 21 October 2022).
Adventure, H.M. Available online:
https://www.hyundai.com/worldwide/en/company/newsroom/hyundai-motor-vitalizesfuture-mobility-in-roblox-Metaverse-space%252C-hyundai-mobility-adventure-0000016713 (accessed on 21 October 2022).
Driver, O. Available online: https://www.oxbotica.com/insight/oxbotica-partners-with-nevs-to-reshape-the-future-of-urbanmobility-with-fleet-of-shared-self-driving-all-electric-vehicles/ (accessed on 21 October 2022).
COWI. The Oslo Study—How Autonomous Cars May Change Transport in Cities; COWI: Lyngby, Denmark, 2019; Volume 79.
Richter, M.A.; Hagenmaier, M.; Bandte, O.; Parida, V.; Wincent, J. Smart cities, urban mobility and autonomous vehicles: How different cities needs different sustainable investment strategies. Technol. Forecast. Soc. Chang. 2022, 184, 121857. [CrossRef]
Allam, Z.; Sharifi, A.; Bibri, S.E.; Jones, D.S.; Krogstie, J. The metaverse as a virtual form of smart cities: Opportunities and challenges for environmental, economic, and social sustainability in urban futures. Smart Cities 2022, 5, 771–801. [CrossRef]
Pamucar, D.; Deveci, M.; Gokasar, I.; Tavana, M.; Köppen, M. A metaverse assessment model for sustainable transportation using ordinal priority approach and Aczel-Alsina norms. Technol. Forecast. Soc. Chang. 2022, 182, 121778. [CrossRef]
Soiné, A.; Flöck, A.N.; Walla, P. Electroencephalography (EEG) Reveals Increased Frontal Activity in Social Presence. Brain Sci.
, 11, 731. [CrossRef] [PubMed]
Cui, G.; Lockee, B.B.; Meng, C. Building modern online social presence: A review of social presence theory and its instructional design implications for future trends. Educ. Inf. Technol. 2012, 18, 661–685. [CrossRef]
Slater, M.; Wilbur, S. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence Teleoper. Virtual Environ. 1997, 6, 603–616. [CrossRef]
Biocca, F. The Cyborg’s Dilemma: Progressive Embodiment in Virtual Environments [1]. J. Comput.-Mediat. Commun. 1997, 3, JCMC324. [CrossRef]
Kreijns, K.; Xu, K.; Weidlich, J. Social presence: Conceptualization and measurement. Educ. Psychol. Rev. 2021, 34, 139–170.
[CrossRef] [PubMed]
Lombard, M.; Ditton, T. At the heart of it all: The concept of presence. J. Comput.-Mediat. Commun. 1997, 3, JCMC321. [CrossRef]
Short, J.; Williams, E.; Christie, B.A. The Social Psychology of Telecommunications; Wiley & Sons: Toronto, ON, Canada, 1976.
Oh, C.S.; Bailenson, J.N.; Welch, G.F. A Systematic Review of Social Presence: Definition, Antecedents, and Implications. Front.
Robot. AI 2018, 5, 114. [CrossRef]
Skalski, P.; Tamborini, R. The role of social presence in interactive agent-based persuasion. Media Psychol. 2007, 10, 385–413.
[CrossRef]
Zuckerman, M.; DePaulo, B.M.; Rosenthal, R. Verbal and nonverbal communication of deception. In Advances in Experimental Social Psychology; Elsevier: Amsterdam, The Netherlands, 1981; Volume 14, pp. 1–59.
Hulme, K.; Kasprzak, E.; English, K.; Moore-Russo, D.; Lewis, K. Experiential learning in vehicle dynamics education via motion simulation and interactive gaming. Int. J. Comput. Games Technol. 2009, 2009, 952524. [CrossRef]
Stewart, D. A platform with six degrees of freedom. Proc. Inst. Mech. Eng. 1965, 180, 371–386. [CrossRef]
Reymond, G.; Kemeny, A. Motion cueing in the Renault driving simulator. Veh. Syst. Dyn. 2000, 34, 249–259. [CrossRef]
Antonya, C.; Irimia, C.; Grovu, M.; Husar, C.; Ruba, M. Co-simulation environment for the analysis of the driving simulator’s actuation. In Proceedings of the 2019 7th International Conference on Control, Mechatronics and Automation (ICCMA), Delft,
The Netherlands, 6–8 November 2019; pp. 315–321.
Dosovitskiy, A.; Ros, G.; Codevilla, F.; Lopez, A.; Koltun, V. CARLA: An open urban driving simulator. In Proceedings of the Conference on Robot Learning, Mountain View, CA, USA, 13–15 November 2017; pp. 1–16.
Mesh, M. Available online: https://learn.microsoft.com/en-us/mesh/overview (accessed on 21 October 2022).
Harms, C.; Biocca, F. Internal consistency and reliability of the networked minds measure of social presence. In Proceedings of the Seventh Annual International Workshop: Presence, Valencia, Spain, 13–15 October 2004.
Hair, J.F.; Anderson, R.; Tatham, R.; Black, W. Factor analysis. Multivariate data analysis. NJ Prentice-Hall 1998, 3, 98–99.
Eurostat. Available online: https://ec.europa.eu/eurostat/web/products-eurostat-news/-/ddn-20201021-2 (accessed on 21
October 2022).
Giménez-Nadal, J.I.; Molina, J.A.; Velilla, J. Trends in commuting time of European workers: A cross-country analysis. Transp.
Policy 2022, 116, 327–342. [CrossRef]
Teodorovicz, T.; Kun, A.L.; Sadun, R.; Shaer, O. Multitasking while driving: A time use study of commuting knowledge workers to assess current and future uses. Int. J. Hum.-Comput. Stud. 2022, 162, 102789. [CrossRef]
Kim, K.; Schubert, R.; Hochreiter, J.; Bruder, G.; Welch, G. Blowing in the wind: Increasing social presence with a virtual human via environmental airflow interaction in mixed reality. Comput. Graph. 2019, 83, 23–32. [CrossRef]
