Hostname: page-component-848d4c4894-r5zm4 Total loading time: 0 Render date: 2024-06-29T03:25:06.122Z Has data issue: false hasContentIssue false
  • English
  • Français

Evaluating the feeling of control in virtual object translation on 2D interfaces

Published online by Cambridge University Press:  02 March 2023

Wenxin Sun ,
Mengjie Huang Open the ORCID record for Mengjie Huang [Opens in a new window] ,
Chenxin Wu ,
Rui Yang ,
Ji Han  and
Yong Yue
Wenxin Sun
Affiliation:
Design School, Xi'an Jiaotong-Liverpool University, Suzhou, China Department of Civil Engineering and Industrial Design, University of Liverpool, Liverpool, UK
Mengjie Huang*
Affiliation:
Design School, Xi'an Jiaotong-Liverpool University, Suzhou, China
Chenxin Wu
Affiliation:
School of Advanced Technology, Xi'an Jiaotong-Liverpool University, Suzhou, China
Rui Yang
Affiliation:
School of Advanced Technology, Xi'an Jiaotong-Liverpool University, Suzhou, China
Ji Han
Affiliation:
Business School, University of Exeter, Exeter, UK
Yong Yue
Affiliation:
School of Advanced Technology, Xi'an Jiaotong-Liverpool University, Suzhou, China
*
Author for correspondence: Mengjie Huang, E-mail: mengjie.huang@xjtlu.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Computer-aided design (CAD) plays an essential role in creative idea generation on 2D screens during the design process. In most CAD scenarios, virtual object translation is an essential operation, and it is commonly used when designers simulate their innovative solutions. The degrees of freedom (DoF) of virtual object translation modes have been found to directly impact users’ task performance and psychological aspects in simulated environments. Little is known in the existing literature about the sense of agency (SoA), which is a critical psychological aspect emphasizing the feeling of control, in translation modes on 2D screens during the design process. Hence, this study aims to assess users’ SoA in virtual object translation modes on mouse-based, touch-based, and handheld augmented reality (AR) interfaces through subjective and objective measures, such as self-report, task performance, and electroencephalogram (EEG) data. Based on our findings in this study, users perceived a greater feeling of control in 1DoF translation mode, which may help them come up with more creative ideas, than in 3DoF translation mode in the design process; additionally, the handheld AR interface offers less control feel, which may have a negative impact on design quality and creativity, as compared with mouse- and touch-based interfaces. This research contributes to the current literature by analyzing the association between virtual object translation modes and SoA, as well as the relationship between different 2D interfaces and SoA in CAD. As a result of these findings, we propose several design considerations for virtual object translation on 2D screens, which may enable designers to perceive a desirable feeling of control during the design process.

Keywords

2D interfaceselectroencephalogramsense of agencyvirtual object translation
Type
Research Article
Information
AI EDAM , Volume 37 , 2023 , e9
Check for updates
Copyright
Copyright © The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abraham, A (2016) Gender and creativity: an overview of psychological and neuroscientific literature. Brain Imaging and Behavior 2, 609–618.CrossRefGoogle Scholar
Acharya, S and Chakrabarti, A (2020) A conceptual tool for environmentally benign design: development and evaluation of a “proof of concept”. Artificial Intelligence for Engineering Design Analysis and Manufacturing 34, 30–44.CrossRefGoogle Scholar
Alibali, MW (2005) Gesture in spatial cognition: expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation 5, 307–331.CrossRefGoogle Scholar
Alicja, K and Maciej, S (2022) Can AI see bias in X-ray images? International Journal of Network Dynamics and Intelligence 1, 48–64.Google Scholar
Alkemade, R, Verbeek, FJ and Lukosch, SG (2017) On the efficiency of a VR hand gesture-based interface for 3D object manipulations in conceptual design. International Journal of Human–Computer Interaction 33, 882–901.CrossRefGoogle Scholar
Alvarez, JA and Emory, E (2006) Executive function and the frontal lobes: a meta-analytic review. Neuropsychology Review 16, 17–42.CrossRefGoogle ScholarPubMed
Aris, SAM, Jalil, SZA, Bani, NA, Kaidi, HM and Muhtazaruddin, MN (2018) Statistical feature analysis of EEG alpha asymmetry between relaxed and non-relaxed. In 2018 2nd International Conference on BioSignal Analysis, Processing and Systems (ICBAPS). IEEE, pp. 171–175.CrossRefGoogle Scholar
Atilola, O and Linsey, J (2015) Representing analogies to influence fixation and creativity: a study comparing computer-aided design, photographs, and sketches. AI EDAM 29, 161–171.Google Scholar
Bai, H, Lee, GA and Billinghurst, M (2012) Freeze view touch and finger gesture-based interaction methods for handheld augmented reality interfaces. In Proceedings of the 27th Conference on Image and Vision Computing New Zealand, pp. 126–131.CrossRefGoogle Scholar
Benedek, M (2018) Internally directed attention in creative cognition. In Jung, RE and Vartanian, O (eds), The Cambridge Handbook of the Neuroscience of Creativity. Cambridge, UK: Cambridge University Press, pp. 180–194.CrossRefGoogle Scholar
Bergstrom-Lehtovirta, J, Coyle, D, Knibbe, J and Hornbæk, K (2018) I really did that: sense of agency with touchpad, keyboard, and on-skin interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–8.CrossRefGoogle Scholar
Besançon, L, Ynnerman, A, Keefe, DF, Yu, L and Isenberg, T (2021) The state of the art of spatial interfaces for 3D visualization. Computer Graphics Forum 40, 293–326.CrossRefGoogle Scholar
Bonnici, A, Akman, A, Calleja, G, Camilleri, KP, Fehling, P, Ferreira, A and Rosin, PL (2019) Sketch-based interaction and modeling: where do we stand? AI EDAM 33, 370–388.Google Scholar
Brewer, RN and Kameswaran, V (2018) Understanding the power of control in autonomous vehicles for people with vision impairment. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 185–197.CrossRefGoogle Scholar
Camba, J, Contero, M and Johnson, M (2014) Management of visual clutter in annotated 3D CAD models: a comparative study. In Marcus, A, Møllenbach, E, Abascal, J and Sturdee, M (eds), International Conference of Design, User Experience, and Usability. Cham: Springer, pp. 405–416.Google Scholar
Cao, J, Zhao, W and Guo, X (2021) Utilizing EEG to explore design fixation during creative idea generation. Computational Intelligence and Neuroscience 2021, 1–10.Google ScholarPubMed
Caspar, EA, De Beir, A, Lauwers, G, Cleeremans, A and Vanderborght, B (2021) How using brain-machine interfaces influences the human sense of agency. PLoS One 16, 1–24.CrossRefGoogle ScholarPubMed
Chang, HJ, Huang, K and Wu, C (2006) Determination of sample size in using central limit theorem for Weibull distribution. International Journal of Information and Management Sciences 17, 31.Google Scholar
Chang, HJ, Wu, CH, Ho, JF and Chen, PY (2008) On sample size in using central limit theorem for gamma distribution. Information and Management Sciences 19, 153–174.Google Scholar
Coyle, D, Moore, J, Kristensson, PO, Fletcher, P and Blackwell, A (2012) I did that! Measuring users’ experience of agency in their own actions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2025–2034.CrossRefGoogle Scholar
Cun, W, Mo, R, Chu, J, Yu, S, Zhang, H, Fan, H and Chen, C (2021) Sitting posture detection and recognition of aircraft passengers using machine learning. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 35, 284–294.CrossRefGoogle Scholar
Dong, Z, Piumsomboon, T, Zhang, J, Clark, A, Bai, H and Lindeman, R (2020) A comparison of surface and motion user-defined gestures for mobile augmented reality. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–8.CrossRefGoogle Scholar
Drucker, SM, Fisher, D, Sadana, R, Herron, J and Schraefel, MC (2013) Touchviz: a case study comparing two interfaces for data analytics on tablets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2301–2310.CrossRefGoogle Scholar
Fillingim, KB, Shapiro, H, Reichling, CJ and Fu, K (2021) Effect of physical activity through virtual reality on design creativity. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 35, 99–115.CrossRefGoogle Scholar
Gallagher, S (2000) Philosophical conceptions of the self: implications for cognitive science. Trends in Cognitive Sciences 4, 14–21.CrossRefGoogle ScholarPubMed
Goh, ES, Sunar, MS and Ismail, AW (2019) 3D object manipulation techniques in handheld mobile augmented reality interface: a review. IEEE Access 7, 40581–40601.CrossRefGoogle Scholar
Gorantla, VR, Tedesco, S, Chandanathil, M, Maity, S, Bond, V, Lewis, C and Millis, RM (2020) Associations of alpha and beta interhemispheric EEG coherences with indices of attentional control and academic performance. Behavioral Neurology 2020, 1–7.CrossRefGoogle ScholarPubMed
Guerino, GC and Valentim, NMC (2020) Usability and user experience evaluation of natural user interfaces: a systematic mapping study. IET Software 14, 451–467.CrossRefGoogle Scholar
Haggard, P (2005) Conscious intention and motor cognition. Trends in Cognitive Sciences 9, 290–295.CrossRefGoogle ScholarPubMed
Hancock, M, Ten Cate, T and Carpendale, S (2009) Sticky tools: full 6DoF force-based interaction for multi-touch tables. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, pp. 133–140.CrossRefGoogle Scholar
Hong, TCK and Economou, A (2022) What shape grammars do that CAD should: the 14 cases of shape embedding. AI EDAM 36, e4, 1–20.Google Scholar
Horvat, N, Martinec, T, Lukačević, F, Perišić, MM and Škec, S (2022) The potential of immersive virtual reality for representations in design education. Virtual Reality 26, 1–18.CrossRefGoogle Scholar
Howard, EE, Edwards, SG and Bayliss, AP (2016) Physical and mental effort disrupts the implicit sense of agency. Cognition 157, 114–125.CrossRefGoogle ScholarPubMed
Hu, ZH, Ding, YS, Zhang, WB and Yan, Q (2008) An interactive co-evolutionary CAD system for garment pattern design. Computer-Aided Design 40, 1094–1104.CrossRefGoogle Scholar
Huang, G (2005) Introducing virtual engineering technology into interactive design process with high-fidelity models. In Proceedings of the Winter Simulation Conference, 2005. IEEE, pp. 10-pp.Google Scholar
Jeunet, C, Albert, L, Argelaguet, F and Lécuyer, A (2018) “Do you feel in control?”: towards novel approaches to characterize, manipulate and measure the sense of agency in virtual environments. IEEE Transactions on Visualization and Computer Graphics 24, 1486–1495.CrossRefGoogle ScholarPubMed
Jia, W and Zeng, Y (2021) EEG signals respond differently to idea generation, idea evolution and evaluation in a loosely controlled creativity experiment. Scientific Reports 11, 1–20.CrossRefGoogle Scholar
Kang, SY, Im, CH, Shim, M, Nahab, FB, Park, J, Kim, DW and Hallett, M (2015) Brain networks responsible for sense of agency: an EEG study. PLoS One 16, 1–24.Google Scholar
Khan, S and Tunçer, B (2019) Speech analysis for conceptual CAD modeling using multi-modal interfaces: an investigation into architects’ and engineers’ speech preferences. AI EDAM 33, 275–288.Google Scholar
Khanna, A, Pascual-Leone, A, Michel, CM and Farzan, F (2015) Microstates in resting-state EEG: current status and future directions. Neuroscience & Biobehavioral Reviews 49, 105–113.CrossRefGoogle ScholarPubMed
Kim, M and Han, J (2019) Effects of switchable DOF for mid-air manipulation in immersive virtual environments. International Journal of Human–Computer Interaction 35, 1147–1159.CrossRefGoogle Scholar
Kim, D, Park, J and Ko, KH (2018) Development of an AR based method for augmentation of 3D CAD data onto a real ship block image. Computer-Aided Design 98, 1–11.CrossRefGoogle Scholar
Klimesch, W (2012) Alpha-band oscillations, attention, and controlled access to stored information. Trends in Cognitive Sciences 16, 606–617.CrossRefGoogle ScholarPubMed
Knoedel, S and Hachet, M (2011) Multi-touch RST in 2D and 3D spaces: studying the impact of directness on user performance. In 2011 IEEE Symposium on 3D User Interfaces (3DUI), pp. 75–78.CrossRefGoogle Scholar
Kochhar, S (1994) CCAD: a paradigm for human-computer cooperation in design. IEEE Computer Graphics and Applications 14, 54–65.CrossRefGoogle Scholar
Kuttikat, A, Noreika, V, Shenker, N, Chennu, S, Bekinschtein, T and Brown, CA (2016) Neurocognitive and neuroplastic mechanisms of novel clinical signs in CRPS. Frontiers in Human Neuroscience 10, 16.CrossRefGoogle ScholarPubMed
Kwon, E, Huang, F and Goucher-Lambert, K (2022) Enabling multi-modal search for inspirational design stimuli using deep learning. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 36, e22, 1–18.CrossRefGoogle Scholar
Lee, GA, Yang, U, Kim, Y, Jo, D, Kim, KH, Kim, JH and Choi, JS (2009) Freeze-set-go interaction method for handheld mobile augmented reality environments. In Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology, pp. 143–146.CrossRefGoogle Scholar
Li, S, Becattini, N and Cascini, G (2021) Correlating design performance to EEG activation: early evidence from experimental data. Proceedings of the Design Society 1, 771–780.CrossRefGoogle Scholar
Liu, L, Li, Y, Xiong, Y, Cao, J and Yuan, P (2018) An EEG study of the relationship between design problem statements and cognitive behaviors during conceptual design. Ai Edam 32, 351–362.Google Scholar
Louis, T, Troccaz, J, Rochet-Capellan, A, Hoyek, N and Bérard, F (2020) When high fidelity matters: AR and VR improve the learning of a 3D Object. In Proceedings of the International Conference on Advanced Visual Interfaces, pp. 1–9.CrossRefGoogle Scholar
Lukoff, K, Lyngs, U, Zade, H, Liao, JV, Choi, J, Fan, K and Hiniker, A (2021) How the design of Youtube influences user sense of agency. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–17.CrossRefGoogle Scholar
Lupinetti, K, Cabiddu, D, Giannini, F and Monti, M (2019) CAD3A: a web-based application to visualize and semantically enhance CAD assembly models. In 2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS). IEEE, pp. 462–469.CrossRefGoogle Scholar
Mathewson, KE, Lleras, A, Beck, DM, Fabiani, M, Ro, T and Gratton, G (2011) Pulsed out of awareness: EEG alpha oscillations represent a pulsed-inhibition of ongoing cortical processing. Frontiers in Psychology 2, 99.CrossRefGoogle ScholarPubMed
Moldovan, A, Nicula, V, Pasca, I, Popa, M, Namburu, JK, Oros, A and Brie, P (2020) A user interface description language for runtime omni-channel user interfaces. Proceedings of the ACM on Human-Computer Interaction 4, 1–52.CrossRefGoogle Scholar
Nanjappan, V, Shi, R, Liang, HN, Xiao, H, Lau, KKT and Hasan, K (2019) Design of interactions for handheld augmented reality devices using wearable smart textiles: findings from a user elicitation study. Applied Sciences 9, 3177.CrossRefGoogle Scholar
Nataraj, R and Sanford, S (2021) Control modification of grasp force covaries agency and performance on rigid and compliant surfaces. Frontiers in Bioengineering and Biotechnology 8, 574006.CrossRefGoogle ScholarPubMed
Pejic, J and Pejic, P (2022) Linear kitchen layout design via machine learning. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 36, e9, 1–12.CrossRefGoogle Scholar
Reisman, JL, Davidson, PL and Han, JY (2009) A screen-space formulation for 2D and 3D direct manipulation. In Proceedings of the 22nd annual ACM Symposium on User Interface Software and Technology, pp. 69–78.CrossRefGoogle Scholar
Rekimoto, J (2014) A new you: from augmented reality to augmented human. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces, pp. 1–2.CrossRefGoogle Scholar
Rogers, K, J, C, Frommel, J, Stamm, S and Weber, M (2019) Exploring interaction fidelity in virtual reality: object manipulation and whole-body movements. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–14.CrossRefGoogle Scholar
Roth, D and Latoschik, ME (2019) Construction of a validated virtual embodiment questionnaire. Arxiv Preprint Arxiv, 1911.10176.Google Scholar
Schneider, H, Eiband, M, Ullrich, D and Butz, A (2018) Empowerment in HCI-A survey and framework. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–14.CrossRefGoogle Scholar
Seghezzi, S, Zirone, E, Paulesu, E and Zapparoli, L (2019) The brain in (willed) action: a meta-analytical comparison of imaging studies on motor intentionality and sense of agency. Frontiers in Psychology 10, 804.CrossRefGoogle Scholar
Su, GE, Sunar, MS and Ismail, AW (2020) Device-based manipulation technique with separated control structures for 3D object translation and rotation in handheld Mobile AR. International Journal of Human-Computer Studies 141, 102433.CrossRefGoogle Scholar
Sun, W, Huang, M, Yang, R, Zhang, J, Wang, L, Han, J and Yue, Y (2020) Workload, presence and task performance of virtual object manipulation on WebVR. In 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 358–361.CrossRefGoogle Scholar
Sun, W, Huang, M, Yang, R, Han, J and Yue, Y (2021) Mental workload evaluation of virtual object manipulation on WebVR: an EEG study. In 2021 IEEE International Conference on Human System Interaction (HSI), pp. 358–361.CrossRefGoogle Scholar
Sun, W, Huang, M, Wu, C and Yang, R (2022 a) Exploring virtual object translation in head-mounted augmented reality for upper limb motor rehabilitation with motor performance and eye movement characteristics. In Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1–3.CrossRefGoogle Scholar
Sun, W, Huang, M, Wu, C and Yang, R (2022 b) Sense of agency on handheld AR for virtual object translation. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 838–839.CrossRefGoogle Scholar
Sun, L, Zhang, Y, Li, Z, Zhou, Z and Zhou, Z (2022 c) inML kit: empowering the prototyping of ML-enhanced products by involving designers in the ML lifecycle. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 36, e8, 1–20.CrossRefGoogle Scholar
Tuddenham, P, Kirk, D and Izadi, S (2010) Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2223–2232.Google Scholar
Veisz, D, Namouz, EZ, Joshi, S and Summers, JD (2012) Computer-aided design versus sketching: an exploratory case study. AI EDAM 26, 317–335.Google Scholar
Vieira, S, Benedek, M, Gero, J, Li, S and Cascini, G (2022) Design spaces: neurophysiological activations in constrained and open design tasks.Google Scholar
Von Wegner, F, Knaut, P and Laufs, H (2018) EEG microstate sequences from different clustering algorithms are information-theoretically invariant. Frontiers in Computational Neuroscience 12, 70.CrossRefGoogle ScholarPubMed
Wan, Z, Yang, R, Huang, M, Zeng, N and Liu, X (2021) A review on transfer learning in EEG signal analysis. Neurocomputing 421, 1–14.CrossRefGoogle Scholar
Wang, Y, MacKenzie, CL, Summers, VA and Booth, KS (1998) The structure of object transportation and orientation in human-computer interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 312–319.CrossRefGoogle Scholar
Wang, P, Bai, X, Billinghurst, M, Zhang, S, Wei, S, Xu, G and Zhang, J (2021) 3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration. Multimedia Tools and Applications 80, 31059–31084.CrossRefGoogle Scholar
Wang, L, Huang, M, Yang, R, Liang, HN, Han, J and Sun, Y (2022) Survey of movement reproduction in immersive virtual rehabilitation. IEEE Transactions on Visualization and Computer Graphics.Google ScholarPubMed
Watson, D, Hancock, M, Mandryk, RL and Birk, M (2013) Deconstructing the touch experience. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces, pp. 199–208.CrossRefGoogle Scholar
Wen, W, Yamashita, A and Asama, H (2017) Measurement of the perception of control during continuous movement using electroencephalography. Frontiers in Human Neuroscience 11, 392.CrossRefGoogle ScholarPubMed
Wen, W, Kuroki, Y and Asama, H (2019) The sense of agency in driving automation. Frontiers in Psychology 10, 2691.CrossRefGoogle ScholarPubMed
Wodehouse, A, Loudon, B and Urquhart, L (2020) The configuration and experience mapping of an accessible VR environment for effective design reviews. AI EDAM 34, 387–400.Google Scholar
Ying, W and Aimin, W (2017) Augmented reality based upper limb rehabilitation system. In 2017 13th IEEE International Conference on Electronic Measurement & Instruments (ICEMI), pp. 426–430.CrossRefGoogle Scholar
Yu, L, Svetachov, P, Isenberg, P, Everts, MH and Isenberg, T (2010) FI3D: direct-touch interaction for the exploration of 3D scientific visualization spaces. IEEE Transactions on Visualization and Computer Graphics 16, 1613–1622.Google ScholarPubMed
Yu, N, Yang, R and Huang, M (2022) Deep common spatial pattern based motor imagery classification with improved objective function. International Journal of Network Dynamics and Intelligence 1, 73–84.CrossRefGoogle Scholar
Zhao, G, Li, Y and Xu, Q (2022) From emotion AI to cognitive AI. International Journal of Network Dynamics and Intelligence 1, 65–72.CrossRefGoogle Scholar
Zhou, F, Duh, HBL and Billinghurst, M (2008) Trends in augmented reality tracking, interaction and display: a review of ten years of ISMAR. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 193–202.CrossRefGoogle Scholar
Zito, GA, Wiest, R and Aybek, S (2020) Neural correlates of sense of agency in motor control: a neuroimaging meta-analysis. PLoS One 15, e0234321, 1–17.CrossRefGoogle ScholarPubMed