Home / IEEE Technology Policy and Ethics / July 2021 / Emergence of AR and AI in Educational Institutions: A COVID-19 System Transition

Emergence of AR and AI in Educational Institutions: A COVID-19 System Transition

By Debjit Majumder, Indian Institute of Engineering Science & Technology, and  Shibpur and Anubhav Goswami, CMR Institute of Technology, Bengaluru

Humans have been subjected to pandemics for centuries. The last pandemic to have a severe hit on humans was in the 20th century. During this time. humans were barely on the brink of a major technological renaissance. Since then, a century has passed, and this race is standing again at the forefront of another severe pandemic referred to as COVID-19. Ever since the outbreak of COVID-19, people have been forced to follow isolation and social distancing mandates. Augmented Reality (AR) has evolved as a major tool to link every piece of life’s puzzle during this tough time for businesses and educational institutions [1]. A report from the AR analytics and consulting firm, Digi-Capital, predicts that by 2025, implementation of AR and virtual reality (VR) in the field of education will merely be 0.02% of the total market size of AR/VR software (as shown in figure 1) [2]. This number needs to increase for the common good of everyone, and especially for the betterment of future education systems. This article attempts to shed some light on the implementation of AR in combination with Artificial Intelligence (AI) to improve online laboratory classes for students and to ensure that they will have an enhanced user experience and better practical exposure from remote locations.

Figure 1: Future market segmentation of AR/VR software [2]
Figure 1: Future market segmentation of AR/VR software [2]
Implementation of AR-AI systems

In response to COVID-19 and in the event of succeeding viral outbreaks, future classrooms are expected to be hyper-immersive and create a learning environment through a purely digital ecosystem closely mimicking in-person interactions. This scenario merges the virtual world and the real world to create a new environment, where physical and digital objects can interact in real-time. For example, using AR/AI to create virtual labs for conducting simulated experiments based on real-world phenomena to provide students the opportunity to have more realistic interactions with the experimental apparatus, and in turn, to gain a better understanding of the activity via computer interface [3]. However, the implementation is tricky. For example, see the steps that would need to be followed in figure 2.

Figure 2: Implementation of AR-AI systems
Figure 2: Implementation of AR-AI systems

With better application of AI, students can conduct experiments using their own hands, but within a virtual environment. Improved simulations of the digital world can be created in the physical world, which will result in better interactive sessions. The Multimodal Interaction Algorithm based on Augmented Reality (ARGEV), and the development of Virtual and Real Fusion Interactive Tool Suite (VRFITS) which uses convolutional neural networks for gesture recognition in AR, are examples of tools that can help expand the capabilities. The accuracy rate of the models was measured around 99.04%, which is quite remarkable [4].

AR-AI Application in Laboratories – Recent Trends

An implementation of the Microsoft HoloLens in a physics laboratory is one of the latest trends in AR-AI systems. These AR smart glasses use AI models to track hand and eye gestures.

igure 3: AR implementation for physics lab experiment – Kirchhoff’s law
Figure 3: AR implementation for physics lab experiment – Kirchhoff’s law

These models are generally composed of compact deep neural-networks that are programmed using 3D models of human hand and eye gestures which were rendered from offline cloud processing and by using cameras to take human hand and eye gesture photos. This HoloLens experiment is used to study Kirchhoff’s law by combining experimental boxes with sensors consisting of digital ammeters and voltmeters as demonstrated in Figure 3. It also contains a processing unit and a wireless interface to transfer real-time sensor data to the HoloLens [5].

Figure 4: AR implementation in dental surgery [6]
Figure 4: AR implementation in dental surgery [6]
More recently, the use of AR and AI in dental science and surgeries has become a major trend. AR-AI technologies are used in dental implantology to help in time and cost reduction. Orthognathic and oral and maxillofacial surgeries have begun to make use of this technology as it allows them to easily expose the complex anatomy involved by expanding facial skeletal osteotomy and craniofacial structures respectively [7]. Implementation of AR in dental surgery is shown above in figure 4.

Figure 5: AR implementation in BIM
Figure 5: AR implementation in BIM

The use of AR and AI in the field of civil engineering, another emerging trend, allows for the development of Building Information Modeling (BIM) technologies. Integration of BIM in a Mobile Augmented Reality (MAR) environment will result in consolidation, optimization, and visualization of building data and models in a proper environment [8], the implementation of which is shown in Figure 5.

Pros and Cons of AR-AI Systems

Although the uses of AR-AI systems mentioned above sound very appealing, they are very difficult to implement in the real world. There are several pros and cons: The main advantage of an AR-AI system lies in the fact that they provide highly interactive opportunities for research and learning. They can create real-world simulations in which students can actually touch and conduct experiments, even when stuck at home during the COVID-19 pandemic. In addition, AR provides easy access to the unobservable phenomenon, and last but not least, it enables game-based learning, which students find very interesting to work with [9]. Although there are so many pros of AR-AI systems, there are also a few cons. These cons, however, are more accurately defined as challenges that modern users need to overcome. The first and foremost being the knowledge gap about AR systems. Then, the usability—in other words, without the availability of proper interfaces, it is very difficult to use AR to convey messages [9]. Lastly, the production costs for AR/VR (Mixed Reality) systems are high, and this makes it very difficult for developing nations to install them even at the grass root level.


With all the advantages that AR and VR provide, it can be said that AR/VR is the future of education in the pandemic and post-pandemic eras. With the rapid technological development and application of machine learning in gesture recognition systems, in the near future, we will witness a massive revolution in the education system and the way teaching is done. This will be marked as a time when education became interesting again and interacting with physical objects while sitting at home will be child’s play for students.


  1. Billinghurst M. Augmented reality in education. New horizons for learning. 2002 Dec;12(5):1-5.
  2. https://www.iflexion.com/blog/augmented-reality-education [Last accessed on 8:37 PM, 30.05.2021]
  3. https://www.google.com/amp/s/www.indiatoday.in/amp/education-today/featurephilia/story/role-of-augmented-virtual-reality-in-education-1417739-2018-12-26 [Last accessed on 11:50 AM, 28.05.2021]
  4. Xiao M, Feng Z, Yang X, Xu T, Guo Q. Multimodal interaction design and application in augmented reality for chemical experiment. Virtual Reality & Intelligent Hardware. 2020 Aug 1;2(4):291-304.
  5. Kapp S, Thees M, Strzys MP, Beil F, Kuhn J, Amiraslanov O, Javaheri H, Lukowicz P, Lauer F, Rheinländer C, Wehn N. Augmenting Kirchhoff’s laws: Using augmented reality and smartglasses to enhance conceptual electrical experiments for high school students. The Physics Teacher. 2019 Jan;57(1):52-3.
  6. Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Sakuma I, Liao H. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE transactions on biomedical engineering. 2014 Jan 17;61(4):1295-304.
  7. Tandon D, Rajawat J. Present and future of artificial intelligence in dentistry. Journal of Oral Biology and Craniofacial Research. 2020 Jul 24.
  8. Williams G, Gheisari M, Chen PJ, Irizarry J. BIM2MAR: An efficient BIM translation to mobile augmented reality applications. Journal of Management in Engineering. 2015 Jan 1;31(1):A4014009.
  9. Galati F, Bigliardi B, Deiana A, Filippelli S, Petroni A. Pros and cons of augmented reality in education. Edulearn19 Proceedings; IATED: Valencia, Spain. 2019:9165-8.


Debjit Majumder is a Bachelor of Technology graduate in the field of Civil Engineering from Indian Institute of Engineering Science & Technology, Shibpur which is one of top 25 technical institutes in India. As a lifelong learner and lover for cross-disciplinary research he is always in search of new inter-disciplinary techniques and research loopholes where he can contribute so that mankind has a better tomorrow. He is also a sports fanatic with special interests in football, cricket and basketball and a proper person who can work in a team. He is an incoming graduate student at Technical University of Munich for Fall Semester 2021.

Anubhav Goswami is a Bachelor of Engineering graduate in the field of Electrical and Electronics Engineering from CMR Institute of Technology, Bengaluru. As a creative person, he possesses a wide range of abilities that combine innovative art and design principles. He is technically well skilled in graphic designing and augmented reality technologies with expertise in a lot of design software. He is a proper blend of leadership and team work and is also a very enthusiastic person in terms of extra-curricular activities. He always hopes to make valuable contribution in this digital era to revolutionize the future of human life.



Dr. Shalli Rani is Associate Professor in CSE with Chitkara University (Rajpura(Punjab)), India. She has 14+ years of teaching experience. She received MCA degree from Maharishi Dyanand University, Rohtak in 2004 and the M.Tech. degree in Computer Science from Janardan Rai Nagar Vidyapeeth University, Udaipur in 2007 and a Ph.D. degree in Computer Applications from Punjab Technical University, Jalandhar in 2017. Her main area of interest and research is Wireless Sensor Networks, Underwater Sensor Networks, and the Internet of Things. She has published/accepted/presented more than 35 papers in international journals /conferences and two books with Springer. She has worked on Big Data, Underwater Acoustic Sensors, and IoT to show the importance of WSN in IoT applications. She received a young scientist award in Feb. 2014 from Punjab Science Congress, in the same field.