CG4002 AR Laser Tag [Computer Engineering Capstone Project]

🥽 Only Capstone Project in Cohort built with AR Head-Mounted Display (HMD) Visualizer

📝 Overview:
The module of CG4002 marks the culmination of my four years of undergraduate experience with AR head-mounted display, where I worked with four teammates to build a fully functional AR laser tag from scratch.

Players will compete against each other 1-on-1, and will each wear a pair of gloves with IMU and gyroscope sensors. They will also don a Microsoft HoloLens 2 that will visualize all game effects and status data. Players can perform certain actions like boxing punches, and the data from the sensors will be passed wirelessly to a neural network model on FPGA for realtime inference. The result will be published on MQTT channel to a broker laptop, which will then be published to the HoloLens for realtime game effect visualization and game status update.

I worked on the visualizer component of the game, and with the support from my lab mates and the teaching team, as well as through collaboration with my teammates, I was beyond excited to deliver the only HMD-based visualizer in my cohort.

🎨 Roles:
AR Designer and Developer:
1️⃣ Iteratively designed and implemented two sets of 3D, body-centric game UIs to allow immersive and intuitive gameplay experience, that shows live game status data like player health, player bullet count and enemy shield health etc, with Unity.
2️⃣ Developed end-to-end communication through MQTT with a broker laptop to receive player action inputs from the IMU and gyroscope sensors on their hands.
3️⃣ Created custom AR gameplay effects corresponding to each player attack action in Unity.
4️⃣ Conducted user testings with 4 different designs of Vuforia image target for best user experience and image target recognition by HoloLens.

💻 Extra Miles:
1️⃣ HoloLens 2 Scene Adaptation
Our team ran into the unique challenge of unexpected application crashing due to HoloLens being unable to recognize the physical surrounding of users if they move too far away from their initial position during gameplay, which other groups did not encounter on their smartphone-based visualizer. We eventually figured out that HoloLens always stores a temporary 3D map of its surrounding as it is being used in that environment, and that pre-mapping all the possible places the user may move to in that environment will significantly decrease the chances of unexpected application shutdown. So my teammates and I mapped the location of our capstone demo into the HoloLens a day before our final demo, and we had no crash on the actual demo day and achieved the highest score in the afternoon session.

💡 Self-Learnt Skills:
1️⃣ MRTK: Learnt how to use MRTK packages to develop for HoloLens, such as the MRTK UI and body anchors, in one month, with no prior knowledge in MRTK and HoloLens development.
2️⃣ MQTT: Learnt how to implement MQTT connections between laptop and HMD in one week with no prior knowledge in computer network.

💰 Results:
1️⃣ A grade for CG4002
2️⃣ Only team in cohort that completed the capstone project on AR HMD
3️⃣ Thumbnail feature on Channel News Asia as part of its documentary on innovation