School of Science and Technology 科技學院
Computing Programmes 電腦學系
| Programme | Bachelor of Arts with Honours in Computing and Interactive Entertainment |
| Supervisor | Dr. William Lai, Dr. Keith Lee |
| Areas | Virtual Reality Applications |
| Year of Completion | 2022 |
Based on the present research, existing virtual reality games do not fulfill some of the main core requirements of upper limb training in rehabilitation exercises. For example, the game approach only supports training the motion of pinching or grasping (Sulimanov & Olano, 2019). However, other training activities, such as hand strength training and hand extension training, are not covered in the game. Thus, the game is not able to provide rehabilitation exercises that are covered in training hand motion, extension, and strength.
On the other hand, virtual reality-based stroke rehabilitation games are one of the choices for physiotherapists to attract patients who are feeling bored during rehabilitation treatments. Some virtual reality games are too focused on exercise functions but ignore the element of encouragement. A lack of short-term objectives causes a lack of engagement for patients (Barrett et al., 2016). The game of reward is important because it helps motivate the patient to do exercise continuously by offering visible rewards (Ferreira & Menezes, 2020). Therefore, a reward function is needed for patients when they use virtual reality games for rehabilitation.
Game level difficulties are also an issue with the current virtual reality-based stroke rehabilitation games. Current virtual reality-based stroke rehabilitation games use two common types of approaches to set up the game difficulties. The first one is to set up the difficulties manually before the game is started (Elor et al., 2018). The physiotherapist needs to set up the game difficulties according to the patient's rehabilitation progress. It is the static approach for setting up game difficulty. In this approach, the physiotherapist is required to adjust the game difficulties when the patient’s rehabilitation has improved. The second one sets up difficulties at different levels (Lin et al., 2020). In this approach, the patient then selects the suitable level for the rehabilitation exercise. However, those game levels have limited the difficulties. It is not possible to dynamically adjust the game difficulties that are matched with the patient's rehabilitation progress. Therefore, those approaches may not adjust to game difficulties that relate to the patient's rehabilitation progress more effectively.
To achieve the aim, the objectives of this project include
Our solution is based on using the virtual reality system as a stroke rehabilitation method to provide a few different rehabilitation exercises for the hand. Stroke rehabilitation will be provided by using the game. The game content is divided into different kinds of minigames. Each game is related to specific stroke rehabilitation. There are three types of minigames for stroke rehabilitation, such as extension, balance, and strength training. Besides providing games, we will also develop games with dynamically adjustable levels of difficulty that ensure game difficulties are suitable for different patients.
For the supporting technologies, we would develop this project by using the Unity platform with OpenXR Plugin.
VR headset
As we mention that VR headset is part of the solution to provide a host of applications. Many VR headsets are existing in the market including Oculus Quest 2, HTC Vive Pro 2 Sony, PlayStation VR, Valve Index VR Kit, Windows Mixed Reality, etc. Those are fulfilling our requirement of tracking user movement. According to our requirement, the VR headset should be comfortable for the user. However, HTC Vive Pro 2 and PlayStation VR are not flexible, because those VR devices require the base stations for tracking user movement. Therefore, we will use Oculus Quest 2 to be our VR device for the solution, based on its portable, and all-in-one system, as there is no base station needed.
Pressure Sensor
The “PressureSensor” class might be useful while detecting the pressure (Unity, 2020) (OpenXR, KHRONOS). Due to the technology gap, we would face this project. Firstly, the basic VR controllers are not supported with pressure sensors so we might make customized controllers on our own. Next, we need an amount of 3D resources for building a few game scenes and reward decorations, buying assets might be an option to alleviate this technical gap.
Cloud server (EC2)
The game system needs a cloud server to deploy the content management system (CMS), and database and is used to store and transfer user performance data between the CMS and the game system.
Comparison of pressure input devices
After conducting research from the internet, we have found a few possible pressure inputs available in the market:
In conclusion, only Ring Con is a suitable option for doing this project, Valve Index VR controller is too expensive for us, and seems not to support Oculus System, also not sold in Hong Kong. Grip System and Tactile Glove we have no idea if they are even supported in Unity and VR or not. More importantly, these products can't be bought at the public market, this is very unfavorable for our market liquidity if we use this device. Ring-con would be our first choice with a reasonable price we can pay with and most commonly, we have found possible ways to let the controller contact with unity which is hopefully can be our best option in using.
Figure 1: Ring-con
Figure 2: Valve Index VR controller
Figure 3: Grip System
Figure 4: Tactile Glove
Controller for training handgrip strength
Although we have found the driver of Ring-Con (ringrunnermg, 2021), we still have to try and it doesn’t work and the compatibility with Oculus Quest System. If this method fails we might possibly have to retrofit the controller on our own. Honestly, we lack experience with retrofitting hardware devices.
Suitable for stroke patients
We design our rehabilitation game based on the rehabilitation existence suggested by “中風多面睇” which publishing by the Hong Kong Department of Health(Department of Health, p. 38-132, 2019). But these games haven't been clinically trialed on stroke patients yet, so some of our game designs might need to modify or even need to be rebuilt. Besides that the auto difficulty system needs a lot of tests for adjusting, the auto difficulty has to be suitable with the patient’s state and difficulty can't be increased or decreased too hard at once.
As we are going to create stroke rehabilitation mini-games for stroke patients, the art direction and story genre should be more or less simple yet interactive. All posture references come from the stroke rehabilitation exercise guideline called 中風多面睇, published by the Department of Health.
Our team will create a baseball field scene. In the gameplay, the player has to use his baseball bat to hit the baseball from the pitcher. More information about game ideas in the game document.
Figure 5: Reference for gesturing in the mini-game “Base Hitter”
Figure 6: Screenshot of the mini-game “Base Hitter”
Figure 7: Reference for gesturing in the mini-game “Super Waiter”
Our team will create a restaurant scene where the player portrays a waiter and takes the plate along the passage. The game simulates the real-life action of being one of the stroke rehabilitation balancing exercises with both hands. More information about game ideas is in the game document.
Figure 8: Screenshot of the mini-game “Super Waiter”
Figure 9: Reference for gesturing in the mini-game “Stop The Fire”
In a virtual house, the player will release the fire extinguisher to put out the fire and refill the water in the extinguisher to use again. More information about game ideas is in the game document.
Figure 10: Screenshot of the mini-game “Stop The Fire”
We would have a menu lobby, which is the first view a user would see while using our product. It is a small lobby environment where users can select a game by clicking these three white panels. The left red button would go to a shop menu for buying game items, and the right red button for logout the game.
Figure 11: Screenshot of game lobby UI
In the shop menu, it will display relevant information once the player clicks on the “Shop” button in the menu lobby. This menu shows various items in a list, for example, different color of item for Game01. The player can acquire them by paying for a specific coin.
Figure 12: Screenshot of shop UI
About Game 1, there will display a game instruction at the beginning for guiding the player on how to start the game and how to play the game. The player can interact with the UI by using the VR controller.
Figure 13: Screenshot of game 1 UI
During the game, the big black screen panel is used to show the upcoming time of the ball.
Figure 14: Screenshot of game 1 UI
About Game 2, there are 2 major parts of UI elements including game progress and interactable items. The game progress will be displayed on the top to represent the status of the game progress, and in the top right corner to show the number of player collected items. About interactable items, it is used to display the start game, tutorial, and back to menu UI elements.
Figure 15: Screenshot of game 2 UI
In Game 3, there are also some interactive UI elements, for example, we used the “Push” word on the door to represent the action of starting a game and the “Exit” word on the other side of the door to represent the action of exit the game.
Figure 15: Screenshots of game 3 UI
During the game, the game status will be displayed on the top for showing the score. Also, the percentage of extinguisher storage will be shown on the dashboard.
Figure 16: Screenshots of game 3 UI
Objective of Study
Our objective is to determine whether the three VR stroke rehabilitation games: (1) Base Hitter, (2) Super Waiter, and (3) Stop The Fire, provide beneficial outcomes in playing the VR stroke rehabilitation games, complying with appropriate UI design, interaction environment, and difficulty adjustment, establishing a suitable and comfortable rehabilitation experience for stroke patients, and increasing future engagement in using the VR application.
Methodology of Study
This study’s approach is based on user experiment design. Our team had arranged for a professional physiotherapist to give us post-stroke patients so that we could test our rehabilitation method. In addition to performing the rehabilitation test on the stroke patients, we asked several colleagues of our professional physiotherapist to observe if they could make the proper gesture while playing the rehabilitation game. We presented a questionnaire for his colleagues’ opinions on this rehabilitation approach after they were tested. It assesses these games as appropriate for stroke sufferers, allowing them to measure the system’s success and determine how effective VR stroke rehabilitation games and traditional stroke therapy in a rehabilitation center can improve the mobility of stroke patients.
Details of Study
With the proposed experiment, we will focus on the gesture while playing the rehabilitation game. Our team would ask a professional physiotherapist to watch whether the testers made the proper gestures while playing the rehabilitation game and assess whether these activities were appropriate for stroke sufferers. For the stroke patients, we have discovered that they are mostly aged and have different physical disabilities. Therefore, we had to converse with aged people with different physical disabilities to instruct them on how to play the games. The following shows the experiment for our games.
Extent of Satisfying Requirements
For the game “Base Hitter”, according to the questionnaire from our physiotherapist's colleagues, we have received generally positive feedback for satisfying the requirements, with an average score of 6.5. But there is some other feedback our team should be concerned about. In the UI aspect, we have commented that the hot air balloons in the background are significantly appealing while batting the ball, whereas stroke patients may not be able to distinguish between the hitting targets and the balloon and affect the outcome of the result. In the game control aspect, one of our physiotherapist's colleagues expressed in the questionnaire that it is difficult to target the ball for batting because of the inaccurate distance between the target and the controller. With the conversation and interaction with one of the stroke patients, we have found that in the game difficulty aspect, because of their physical limitation, she is not able to stretch the hand to the hitting targets, therefore, we must hold and direct her hand in order to reach the hitting targets.
For the game “Super Waiter”, according to the questionnaire from our physiotherapist's colleagues, we have received generally positive feedback for satisfying the requirements, with an average score of 6.5, very similar to the first game. but there are also some other feedbacks our team should also be concerned about. In the game control aspect, we have been told that the game camera is especially unstable in comparison with other games, that the patient may easily get dizzy during and after playing the game, especially for the elderlies or people who cannot withstand the instability. In the game content aspect, one of our physiotherapist's colleagues has expressed in the questionnaire that the game pace is too quick for post-stroke patients, as the response of the post-stroke patients is not as quick as normal people. From our observation of another stroke patient, we have found that from the difficulty aspect, she is unfamiliar with the strategy of controlling the game, and this leads to a low score. This has shown that the difficulty adjustment is above the level of the stroke patients from the start, and becomes too difficult for them to play.
For the game “Stop the fire”, we have received generally negative feedback for satisfying the requirements. As there is not enough time for review from our physiotherapist's colleagues, there is no questionnaire feedback provided. But according to our interaction with one of the stroke patients, in the game control aspect, he is unable to stress the Ring-Con to the depth where the system detects the control from the Ring-Con. Moreover, if the stroke patient cannot use his both hands, the rehabilitation test cannot be conducted. In the game design aspect, we have found that the spawn angle for the fire is too broad, as the stroke patient is not able to turn his hand towards the fire target.
Strengths and Weaknesses of Solution
For the effectiveness of the hand extension training from the game “Base Hitter”, by the response of the questionnaire feedback, the colleagues from our physiotherapist generally agree that this game has improved hand extension performance of the stroke patient, and the stroke patient we have tested for do feel tense and tiredness when she stress out her arm after the rehabilitation game is played. This is a good sign, as the game has enhanced the activities of the patient's have strength, providing evidence of the game design effectiveness.
For the effectiveness of the hand balance training from the game “Super Waiter”, by the response of the questionnaire feedback, our physiotherapist's colleagues also generally agree that this game does improve stroke patient's hand balance. Our stroke patient who has been tested for does feel tired after holding her arm at a position after the rehabilitation game is played. This has also shown the hand balancing activities of the patient, providing evidence of the game design effectiveness.
For the effectiveness of the hand strength training from the game “Stop the fire”, by the reaction from our stroke patient, he does feel tired while stressing his both arms after playing the rehabilitation game. Therefore, although the design of the system is not as good as we were expecting, the game does enhance stroke patients hand strength, providing evidence to the effectiveness of the game design.
Other Findings
Besides the findings related to the requirements, there are more findings that would affect the rehabilitation experience:
There are existing technological errors. As the game “Stop the fire” requires the highest technological skills, It does contain technological slips. For example, the sensor of the Ring-Con controller is too narrow, therefore while the stroke patient is not able to extend his/her hand, the system is unable to detect the Ring-Con controller, and the stroke patient is not able to complete the game.
Moreover, Oculus Quest 2 Headset is commented to be too heavy for the player to hold on and play the rehabilitation games, where they may easily get tired after playing the rehabilitation games.
Aims and Objectives
Summary of Achievements
Limitations of Solutions and Methods
The limitation of the project is the VR headset itself, because the Oculus Quest 2 weighs only 454g, which is a very light model in VR headsets, but still makes the patient feel heavy. We can't change the headset weight, but we can change the headset headband to make the head have a better weight balance to make the user feel comfortable.
Figure 17: Oculus Quest 2
Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.
After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.
Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.
The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.
Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.
Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.
| Course Code | Title | Credits | |
|---|---|---|---|
| COMP S321F | Advanced Database and Data Warehousing | 5 | |
| COMP S333F | Advanced Programming and AI Algorithms | 5 | |
| COMP S351F | Software Project Management | 5 | |
| COMP S362F | Concurrent and Network Programming | 5 | |
| COMP S363F | Distributed Systems and Parallel Computing | 5 | |
| COMP S382F | Data Mining and Analytics | 5 | |
| COMP S390F | Creative Programming for Games | 5 | |
| COMP S492F | Machine Learning | 5 | |
| ELEC S305F | Computer Networking | 5 | |
| ELEC S348F | IOT Security | 5 | |
| ELEC S371F | Digital Forensics | 5 | |
| ELEC S431F | Blockchain Technologies | 5 | |
| ELEC S425F | Computer and Network Security | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| ELEC S201F | Basic Electronics | 5 | |
| IT S290F | Human Computer Interaction & User Experience Design | 5 | |
| STAT S251F | Statistical Data Analysis | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| COMPS333F | Advanced Programming and AI Algorithms | 5 | |
| COMPS362F | Concurrent and Network Programming | 5 | |
| COMPS363F | Distributed Systems and Parallel Computing | 5 | |
| COMPS380F | Web Applications: Design and Development | 5 | |
| COMPS381F | Server-side Technologies and Cloud Computing | 5 | |
| COMPS382F | Data Mining and Analytics | 5 | |
| COMPS390F | Creative Programming for Games | 5 | |
| COMPS413F | Application Design and Development for Mobile Devices | 5 | |
| COMPS492F | Machine Learning | 5 | |
| ELECS305F | Computer Networking | 5 | |
| ELECS363F | Advanced Computer Design | 5 | |
| ELECS425F | Computer and Network Security | 5 |
