School of Science and Technology 科技學院
Computing Programmes 電腦學系
| Programme | Bachelor of Computing with Honours in Internet Technology |
| Supervisor | Prof. Vanessa Ng |
| Areas | Augmented Reality Applications |
| Year of Completion | 2020 |
| Award | YPEC 2020 Best Innovation Award (Sub Degree +Undergraduate section) |
The aim of this project is to encourage dyslexia children to learn chinese characters in a self-directed way through gamification of training exercise and machine learning.
The main objective of the project is to develop a mobile application that can analyze the dyslexia children's performance when they are using the application and provide a suitable response.
The project has also defined a number of sub-objectives as follows:
Our project is designed to encourage Dyslexia children to learn chinese characters in reading and writing. In order to improve the user experience, we decided to apply the prototype and experimental methodologies.
For prototype methodology, we prefer to build a prototype that can achieve the user needs. We will first develop a basic structure of a mobile app from our previous research on Dyslexia, then we will build a prototype from the structure and improve the app from the user response. After applying this methodology, it can help us rapidly develop the prototype which is suitable for the user. Also, we can detect the bug and reduce part of them before we begin to develop the mobile app.
For experimental methodology, we want to test whether our app can improve user reading and writing on chinese characters.We will separate two groups of users, and one group will use our mobile app and another group will not use our mobile app.After one hour, we will give a test to them, from the test result can show that our mobile app can improve or not. After we get the result, we can get data of what will affect the result and we can improve our game and logic from these datasets.
Techniques and technologies used:
Unity
Game engine for developing our solution
Microsoft Azure cloud platform
For machine learning
Microsoft Azure Custom Vision Service
For machine learning (recognizing normal Chinese character)
Google Cloud Vision API Service
For machine learning (recognizing normal Chinese character)
Vuforia
Game Engine for Augmented Reality development environment
Figure 1: Overview System Design
Our solution involves three major components, an AR Game component, a camera recognition component and a handwriting recognition and training component. The game component on the left hand side involves Vuforia engine and its web service, the camera recognition and handwriting recognition component on the right hand side involves different techniques like cloud service and machine learning algorithm. Note that the game part and camera part require permission in order to access the mobile camera and use it as the input.
Main Screen
There are three buttons in the main screen which allows the dyslexia children access to different components of our application. The game console button, camera button and the book icon represent the augmented reality game component, camera component and training component respectively.
Figure 2: Main Menu
Figure 3: Key Function – Stroke Detection
Training Component
Figure 4: Training Menu
Figure 5: Search Mode
Figure 6: Random Mode
When the user enters the training part, two different modes of training will be allowed for the user to choose, the two modes are shown in Figure 4.
The first mode is search mode (搜尋文字), once it is clicked it will enter the searching mode which allows user to look for specific character as writing exercise.
The second mode is random mode (隨機文字), it will assign a random character to the user to write once it is clicked.
Camera Component
Figure 7: Game Component – Photo Capturing
Figure 8: Game Component – Analyzed Result
Figure 7 shows the photo capturing mode of the camera component. Once the user enters the camera, it will open the mobile camera (permission required). Then the user will be able to capture the handwriting that he/she wanted to recognize.
Figure 8 shows the analyzed result from the two cloud machine learning models. We have organized the output of the result and display the result back to the user.
Note that if the user wrote the character wrong, it will display the correct writing of that specific character. Also, it will indicate whether the components of the character are horizontally swapped or vertically swapped.
Game Component
Figure 9: Game Menu
Figure 10: Unclear level
Figure 11: Clear level
Everytime when the user enters a level, it will start the mobile camera first, then it will look for the Augmented Reality Image Target.
In Figure 10, there are two image targets placed on the table. Once the application recognizes the image targets, it will generate the object (puzzle) and map it to the corresponding image target.
Then the user can play with the image targets by moving it. And the corresponding puzzle will also be moved in the camera.
Once the user built the correct Chinese character with the puzzle. A prefab (object model) will be generated to indicate that the user has completed this stage.
User evaluation
We invited 3 child education experts and 1 special education expert and conducted user evaluation towards them.
The questionnaire result shows the positive point of view toward the implementation of augmented reality and machine learning with special education. According to Figure 5.2, most of the interviewees agreed the usefulness of AR and machine learning toward special education (response answer ranged from neutral to strongly agree).
Also, it makes comparison between the traditional learning style and our solution and it shows a positive result towards our solution.
Figure 12: Featured Question and Response 1
Figure 13: Featured Question and Response 2
Figure 14: Featured Question and Response 3
The aim of this project has been achieved partially. We developed a solution which integrates the Augmented Reality and Machine Learning technologies successfully. The training component and game component of the application can assign exercise to the dyslexia children and make responses based on their performance. The camera component can be used as a support tool for the children to check his/her handwriting.
Future work
We suggest contacting different schools, Special Education Needs Association in the future to access different dyslexia children so that we may acquire their handwriting work, which can be used to train the Microsoft model.
We also suggest to perform a model evaluation again on the Microsoft model after new datasets have been used to train the model.
Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.
After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.
Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.
The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.
Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.
Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.
| Course Code | Title | Credits | |
|---|---|---|---|
| COMP S321F | Advanced Database and Data Warehousing | 5 | |
| COMP S333F | Advanced Programming and AI Algorithms | 5 | |
| COMP S351F | Software Project Management | 5 | |
| COMP S362F | Concurrent and Network Programming | 5 | |
| COMP S363F | Distributed Systems and Parallel Computing | 5 | |
| COMP S382F | Data Mining and Analytics | 5 | |
| COMP S390F | Creative Programming for Games | 5 | |
| COMP S492F | Machine Learning | 5 | |
| ELEC S305F | Computer Networking | 5 | |
| ELEC S348F | IOT Security | 5 | |
| ELEC S371F | Digital Forensics | 5 | |
| ELEC S431F | Blockchain Technologies | 5 | |
| ELEC S425F | Computer and Network Security | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| ELEC S201F | Basic Electronics | 5 | |
| IT S290F | Human Computer Interaction & User Experience Design | 5 | |
| STAT S251F | Statistical Data Analysis | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| COMPS333F | Advanced Programming and AI Algorithms | 5 | |
| COMPS362F | Concurrent and Network Programming | 5 | |
| COMPS363F | Distributed Systems and Parallel Computing | 5 | |
| COMPS380F | Web Applications: Design and Development | 5 | |
| COMPS381F | Server-side Technologies and Cloud Computing | 5 | |
| COMPS382F | Data Mining and Analytics | 5 | |
| COMPS390F | Creative Programming for Games | 5 | |
| COMPS413F | Application Design and Development for Mobile Devices | 5 | |
| COMPS492F | Machine Learning | 5 | |
| ELECS305F | Computer Networking | 5 | |
| ELECS363F | Advanced Computer Design | 5 | |
| ELECS425F | Computer and Network Security | 5 |
