School of Science and Technology 科技學院
Computing Programmes 電腦學系
| Programme | Bachelor of Computing with Honours in Internet Technology |
| Supervisor | Dr. Henry LEUNG Man Fai |
| Areas | Innovative Applications |
| Year of Completion | 2021 |
The project aim of this project is to develop a mobile application with a medication recognition and information system and reminding system in order to assist people with low vision to take and purchase the over the counter medicine correctly.
A. Medication Recognition and Information System:
The medication information system is to classify the medicine package through the Android mobile phone's camera. After the user capture the image through the application, the image will be sent to the server for recognition and retrieve the medication information from the database. The obtained information will be sent back to the user's application. After that, the system will provide the organized medicine information to the user including drug name, dosage, efficacy, caution etc.
B. Reminding System:
The reminding system in this project is a voice-based reminder to remind the user to take the correct medication on time. The user with vision impairment can set the schedule of reminder. Also, the user can register the medicine that they need to take on time by using the medication recognition system. By the time of taking medicine, the system will remind the user to take that registered medication.
In order to achieve the aim, there are a number of objectives in my project as follows:
A. Convolutional Neural Network:
Convolutional Neural Network (CNN) is a technology that provide high performance for image classification. It is applied in different areas such as handwritten character classification, face recognition and diagnosis disease from X-ray images, etc. Accounting to O’Shea and Nash [13], Convolutional Neural Network is composed by these three kinds of layer: convolutional layers, pooling layers, and fully-connected layers. Albawi, Mohammed and Al-Zawi [14] stated that the performance of the CNN model is affected by the number of layers in the model and the number of layers increase the training time of the model.
Transfer learning is introduced to address this problem. Hussain, Bird and Faria [15] mentioned that transfer learning is a method that repurpose the pre-trained model from one task to other tasks. Shu [16] presents that transfer learning is a time saving method to obtain a better performance of the model with a small dataset.
B. VGG Neural Network:
Simonyan and Zisserman [17] developed the VGG neural network for image classification. VGG16 and VGG19 are commonly used model for transfer learning. Regarding the VGG16 model, there are 13 convolutional layers plus 3 fully-connected layers in the network. Compared with VGG16, there are 16 convolutional layers plus 3 fully-connected layers in the VGG19 model.
C. ResNet Neural Network:
He, Zhang, Ren and Sun [18] proposed the residual learning technique for training a deeper model and developed the ResNet models. It is stated the residual learning technique can overcome the degradation problem during the training of deep neural network. The ResNet50 model is another frequently used model for transfer learning. The architecture of ResNet50 model is shown below.
| Technology | Description |
| Tensorflow, Keras | Tensorflow and Keras provided a powerful function for transfer learning and image preprocessing. These libraries are used in training an image classification model and the server for preprocessing the received image data. |
| Optical Character Recognition | Google Vision provides an accurate Optical Character Recognition API for the user to extract the text from the image and return a json object with the text extracted from the image. |
| SequenceMatcher | SequenceMatcher from difflab is very useful for comparing the similarity of two string in Python. This will be used in the server. |
| Flask Web-application Framework | Flask is an easy to use framework for developing a server in Python for handling incoming HTTP requests. |
| MongoDB | MongoDB Atlas is a NoSQL cloud database for storing JSON document. |
| Android | Android is an operating system for a mobile phone. The application will be installed in this environment. |
| Smartphone | People with low vision can use our application through the Android smartphone. They can capture image for recognition through the camera and interact with the application by tough screen and screen reader application. |
| Android BroadcastReceiver, AlarmManager | Two Android library for receive the signals of reboot and time for the reminding. |
| Okhttp | Okhttp is an Android library for sending the HTTP request to the server. |
| CameraX | Okhttp is an Android library for calling the camera function in the Android devices. |
Table 1: Key Technologies
A. Hierarchical Structure Diagram:
Figure 1: Hierarchical Structure Diagram
The whole system in this project is mainly divided into three parts: Android application, Flask server and MongoDB database. There are two sub-system in the Android application which are Reminding system and Recognition system. In the Reminding system, user can set the reminder to remind them to take the medication. The recognition system can provide the medication information after the medicine is identified. Both of these sub-systems require to call the camera intent (The designed and easy to use camera for people who have an issue of vision impairment) to take the photo of the medicine. After taking the photo, the photo will be sent to the server by sending a HTTP request with the Base 64.
Regarding the Flask server, there are three functionalities: handling the incoming HTTP request, getting the medication information from database, recognizing the medication. The server will send the query to the database by using the MongoDB API call. In the recognition system, the image classification model, and Optical Character Recognition (OCR) will be used. The server will call the Google Vision API for the OCR function.
With respect to the MongoDB database, the medicine information will be stored in the cloud database including the drug name, dosage, caution, ingredient, and efficacy. MongoDB is a NoSQL cloud database so that the information is stored in a JSON document.
B. User Interface Design:
Figure 2: Location of help button
Figure 3: Simple camera design
Figure 4: Info table
Figure 5: Reminder list
Figure 6: Reminder setting form
Figure 7: Reminder screen
Performance Evaluation
The aim of the performance evaluation testing is to evaluate the performance of the recognition system. Different factors will be measured in this testing including the accuracy, precision, recall and F1-score.
| Class | Precision | Recall | F1-score |
| Clarityne | 1.00 | 0.96 | 0.98 |
| Extra Fast Coltalin | 1.00 | 0.99 | 0.99 |
| Cortal (Adults) | 1.00 | 0.97 | 0.98 |
| DecaughII | 1.00 | 0.92 | 0.96 |
| Mucinex | 1.00 | 0.95 | 0.97 |
| Ulance | 1.00 | 0.95 | 0.97 |
| Out of distribution (i.e., not recognized) | 0.66 | 1.00 | 0.80 |
Table 2: The evaluation result: Precision, Recall and F1-Score
Usability Evaluation
The aim of the usability testing is to evaluate the usability of the application for low vision user. Office of the Government Chief Information Officer [19] provided a Best Practice Checklist for developers to evaluate the accessibility of the application. The requirements in the checklist are the important features for people with disability including visually impaired to access and use the application. The checklist for Baseline level accessibility application will be used to evaluation the usability of the application.
In order to check whether the application fulfill the Baseline level for visually impaired people to use or not, the procedure of the testing is that the application will be checked with each requirement in the Best Practice Checklist in Appendix A. Some requirements are pre-marked as “Skip”. It means that the requirements are not needed to check. Moreover, the irrelevant requirements can be mark as “N/A”.
Regarding the evaluation result, all related requirements are fulfilled. Therefore, the Baseline level of accessibility application is achieved. Hence, the usability of the application is tested to be enough for low vision user.
This project aims to develop a system for assisting the low vision people to take the over the counter medication correctly. The visually impaired people are facing different problems when they need to take the over the counter medication including taking wrong medicine, wrong dosage. By using our system, the low vision user should be able to recognize the over the counter medication, receive the correct medication information and take the medication on time.
There are seven objectives are mentioned. The purpose of these objectives is for achieving the aim which mentioned above. All objectives are satisfied. By evaluating the system, the result shows that the system can provide a good performance for recognizing the over the counter medicine and the application have enough accessibility for low vision user. Therefore, the aim of the project is also achieved.
Suggestions on future work
After obtaining the result of the evaluation process and considering the limitations, some future works can be done to improve the system.
First of all, a classifier can be added to the system to identify whether the image contain two or more medications. If there are two or more medications in the picture, the system should require the user to capture another photo. More research should be done to find or develop the method for the classifier. By adding this classifier, the limitation of prediction error for the image that contain two medications can be improved. Hence, the system will become more reliable for the low vision user to recognize the medication and get the corresponding information.
Secondly, regarding the usability of the Android application, the usability can be further improved to make the application more suitable for low vision user to use. Beside the Baseline level checklist, Office of the Government Chief Information Officer [19] also provided a Best Practice Checklist for Advanced level of accessibility application. That checklist provides some advanced requirements for accessibility application to make the application more convenient for people with disability to use. Therefore, the usability of the application can be further improved in the future.
Finally, more medications should be added to the database and the image classification model. There are many over the counter medications that people can purchase at supermarket or drugstore in Hong Kong. Therefore, we should include more medications in our system to make the low vision user feel more convenient and confidant to take the over the counter medications correctly.
Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.
After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.
Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.
The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.
Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.
Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.
| Course Code | Title | Credits | |
|---|---|---|---|
| COMP S321F | Advanced Database and Data Warehousing | 5 | |
| COMP S333F | Advanced Programming and AI Algorithms | 5 | |
| COMP S351F | Software Project Management | 5 | |
| COMP S362F | Concurrent and Network Programming | 5 | |
| COMP S363F | Distributed Systems and Parallel Computing | 5 | |
| COMP S382F | Data Mining and Analytics | 5 | |
| COMP S390F | Creative Programming for Games | 5 | |
| COMP S492F | Machine Learning | 5 | |
| ELEC S305F | Computer Networking | 5 | |
| ELEC S348F | IOT Security | 5 | |
| ELEC S371F | Digital Forensics | 5 | |
| ELEC S431F | Blockchain Technologies | 5 | |
| ELEC S425F | Computer and Network Security | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| ELEC S201F | Basic Electronics | 5 | |
| IT S290F | Human Computer Interaction & User Experience Design | 5 | |
| STAT S251F | Statistical Data Analysis | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| COMPS333F | Advanced Programming and AI Algorithms | 5 | |
| COMPS362F | Concurrent and Network Programming | 5 | |
| COMPS363F | Distributed Systems and Parallel Computing | 5 | |
| COMPS380F | Web Applications: Design and Development | 5 | |
| COMPS381F | Server-side Technologies and Cloud Computing | 5 | |
| COMPS382F | Data Mining and Analytics | 5 | |
| COMPS390F | Creative Programming for Games | 5 | |
| COMPS413F | Application Design and Development for Mobile Devices | 5 | |
| COMPS492F | Machine Learning | 5 | |
| ELECS305F | Computer Networking | 5 | |
| ELECS363F | Advanced Computer Design | 5 | |
| ELECS425F | Computer and Network Security | 5 |
