School of Science and Technology 科技學院
Computing Programmes 電腦學系
| Programme | Bachelor of Science with Honours in Computing |
| Supervisor | Dr. Andrew Lui |
| Areas | 3D Reconstruction and Computer Vision |
| Year of Completion | 2011 |
The aim of the project is to develop a low cost, portable 3D facial scanner based on structured light. Facial anthropometries will be obtained from the reconstructed 3D model for the further investigation of mask fitness test. A 3D facial scanner is a setup which allows human face to be scanned and delivered in the form of 3 dimensional models. Structured light (SL) are specific patterns that acts as the coding parameters for 3D model reconstruction. Anthropometries are measurements of facial features and mask fitness test (MFT) is a standard for selecting suitable mask for users. Typically, anthropometric data are measured to assist the process of MFT. In this project, a portable 3D facial scanner will be developed to obtained anthropometries required for MFT. Another project of MFT will make use of the anthropometries to select suitable mask for target users.
The project objectives are described in detail below:
3D scanning is developing rapidly due to hardware advances in recent years. It has been widely employed in medical and product engineering, movies and games production, cultural heritage reconstruction and homeland security. Although plane images could allow 3D model reconstruction, parameter means algorithms are usually used due to its higher accuracy. Most of them use laser scan where the tools are usually large and expensive. Some smaller size 3D mega capturers may cost up to hundred thousand still. In fact, 3D scanning could be applied to broader situations. As a break through, a brand new algorithm which uses structured light (SL) as scanning parameter explored. The tools involved are relatively less expensive but the technology is yet to be developed. In this project, we will investigate and develop the algorithm of SL scanning to create 3D facial model in order to obtain facial anthropometry for mask fitting test (MFT). MFT is a test for choosing suitable mask for human. The importance of choosing a mask with optimal fitness could be hazardous to health since failing may result in contamination of air borne diseases and dust particles in clinics and industries respectively.
Facial anthropometries are measurements that represent facial features. There are typically 10 anthropometric data critical for measuring the fitness of respirator according to Han (Han and Choi, 2003) shown below:
Han proposed that (9)Bitragion-Menton Arc and (1)Bizygomatic Breadth would be the two parameters that considered for Korean face seal leakage. However, few researches have been conducted basing on facial characteristics of Chinese according to Yang (Yang, 2007) who investigate the differences of facial anthropometric dimensions between Chinese and Americans. Result indicates a necessity to construct new respiratory fit test panels for Chinese. In this case, MFT for Asians is an issue that worth to investigate.
3D reconstruction part involves the extraction and analysis of 3D information. In general, the extraction requires hardware support while the analysis is done entirely in software. Extraction of data is performed using the digital fringe projecting-capturing system. Among different types of 3D scanning algorithms, the four-step phase shifting approach of phase wrapping and a simple but adequate technique of flood fill phase unwrapping has been employed. 3D reconstruction is achieved using phase-plane triangulation.
While anthropometry extraction can be done manually, a novel automatic anthropometry extraction system has been introduced. The principle lies on the position estimation of facial features and recognition of features mathematical specifications. Data acquisition has been greatly facilitated using the system. Evaluation has been conducted using a 3D mega capturer as golden standard. The anthropometries of both scanners are compared to determine the reliability of my scanner. The flow of the entire project is shown below:
The following shows the layout of the session which will be described briefly:
Extraction of 3D information in the projector-camera system;
Analyze and transform extracted information in the wrapping phase and unwrapping phase;
Reconstruction of 3D model;
Extraction of anthropometries;
Evaluation of reliability and improvements.
A projector-camera system is the main setup of a 3D scanner. It is responsible for the production and extraction of information required for 3D reconstruction. Our setup comprises of a web-camera, a pocket projector and a notebook computer as shown below:
A four-step phase-shifting algorithm has been adopted due to the ease of implementation and hardware facilitation. Four phase-shifted fringe patterns are casted onto a human face using a projector. Surface characteristic of face causes the distortion of patterns which hides 3D information and the corresponding fringe images are captured accordingly. The images are then sent to a program for phase calculation. The following shows the design of the digital fringe projecting-capturing system:
The fringe image analyzes part is done in software and can be divided into two parts: the wrapping phase and the unwrapping phase. Wrapping phase involves the application of four-step phase-shifting approach. The following shows a wrapped phase map obtained:
In the unwrapping phase, the phase information of the wrapped phase map will be processed to remove the 2π discontinuity. Generally, phase information of a pixel is compared with that of the neighboring. The details of the phase analysis is shown below:
3D reconstruction is achieved through phase-plane triangulation suggested by Zhang (Zhang, 2002). The resulting 3D facial model is in the form of point clouds. Since our ultimate goal is to obtain anthropometric data from human face, other points such as background and shadows are considered as redundant. The blue screen technique is applied where the background color and shadows of a selected range are removed directly. Required volume of point clouds can also be selected manually using a cube mask where point clouds outside the specified volume of cube will be removed. To convert the point clouds into a real water tight model, algorithm of quad mesh has been applied. The result of a 3D model is shown below:
A preliminary automatic anthropometry extraction system has been implemented to assist data acquisition. The approach is based on estimation of relative positions of facial features and the features themselves have a mathematical specification. Take nose protrusion as an example, nose tip is generally the highest protruding feature of a face. The recognition of the nose tip then allows the location of the nose base (subnasale) where it should be the deepest point right below the nose tip. The relative positions of other facial features such as the nose root, chin position, bizygomatic breadth can also be found accordingly. Most facial features can be found within a 0.5cm range margin. The remaining error can be fine-tuned manually. The following shows a result of automatic anthropometry extraction.
Evaluation has been conducted with a 3D mega capturer as golden standard. Our scanner uses an image resolution of 1280 x 1024 during the evaluation. Seven human subjects were sampled using my 3D scanner and the 3D mega capturer within one hour to ensure sample consistency. Since our goal is to apply this 3D scanning technology on mask fitting, the comparison should be made on the facial anthropometries. The following shows the root-mean-square (RMS) error found in our 3D model compared with that of the mega one:
| Facial features | Root- Mean-Square Error (cm) | Minimum Error (cm) | Maximum Error (cm) |
|---|---|---|---|
| 1 Menton-to-Nasal | 0.38 | 0.05 | 0.85 |
| 2 Menton-to-Subnasal | 0.32 | 0.06 | 0.57 |
| 3 SubNasal-to-NasalRoot | 0.46 | 0.06 | 0.77 |
| 4 Nose Width | 0.02 | 0 | 0.06 |
| 5 Lip Length | 0.23 | 0.01 | 0.39 |
| 6 Nose Protrusion | 0.00 | 0 | 0.01 |
| Average Error | 0.24 | 0.03 | 0.44 |
The worst performing feature was SubNasal2NasalRoot with RMS of 4.6 mm while an average RMS of 2.4 mm resulted. The minimum and maximum error were also compared which reveals most errors were contributed by one or two particularly poor measurements. The anthropometry comparison of each human subject in details is shown below:
| Features | Subject #1 | Subject #2 | Subject #3 | Subject #4 | Subject #5 | Subject #6 | Subject #7 |
|---|---|---|---|---|---|---|---|
| Proto Gold | Proto Gold | Proto Gold | Proto Gold | Proto Gold | Proto Gold | Proto Gold | |
| 1 | 12.4 12.45 | 11.34 11.26 | 12.48 13.33 | 13.08 12.79 | 11.34 10.9 | 12.50 12.59 | 12.42 12.36 |
| 2 | 7.00 7.07 | 6.62 6.68 | 7.13 7.70 | 7.48 7.39 | 7.04 6.95 | 6.94 7.50 | 7.46 7.28 |
| 3 | 5.47 5.53 | 4.99 4.85 | 5.38 5.79 | 6.29 5.78 | 4.55 4.20 | 6.22 5.45 | 5.78 5.21 |
| 4 | 4.17 4.11 | 4.74 4.74 | 3.77 3.76 | 3.93 3.93 | 3.84 3.84 | 4.07 4.07 | 4.34 4.34 |
| 5 | 5.81 5.80 | 5.51 5.62 | 5.20 5.28 | 5.18 5.57 | 5.05 4.67 | 4.24 4.50 | 5.73 5.68 |
| 6 | 1.55 1.54 | 1.57 1.57 | 1.90 1.90 | 1.60 1.60 | 1.59 1.59 | 1.95 1.95 | 2.09 2.09 |
The worst performing feature was SubNasal2NasalRoot with RMS of 4.6 mm while an average RMS of 2.4 mm resulted. The minimum and maximum error were also compared which reveals most errors were contributed by one or two particularly poor measurements.
Evaluation has been conducted using a 3D mega capturer as golden standard. Six anthropometries of seven human subjects has been sampled and extracted by my scanner and the mega one for comparison. The scanning process is taken within one hour to reduce inconsistency. Result shows a maximum RMS error of 4.6 mm while an average RMS error of 2.4 mm. This section aims to account for the error and possible solutions.
The result of my scanner is not surprising since no accurate calibration of both camera and projector have been performed. As mentioned in section 4.2.1, the lens distortion error is only roughly solved by using a zoom in method. Although the zoom in scale of this evaluation has been increased into 240%, lens distortion still exists.
Actually, there were visible head movements found in the image sequences. It may be due to the relatively long scanning process. After all, human are movable objects, involuntary head movement due to body balancing or others reasons are reasonable.
The minimum and maximum error were compared which reveals most errors were contributed by one or two particularly poor measurements. The error possibily stem from the lens distortion, but other factors such as the manual extraction of anthropometries may also contribute to it. While the anthropometry of the 3D mega capturer part is extracted by a student helper, anthropometry of my scanner is extracted by me which may cause inconsistency.
The extraction error can be solved by using the same person on anthropometry extraction while solutions of the moving subject error have been discussed in previous chapters (the projector-camera synchronization), I am not going to talk about it here. More will be talk on the lens distortion error since it is possibly the main contribution of error.
Usually, lens distortion of cameras and projectors will result in significant measurement errors. To allow highly accurate measurements, calibration of both camera and projector is needed.
For camera calibration, the factorized approach originally proposed by Zhang (Zhang, 2000) can be adopted. It involves photo capturing of a known sized checkerboard in two or more orientations. The photos are then analyzed by a calibration program which yields a calibration matrix. Photo editing can be performed using the matrix as long as the intrinsic parameters remain unchanged.
For projector calibration, the novel approach of Douglas (Douglas and Gabriel, 2009) can be adopted. It is in fact a modification of the factorized approach of Zhang. It involves projecting a checker board image onto a calibration whiteboard with four printed checkerboard corners. Again, two or more orientations of views are recorded and analyzed.
This project investigates the research and development of a low cost, portable 3D scanner with high speed and accuracy using structured light. Resulted 3D model will be applied to assist on the extraction of anthropometries for mask fitting test.
Phase-shifted structured light has been chosen as the scanning parameter and the four-step phase shifting approach has been adopted. The extraction and analysis of 3D information involves the calculation of relative intensity and phase reconstruction in the wrapping phase. Further conversion of information in phase unwrapping is performed using the simple but adequate algorithm of flood fill. The reconstruction of 3D model is achieved using phase-plane triangulation.
The anthropometry extraction is achieved through the development of an automatic anthropometry extraction system. Data acquisition is greatly facilitated. Further improvements can be achieved following the advances of the 3D model.
Evaluation has been conducted using a 3D mega capturer as gold standard. The result is acceptable with an average root-mean square error of 0.24 cm.
In conclusion, completing this project can provide a more economical alternative for 3D scan. It is feasible for 3D scanners to be available by mass population and even in mobile devices. MFT can also be performed in a cheaper and more effective way consequently.
Copyright Li Hon Lam Adam and Andrew Lui 2011
Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.
After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.
Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.
The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.
Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.
Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.
| Course Code | Title | Credits | |
|---|---|---|---|
| COMP S321F | Advanced Database and Data Warehousing | 5 | |
| COMP S333F | Advanced Programming and AI Algorithms | 5 | |
| COMP S351F | Software Project Management | 5 | |
| COMP S362F | Concurrent and Network Programming | 5 | |
| COMP S363F | Distributed Systems and Parallel Computing | 5 | |
| COMP S382F | Data Mining and Analytics | 5 | |
| COMP S390F | Creative Programming for Games | 5 | |
| COMP S492F | Machine Learning | 5 | |
| ELEC S305F | Computer Networking | 5 | |
| ELEC S348F | IOT Security | 5 | |
| ELEC S371F | Digital Forensics | 5 | |
| ELEC S431F | Blockchain Technologies | 5 | |
| ELEC S425F | Computer and Network Security | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| ELEC S201F | Basic Electronics | 5 | |
| IT S290F | Human Computer Interaction & User Experience Design | 5 | |
| STAT S251F | Statistical Data Analysis | 5 |
| Course Code | Title | Credits | |
|---|---|---|---|
| COMPS333F | Advanced Programming and AI Algorithms | 5 | |
| COMPS362F | Concurrent and Network Programming | 5 | |
| COMPS363F | Distributed Systems and Parallel Computing | 5 | |
| COMPS380F | Web Applications: Design and Development | 5 | |
| COMPS381F | Server-side Technologies and Cloud Computing | 5 | |
| COMPS382F | Data Mining and Analytics | 5 | |
| COMPS390F | Creative Programming for Games | 5 | |
| COMPS413F | Application Design and Development for Mobile Devices | 5 | |
| COMPS492F | Machine Learning | 5 | |
| ELECS305F | Computer Networking | 5 | |
| ELECS363F | Advanced Computer Design | 5 | |
| ELECS425F | Computer and Network Security | 5 |
