Embedded face recognition using cascaded structures

During the past few years, there is an increasing demand for smart devices in consumer electronics. These smart devices should be capable of consciously sensing their surroundings and adapting their services according to their environments. Face recognition provides a natural visual interface for such applications and can be embedded into corresponding devices to facilitate context awareness. The continuous growth of computing power brings face recognition within the reach of consumer devices and embedded applications. Compared to traditional face recognition in professional applications, face recognition in embedded/consumer applications is characterized by a large variability of operating environments and limited computation power and image quality. We aim at designing a face-recognition system which has a performance that is competitive to a professional system but has a significantly higher efficiency in terms of computation. In this thesis, we aim at employing multiple algorithms that coordinate with each other for enhanced face-recognition performance while managing the overall complexity. More specifically, we propose new techniques for three major processing stages in face recognition, namely, face detection, facial feature extraction and face identification. At each stage, our major contribution is the design of a number of novel algorithms that are further combined into a cascaded structure. In this cascaded framework, we focus mainly on the following two aspects: (1) design of individual algorithms to meet system requirements, and (2) optimization of algorithm ordering and interfacing to improve the overall system performance. For face detection, we have proposed two pruning detection cascades, where we use fast detectors to quickly discard large non-face background areas and more accurate detectors at succeeding stages to refine the detection results. In this way, both the high detection accuracy and the processing efficiency can be achieved simultaneously. The first cascade is based on various feature detectors, namely, a color-based detector, a feature-geometry-based detector and a neural-network-based detector. The second cascade is based on a set of neural-network ensembles. First, for improved detection accuracy, we propose a novel training technique to form a coordinated ensemble of neu ral networks. Second, for improved detection efficiency, we build a cascade of neural-network ensembles with scalable structures. The approach achieves one of the highest detection accuracies in literature with a significantly reduced computation cost. The proposed structure is also suitable to be implemented in parallelized hardware architectures. For facial feature extraction, we have first developed an improved algorithm of the Active Shape Model (ASM), which extends ASM by using Haarbased local feature modeling. The enhanced modeling enriches the representation power of ASM and leads to doubled convergence capability and a 17% improvement in accuracy. Afterwards, we have developed a cascaded extraction framework for a set of model-based extraction algorithms. We have defined a set of principles to guide the construction of such a framework, which examines the performance relations between adjacent algorithms. As an implementation, we propose a three-algorithm cascade for facial feature extraction, which consists of a sparse graph model, a component-based texture model and a component-based appearance model. These algorithms capture different characteristics of facial features, giving an increasing extraction accuracy coupled with a decreasing convergence. By tuning the performance of each algorithm based on the output statistics of its preceding algorithm, a feature model can be progressively ‘pulled’ to the correct position. The experiments show that our approach is not sensitive to large model deviations and achieves a high extraction accuracy (24% gain compared to ASM). For face identification, we have explored a selective cascade for improved identification performance. The selection cascade gradually reduces the candidate size and derives a customized classification function for each candidate set. We have applied Linear Discriminant Analysis in this framework and illustrated the effectiveness of the approach (23% reduction of identification error). Furthermore, we have investigated a new adaptive feature selection as a more efficient implementation of the cascaded identification. This approach selects a set of the most discriminating features for each person based on the so-called class-specific maximum marginal diversity. According to the selected features, an efficient matching function is defined. Our cascaded algorithm effectively improves the identification of single algorithms and outperforms several wellknown face identification algorithms (e.g. by a reduction of identification error by 18%). We have successfully applied selections of our developed face-recognition techniques in several applications, such as smart user identification in a connected home environment, face recognition for secure biometric identification and face recognition for video surveillance and database retrieval. In several extensive tests, the system has demonstrated competitive performance with respect to accuracy, efficiency and robustness.