An Intensive Analysis of Clustering and Dimensionality Reduction: Applying the Expectation-Maximization Algorithm and Principal Component Analysis to Classify Handwritten Digits

Abstract

This thesis is concerned with a deep understanding of the clustering Expectation-Maximization (EM) Algorithm under different assumptions and given different inputs. There are three assumptions governing the values of the covariance matrix involved in the implementation of the EM algorithm under a mixture of Multivariate Gaussian models. The performance of the EM algorithm under these three assumptions is closely examined using data from the Mixed National Institute of Standards and Technology (MNIST) database. In this database are images of handwritten digits, 0-9.Each image is a member of one of these 10 groups, and the performance of the algorithm is judged based on how well it can find these unobserved labels and cluster the digits into groups. Furthermore, the dimensionality reduction technique, Principal Component Analysis, is applied at times throughout the paper, to study the effects of its lossy compression and how the dimensionality of the data influences the performance of the EM algorithm. After a rigorous study of the behavior of the EM algorithm significant results were established, including the choice of the spherical covairance matrix as the overall best performing assumption.