The Fairness Tradeoff: Mixture of Experts Modeling to Reduce Misclassification of Minority Classes under Computational Constraints

Abstract

Fitting a model to heterogeneous data incurs a high computational cost for models that have high predictive power. However, for small smart devices, computational capacity is a highly constraining factor. These constraints result in a computational complexity-accuracy trade-off that produces discriminatory results, particularly in applications that use models trained with data sets exhibiting domain shifts that skew towards a majority population, resulting in underrepresented minorities. This domain shift necessitates a model that is able to flexibly fit data in a computationally effective way. MoE is a type of ensemble learning that trains multiple models to master certain subtasks as well as an additional gating model that decides which model can perform the best for a given input. Since the submodels specialize in a certain task, they tend to be simpler than a singular model designed to fit all the domains in the data, which is useful for fitting heterogeneous data. In this paper, I implement a Mixture of Experts (MoE) model with a sparse gating network to mitigate the effects of this trade-off as inducing sparsity has been shown to minimize computational costs. The gating model implemented specifically is designed to use only one expert at a time instead of just a select few. The model was able to achieve high accuracies on image classification tasks for data sets with differing synthesized heterogeneity, however, due to the simplicity of the data set utilized, there was not much of a difference in accuracy when comparing results of a Multi-layer Perceptron (MLP) model. I conclude that it is possible that my specific implementation requires more fine-tuning to obtain the cost-effectiveness observed in other sparse models.