Extending AccelTran to Vision-Language Tranformers

Abstract

The Transformer model has been pivotal in fields like Natural Language Processing and Computer Vision, among others. Recent efforts to combine single-mode data into more complex multimodal models have demonstrated impressive results. However, using these large models, particularly on resource-constrained devices can be infeasible. To address this gap, our project focuses on adapting the AccelTran framework, initially created for language-based transformers, to work with multimodal data. We have modified the AccelTran framework to support Vision Transformers, which includes adding a new "Patchifier" module to the accelerator. We have also modified the DynaTran module to support mid-inference adaptation. Additionally, we have updated the existing Python simulator to support vision operations and vision-language models.