Liteflow— Building a Lightweight Data Pipeline Management System

Abstract

Building extract-transform-load data pipelines is an essential component of modern data-driven programming. They enable the extraction of raw data, transform it into usable formats, and load it into a target system for analysis/storage. However, as these pipelines become more complex, managing them can become challenging. Airflow is a popular platform for managing data pipelines, but it has several shortcomings, including numerous dependencies, a steep learning curve, and limited compatibility with some operating systems. To address these issues, we introduce Liteflow, a lightweight platform that aims to provide similar functionalities as Airflow while simplifying the data pipeline development process. Liteflow includes a minimal API and a clear outline for developing pipelines. By reducing the number of dependencies and providing a simpler learning curve, Liteflow streamlines the process of building and managing ETL pipelines, making it a suitable alternative to Airflow for beginners.