LLM-Driven Automation of Reverse Engineering for Proprietary Network Protocols

Abstract

Proprietary network protocols remain widespread in the Android ecosystem, yet understanding their behavior and security properties still depends heavily on manual reverse engineering. This dynamic creates a fundamental scalability gap: measurement systems can identify such protocols at large scale, but in-depth analysis remains slow, expert-driven, and difficult to extend across large application sets. This thesis presents an LLM-driven, agent-based pipeline for reverse engineering proprietary network protocols from Android applications. The agent is provided with capabilities for static and dynamic analysis tool use within a structured, multi-stage workflow. The pipeline fully replicated 5 of 6 protocol families previously analyzed by experts, providing evidence that it can recover protocol structure and decryption logic with high fidelity. Applied more broadly to 108 Android applications, the pipeline produced 119 protocol analyses covering 88 distinct protocol implementations. Among 63 distinct proprietary protocol implementations analyzed in depth, 61.9% yielded successful protocol analyses, including 33.3% that were decrypted and 28.6% that were fully reconstructed without passive payload recovery. Across all 119 protocol analyses, the pipeline classified 25.2% as passively decryptable, and for those cases, the thesis verified 93.3% were vulnerable to passive decryption. These results show that LLM-driven agent systems can meaningfully expand the scale of reverse engineering. Important limits remain: heavily obfuscated native code, runtime-only behavior, and anti-instrumentation defenses continue to constrain full recovery. Beyond evaluating the pipeline itself, the analysis also shows that insecure proprietary communication remains common, heavily reused through shared SDK infrastructure, and capable of exposing sensitive application and device data at scale.