Language-driven Video Editing Agents: Human-AI Creative Collaboration Dynamics Across Interface Modalities

Abstract

Despite the importance of video as a medium for communication, video editing is a skill with a high barrier to entry. To democratize access to content creation, we introduce the Video Editing and Reasoning Agent (VERA), an AI agent that can edit videos based on human instructions. VERA autoregressively conditions on an encoded video representation, viewer context, and command during inference to complete various editing tasks. We show that by equipping the agent with a set of core video editing skills, VERA can reason about combining one or more of them to produce complex edits. To facilitate interaction with the agent, we contribute two interfaces: voice with one-at-a-time feedback and text with aggregated history. We evaluate the VERA system with human participants (N=8) and find statistically significant evidence of the priming effect, in which the initial interface participants are exposed to shapes their subsequent interactions with and perception of the agent. We show that participants who began with the voice-based interface tended to issue commands that were broader in scope and more abstract than those who started with the text-based one. Voice users are also more likely to perceive the agent as a collaborator as opposed to an assistant. Our study's results contribute to the development of human-agent creative collaboration systems that help bring creative vision to life.