Pre-Conference Talk by XU Chenshu | Trajectory as Creative Intent: Rethinking Drag-Based Interaction in Generative Visual Editing

Trajectory-based interaction, where users draw spatial paths to guide generative editing, has become a popular paradigm across 2D image editing, video generation, 3D scene editing, and human motion synthesis. We present a unified taxonomy of these methods along four dimensions: output modality, trajectory representation, trajectory grounding, and human-AI agency. A striking pattern emerges: across all methods and modalities, the AI operates strictly as a spatial command executor, with collaborative and model-initiated categories entirely vacant. We identify three systematic gaps: ambiguity is resolved by increasing user burden rather than developing interpretive capacity; pipelines are open-loop; and evaluation rewards geometric precision over creative quality. We discuss directions toward systems where trajectories serve as the starting point of creative dialogue rather than commands to be executed.

This is a Pre-Conference talk for Forty-Third International Conference on Machine Learning (ICML 2026).