Order Matters: 3D Shape Generation from Sequential VR Sketches

Yizi Chen*1, Sidi Wu*1, Tianyi Xiao1, Nina Wiedemann1, Loic Landrieu2
1ETH Zurich 2LIGM, ENPC, IP Paris, Univ Gustave Eiffel, CNRS
* Equal contribution
Paper Code

Abstract

VR sketching lets users explore and iterate on ideas directly in 3D, offering a faster and more intuitive alternative to conventional CAD tools. However, existing sketch-to-shape models ignore the temporal ordering of strokes, discarding crucial cues about structure and design intent. We introduce VRSketch2Shape, the first framework and multi-category dataset for generating 3D shapes from sequential VR sketches. Our contributions are threefold: (i) an automated pipeline that generates sequential VR sketches from arbitrary shapes, (ii) a dataset of over 20k synthetic and 900 hand-drawn sketch-shape pairs across four categories, and (iii) an order-aware sketch encoder coupled with a diffusion-based 3D generator. Our approach yields higher geometric fidelity than prior work, generalizes effectively from synthetic to real sketches with minimal supervision, and performs well even on partial sketches. All data and models will be released open-source.

Overview

Overview Image

An input VR sketch is tokenized into a sequence of points organized along ordered strokes. Each 3D point is encoded using 3D Fourier features and an MLP, while stroke and point indices are encoded with 1D Fourier features followed by a linear projection. The resulting embeddings are summed and passed through a lightweight BERT encoder. The encoded token sequence is then used to condition SDFusion, a diffusion-based 3D shape generation model.

Our results

Overview Image

Our approach generates shapes that are detailed, structurally accurate, and topologically faithful to the target geometry.

Unseen class

Overview Image

Results on sketches depicting object categories not present in the training data, including bottles, lamps, and cars from ShapeNet, and monitors, toilets, and beds from ModelNet. Despite the domain shift, our model generally produces plausible shapes aligned with the sketch intent.

Free hand sketches

Overview Image

Our model generalizes well to free-hand sketches drawn without any reference shape for airplanes, chairs/sofas, tables, and cabinets, producing detailed and plausible reconstructions that reflect the user's intent.

BibTeX

@article{Order Matters: 3D Shape Generation from Sequential VR Sketches,
  author    = {Yizi Chen, Sidi Wu, Tianyi Xiao, Nina Wiedemann, Loic Landrieu},
  title     = {Order Matters: 3D Shape Generation from Sequential VR Sketches},
  journal   = {Arxiv},
  year      = {2025},
}