DragMesh: One-Drag 3D Joint Motion Generation on Existing Assets

Research Background

Recent advances in 3D generation (Mesh, NeRF, 3D Gaussian Splatting) have made creating complete 3D objects easier. However, most articulated 3D generation methods focus on building entire movable assets from scratch — geometry, part segmentation, and joint structure together. In practice, users often already possess a 3D model (e.g., a cabinet, microwave, or drawer) and only need to articulate a specific part. Re-generating the whole object for each interaction wastes computation on static regions. DragMesh addresses this by keeping the existing asset and performing kinematic inference and motion generation solely for the target movable part.

Key Challenges

Challenge 1: A single drag does not directly reveal the kinematic structure. The system observes drag position and direction, but must infer joint type (revolute vs. prismatic), axis, and origin. Errors here produce physically implausible motion even if the animation looks smooth.

Challenge 2: Local motion must satisfy kinematic constraints while remaining lightweight. Per-object motion model training or optimization is too slow for real-time interaction. A reusable motion prior that generates full trajectories from one drag while respecting joint constraints is needed. Additionally, mesh (good for structure/rigid transforms) and 3DGS (good for rendering) require a shared motion representation.

Core Method

Kinematics Prediction (KPP-Net)

KPP-Net takes object geometry, movable-part mask, and user drag as input. It uses dual-stream feature encoding: a global branch for overall geometry and drag context, and a local branch focused on the target part. Separate prediction heads output joint type (classification), axis direction (regression), and joint origin (regression). This provides explicit kinematic priors for downstream generation.

Trajectory Generation (DQ-VAE)

Motion is represented as 8-dimensional dual quaternions per frame — avoiding gimbal lock (unlike Euler angles), unifying rotation and translation (unlike axis-angle), and being more compact than full transformation matrices. A non-autoregressive Transformer generates a 16-frame sequence in parallel, preventing error accumulation and suiting real-time use. Joint features are injected continuously across Transformer layers via FiLM conditioning, and a Kinematic Correction Module post-processes decoder output to enforce joint constraints.

Cross-Modal Semantic Skinning (Mesh → 3DGS)

Instead of learning motion separately on 3DGS, DragMesh transfers the mesh-side motion prior to Gaussian primitives through: (1) geometry-guided Gaussian initialization using mesh vertices, (2) spatial alignment via Importance-Weighted Centroid Alignment, (3) KNN label propagation (K=10) of movable/static part labels from mesh to Gaussians, and (4) confidence-adaptive blending at part boundaries to reduce tearing during large motions. Both representations share the same dual quaternion trajectory.

Experiments

Evaluated on GAPartNet (23 categories, 1,045 articulated objects; 941 train, 104 object-disjoint validation).

Kinematics Prediction

Joint-type accuracy: 78.8% overall.

Axis error median: 0.477°.

Normalized origin error median: 0.0794.

Revolute joint type accuracy: 94.7%; Prismatic: 59.6% — main errors concentrate on prismatic type classification and subsequent axis estimation.

Cross-Modal Semantic Skinning Ablation

Importance-Weighted Centroid Alignment achieves 1.000 ± 0.000 IoU on 9 GAPartNet objects, outperforming simple centroid and ICP alignment.

KNN neighbor count K=10 yields stable label propagation.

Geometry-Guided Gaussian Initialization

On a microwave object, time to reach PSNR 25 dropped from 9.56 s (random bounding-box init) to 6.91 s (~38% faster).

Floater ratio reduced from 1.73% to 0.02%.

Runtime Efficiency

A single pre-trained kinematic core drives both rendering modalities:

Lightweight Mesh path: 120 FPS.

3DGS path: 85 FPS (measured after static 3DGS reconstruction; includes DQ-VAE decoding, semantic skinning, and rendering; excludes segmentation preprocessing and optional VLM API latency).

By concentrating computation on the moving part and reusing the motion prior across objects, DragMesh avoids per-object motion training and full-asset regeneration.

Conclusion & Future Work

DragMesh demonstrates that for existing 3D assets, interactive articulation can be achieved by predicting joint parameters from one drag, generating constrained dual quaternion trajectories, and transferring motion across representations. This is directly applicable to game assets, digital twins, simulation, and robotics where geometry exists but articulation is missing. Future directions include extending from single-joint to multi-joint kinematic chains, handling real-world 3D assets, contact, dynamics, and more complex object structures.