LeafFit: Plant Assets Creation from 3D Gaussian Splatting

We propose LeafFit, a pipeline that converts 3D Gaussian Splatting (3DGS) of individual plants into editable, instanced mesh assets. While 3DGS faithfully captures complex foliage, its high memory footprint and lack of mesh topology make it incompatible with traditional game production workflows. We address this by leveraging the repetition of leaf shapes; our method segments leaves from the unstructured 3DGS, with optional user interaction included as a fallback. A representative leaf group is selected and converted into a thin, sharp mesh to serve as a template; this template is then fitted to all other leaves via differentiable Moving Least Squares (MLS) deformation. At runtime, the deformation is evaluated efficiently on-the-fly using a vertex shader to minimize storage requirements. Experiments demonstrate that LeafFit achieves higher segmentation quality and deformation accuracy than recent baselines while significantly reducing data size and enabling parameter-level editing.

💡 Research Summary

LeafFit presents a complete pipeline that transforms a 3D Gaussian Splatting (3DGS) capture of a single plant into compact, editable, instanced mesh assets suitable for modern game engines. While 3DGS excels at rendering fine foliage with high visual fidelity, its raw representation consists of hundreds of thousands of anisotropic Gaussian primitives, leading to a large memory footprint and a lack of explicit topology, which hampers direct use in production pipelines. LeafFit tackles these issues by exploiting the natural repetition of leaf shapes within a plant.

The first stage is leaf instance segmentation. The user selects a Gaussian primitive near the plant’s root, which serves as the source for a geodesic distance field computed via heat diffusion on the point cloud formed by Gaussian centers. Distance values increase smoothly toward leaf tips, so local maxima of the distance field correspond to leaf apexes. Because Gaussian density is non‑uniform, the method first applies farthest‑point sampling to obtain a uniformly distributed subset of primitives, then detects apexes on this subset, yielding robust segmentation even in noisy data. Optional manual refinement tools (lasso, box selection) are provided for challenging cases.

After segmentation, a single representative leaf (the template) is chosen. Using the Ball Pivoting Algorithm, a thin, sharp mesh is reconstructed directly from the Gaussian centers and covariances of the template leaf, preserving both front and back surfaces while avoiding the over‑thickening typical of implicit‑surface extraction. This template mesh becomes the canonical shape for all leaves.

Non‑rigid alignment is achieved with a differentiable Moving Least Squares (MLS) deformation. For each leaf instance, a small set of control points is extracted; the MLS formulation learns a deformation field (\Phi_j) that maps the template’s control points to those of the target leaf, minimizing a least‑squares energy. Crucially, only the control‑point positions are stored. At load time or during rendering, a vertex shader evaluates (\Phi_j) on the template vertices, generating the per‑leaf geometry on the fly. This strategy reduces the total asset size dramatically: the template mesh contains only a few thousand vertices, and each leaf adds merely dozens of control points.

Quantitative experiments on diverse plant species demonstrate that LeafFit achieves up to a 13× reduction in storage compared with implicit‑surface pipelines (e.g., Marching Cubes), while delivering frame rates from 936 FPS (baseline mesh) to 14,694 FPS (instanced MLS‑generated meshes). Segmentation IoU and Chamfer Distance metrics show superior accuracy over recent Gaussian‑to‑mesh baselines. Moreover, because the deformation parameters are exposed, artists can edit leaf size, curvature, and texture in real time, enabling parameter‑level control without re‑capturing or re‑meshing.

The paper’s contributions are: (1) a robust, training‑free leaf segmentation method based on geodesic distances and farthest‑point sampling, (2) a template‑driven, differentiable MLS registration that supports real‑time, per‑leaf deformation, (3) a lightweight surface extraction pipeline using BPA that preserves thin foliage geometry, and (4) a full game‑engine‑ready asset workflow that dramatically cuts memory usage while maintaining visual fidelity. Limitations include occasional failure on highly overlapping leaf layers and increased shader cost when the number of control points grows large. Future work will explore multi‑template strategies, hardware‑accelerated MLS, and extending the approach to whole‑plant structures (stems, branches) for end‑to‑end vegetation asset creation.