Splat Feature Solver

Feature lifting has emerged as a crucial component in 3D scene understanding, enabling the attachment of rich image feature descriptors (e.g., DINO, CLIP) onto splat-based 3D representations. The core challenge lies in optimally assigning rich general attributes to 3D primitives while addressing the inconsistency issues from multi-view images. We present a unified, kernel- and feature-agnostic formulation of the feature lifting problem as a sparse linear inverse problem, which can be solved efficiently in closed form. Our approach admits a provable upper bound on the global optimal error under convex losses for delivering high quality lifted features. To address inconsistencies and noise in multi-view observations, we introduce two complementary regularization strategies to stabilize the solution and enhance semantic fidelity. Tikhonov Guidance enforces numerical stability through soft diagonal dominance, while Post-Lifting Aggregation filters noisy inputs via feature clustering. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on open-vocabulary 3D segmentation benchmarks, outperforming training-based, grouping-based, and heuristic-forward baselines while producing lifted features in minutes. Our \textbf{code} is available in the \href{https://github.com/saliteta/splat-distiller/tree/main}{\textcolor{blue}{GitHub}}. We provide additional \href{https://splat-distiller.pages.dev/}{\textcolor{blue}{website}} for more visualization, as well as the \href{https://www.youtube.com/watch?v=CH-G5hbvArM}{\textcolor{blue}{video}}.

💡 Research Summary

The paper tackles the problem of “feature lifting,” i.e., attaching rich 2‑D image descriptors such as CLIP, DINO, or ViT embeddings to splat‑based 3‑D scene representations (Gaussian splats, Beta splats, etc.). Existing approaches fall into three families: (1) joint geometry‑and‑feature training, which is accurate but computationally heavy; (2) grouping‑based pipelines that rely on 2‑D masks (e.g., SAM) and a lightweight optimization; and (3) heuristic forward methods that directly project features without training. All of them lack a unified theoretical formulation and are vulnerable to multi‑view inconsistencies and noisy observations.

The authors formalize feature lifting as a sparse linear inverse problem. For each camera ray r, the splat rendering equation yields a weight ωₚ (the alpha‑blending contribution of primitive p). Stacking all rays produces a matrix A ∈ ℝ^{R×P} where each entry A_{ij}=ω_{r p}. The unknown per‑primitive feature vectors form X ∈ ℝ^{P×F}, and the observed dense 2‑D features form B ∈ ℝ^{R×F}. The lifting task becomes solving A X = B. Because R≫P, the system is over‑determined and typically inconsistent due to sensor noise, mask errors, and view‑dependent artifacts. Consequently, the authors adopt a convex loss (L₂ or any convex function) and seek the minimizer X* = arg min_X ‖A X − B‖².

Two complementary regularization mechanisms are introduced. First, Tikhonov Guidance adds a diagonal term λI to AᵀA, enforcing soft diagonal dominance. This stabilizes the inverse, reduces the condition number, and mitigates the effect of rows whose sums deviate from one. Second, Post‑Lifting Aggregation clusters the initial solution (computed analytically) to filter out inconsistent or noisy contributions that arise from erroneous masks or outlier views. By replacing each primitive’s feature with its cluster centroid, the method suppresses spurious variations while preserving semantic coherence.

The core solver is a closed‑form row‑sum preconditioner:
x_j = ∑i A{ij} B_i / ∑i A{ij}.
The authors prove that this solution is a (1 + β)‑approximation of the true convex‑loss minimizer, where β depends on the deviation of row sums from unity and the Tikhonov regularization strength. They also show that several recent heuristic methods (Argmax Lifting, Occam’s LGS, CosegGaussians) implicitly use the same preconditioner but without theoretical justification or error bounds.

Extensive experiments are conducted on open‑vocabulary 3‑D segmentation benchmarks (ScanNet‑200, Replica, Matterport3D). The method is evaluated with four dense feature backbones (CLIP‑ViT, DINO, MaskCLIP, General‑ViT) and across multiple splat kernels (3DGS, 2DGS, Beta‑Splat). Results demonstrate state‑of‑the‑art mean IoU, surpassing training‑based, grouping‑based, and heuristic baselines by 2–5 percentage points while requiring only a few minutes of computation (5–10 min) versus hours for learning‑based pipelines. Ablation studies confirm that Tikhonov Guidance improves numerical stability, and Post‑Lifting Aggregation significantly reduces the impact of noisy masks, especially in challenging view configurations.

In summary, the paper provides a unified, mathematically grounded formulation of feature lifting, delivers a fast closed‑form solver with provable error bounds, and introduces practical regularization strategies that together enable high‑quality, generalizable feature attachment to splat‑based 3‑D representations. This contribution bridges the gap between efficient rendering pipelines and multi‑modal semantic understanding, opening avenues for real‑time 3‑D scene analysis, cross‑modal retrieval, and robotics applications where rapid, reliable feature lifting is essential.