Joint Object-Material Category Segmentation from Audio-Visual Cues

It is not always possible to recognise objects and infer material properties for a scene from visual cues alone, since objects can look visually similar whilst being made of very different materials. In this paper, we therefore present an approach that augments the available dense visual cues with sparse auditory cues in order to estimate dense object and material labels. Since estimates of object class and material properties are mutually informative, we optimise our multi-output labelling jointly using a random-field framework. We evaluate our system on a new dataset with paired visual and auditory data that we make publicly available. We demonstrate that this joint estimation of object and material labels significantly outperforms the estimation of either category in isolation.

💡 Research Summary

The paper tackles the problem of semantic segmentation when visual cues alone are insufficient to distinguish objects that look alike but are made of different materials. Inspired by the human ability to infer material properties by tapping objects, the authors propose a multimodal framework that fuses dense visual features with sparse auditory cues obtained by tapping objects with a human knuckle. The core of the approach is a two‑layer fully‑connected Conditional Random Field (CRF). One layer models object class labels, the other models material labels. Each layer contains unary potentials (derived from CNN‑based visual classifiers) and dense pairwise potentials (Gaussian mixture kernels) that enforce long‑range consistency.

Material unary potentials are a convex combination of visual class probabilities and audio‑derived probabilities; when audio is unavailable for a pixel, a uniform distribution is used to prevent over‑confidence of the visual classifier. The two layers are linked by a joint potential that captures statistical correlations between objects and materials (e.g., desks are usually wood). This joint term allows material predictions to refine object predictions and vice‑versa, enabling mutual reinforcement during inference.

To evaluate the method, the authors built a new dataset because existing segmentation benchmarks lack audio annotations. They captured nine indoor scenes with an ASUS Xtion Pro RGB‑D camera, reconstructed each scene in 3D, and manually annotated 20 object classes and 11 material types in the 3D space. Ground‑truth 2D labels were generated by ray‑casting the 3D models, dramatically reducing labeling effort. For audio, they recorded about 600 high‑quality tap sounds from 50 objects across nine material categories using a condenser microphone at 44.1 kHz. Each sound is associated with a small set of pixels (median 575 pixels, 0.18 % of an image).

Experiments compare three configurations: (1) visual‑only CRFs for objects and materials, (2) a single‑layer CRF that incorporates audio only in the material unary, and (3) the proposed joint two‑layer CRF. Results on a held‑out test set (scenes not seen during training) show that the joint model outperforms the others by 5–12 % absolute Intersection‑over‑Union (IoU) on both object and material segmentation. The gains are especially pronounced for materials that are visually ambiguous (plastic vs. wood, ceramic vs. gypsum). Moreover, the joint term improves object segmentation accuracy, confirming that object and material labels are mutually informative.

The paper’s contributions are threefold: (i) a principled way to integrate sparse auditory cues into a dense CRF framework, (ii) a joint object‑material CRF that leverages cross‑label dependencies, and (iii) a publicly released RGB‑D‑audio dataset with dense per‑pixel object and material annotations. Limitations include the reliance on manually collected tap sounds and the current focus on static scenes. Future work could explore automated tapping with robotic manipulators, real‑time inference with lightweight models, and the inclusion of other contact‑based sounds (e.g., friction, vibration).

In summary, by marrying sound and sight, the authors demonstrate that auditory information can substantially enrich semantic segmentation, particularly for material discrimination, and that jointly optimizing object and material labels yields better performance than treating them independently. The released dataset and code provide a valuable resource for further research in multimodal perception.