Image Inpainting Using Sparsity of the Transform Domain

In this paper, we propose a new image inpainting method based on the property that much of the image information in the transform domain is sparse. We add a redundancy to the original image by mapping the transform coefficients with small amplitudes to zero and the resultant sparsity pattern is used as the side information in the recovery stage. If the side information is not available, the receiver has to estimate the sparsity pattern. At the end, the recovery is done by consecutive projecting between two spatial and transform sets. Experimental results show that our method works well for both structural and texture images and outperforms other techniques in objective and subjective performance measures.

💡 Research Summary

The paper introduces a novel image inpainting framework that exploits the inherent sparsity of natural images in the transform (frequency) domain. The authors observe that, after applying a linear transform such as the Discrete Cosine Transform (DCT) or the Fast Fourier Transform (FFT), the majority of coefficients have small magnitudes and can be set to zero without severely degrading the visual content. By thresholding these coefficients, a “sparse image” is generated and the locations of the zeroed coefficients constitute a sparsity pattern. This pattern is treated as side information (SI) that can be transmitted alongside the corrupted image, thereby adding redundancy that aids the recovery process.

Two convex sets are defined for the reconstruction: (1) the set of all images whose transform coefficients respect the known sparsity pattern, and (2) the set of all images that agree with the known (uncorrupted) pixel values in the spatial domain. The first set is enforced by applying the transform, zero‑forcing the coefficients according to the sparsity pattern, and performing an inverse transform. The second set is enforced by a simple pixel‑wise replacement using the loss mask. The reconstruction proceeds by alternating projections onto these two convex sets, i.e., the classic Projection Onto Convex Sets (POCS) algorithm. Because both sets are convex and their intersection is non‑empty (provided the sparsity pattern is consistent with the uncorrupted pixels), the POCS iterations are guaranteed to converge to a point that satisfies both constraints.

When side information is unavailable—an often realistic scenario—the authors propose a preprocessing stage based on the Time‑Varying (TV) method. TV reconstructs the corrupted image from its non‑uniformly sampled pixels using low‑pass filtering and interpolation. The TV‑reconstructed image is then used to estimate the sparsity pattern by applying the same thresholding operation. This estimated pattern is fed into the POCS loop, effectively approximating the missing side information.

Experimental validation is performed on three benchmark images: Lena (structural), Barbara (mixed), and Baboon (high‑frequency texture). The authors test both DCT and FFT domains and vary the sparsity level from 90 % to 95 %. Results are reported in terms of Peak Signal‑to‑Noise Ratio (PSNR) measured against the original image and against the sparse image. For example, with 95 % sparsity in the DCT domain, the Lena image achieves 31.5 dB PSNR relative to the original and 63.5 dB relative to the sparse version after 500 POCS iterations (1000 iterations for selected cases). In the absence of side information, the FFT domain yields slightly higher PSNR values, confirming the robustness of the estimation step.

The proposed method is also compared against five state‑of‑the‑art inpainting techniques: the Total‑Variation (TV) method, Bertalmio’s PDE‑based approach, Criminisi’s exemplar‑based algorithm, Wong’s exemplar‑based method, and a patch‑sparsity technique. Across all three test images, the new method consistently outperforms the competitors by 1–2 dB in PSNR and avoids the smoothing artifacts typical of TV and the unnatural object insertion seen in exemplar‑based methods. Computationally, each POCS iteration requires a forward and inverse transform (O(N log N) complexity), making the overall runtime comparable to TV and substantially faster (by a factor of 0.01–0.1) than the exemplar‑based algorithms.

Key contributions of the paper are: (1) the innovative use of transform‑domain sparsity as side information, (2) a practical scheme for estimating this side information when it cannot be transmitted, and (3) the integration of these ideas within a POCS framework that guarantees convergence. The authors also discuss trade‑offs: higher sparsity levels improve recovery of large missing regions but increase blurring of the transmitted (sparse) image, and the need for a side‑information channel introduces additional overhead.

In conclusion, the authors demonstrate that transform‑domain sparsity can be harnessed effectively for image inpainting, especially for large block losses where traditional PDE or exemplar methods struggle. They suggest future work on embedding the sparsity operator into existing compression pipelines (e.g., JPEG) to reduce overhead, and on employing deep‑learning‑based estimators for the sparsity pattern to further improve performance when side information is unavailable.