📝 Original Info
- Title: Discrete Denoising with Shifts
- ArXiv ID: 0708.2566
- Date: 2016-11-17
- Authors: Researchers from original ArXiv paper
📝 Abstract
We introduce S-DUDE, a new algorithm for denoising DMC-corrupted data. The algorithm, which generalizes the recently introduced DUDE (Discrete Universal DEnoiser) of Weissman et al., aims to compete with a genie that has access, in addition to the noisy data, also to the underlying clean data, and can choose to switch, up to $m$ times, between sliding window denoisers in a way that minimizes the overall loss. When the underlying data form an individual sequence, we show that the S-DUDE performs essentially as well as this genie, provided that $m$ is sub-linear in the size of the data. When the clean data is emitted by a piecewise stationary process, we show that the S-DUDE achieves the optimum distribution-dependent performance, provided that the same sub-linearity condition is imposed on the number of switches. To further substantiate the universal optimality of the S-DUDE, we show that when the number of switches is allowed to grow linearly with the size of the data, \emph{any} (sequence of) scheme(s) fails to compete in the above senses. Using dynamic programming, we derive an efficient implementation of the S-DUDE, which has complexity (time and memory) growing only linearly with the data size and the number of switches $m$. Preliminary experimental results are presented, suggesting that S-DUDE has the capacity to significantly improve on the performance attained by the original DUDE in applications where the nature of the data abruptly changes in time (or space), as is often the case in practice.
💡 Deep Analysis
Deep Dive into Discrete Denoising with Shifts.
We introduce S-DUDE, a new algorithm for denoising DMC-corrupted data. The algorithm, which generalizes the recently introduced DUDE (Discrete Universal DEnoiser) of Weissman et al., aims to compete with a genie that has access, in addition to the noisy data, also to the underlying clean data, and can choose to switch, up to $m$ times, between sliding window denoisers in a way that minimizes the overall loss. When the underlying data form an individual sequence, we show that the S-DUDE performs essentially as well as this genie, provided that $m$ is sub-linear in the size of the data. When the clean data is emitted by a piecewise stationary process, we show that the S-DUDE achieves the optimum distribution-dependent performance, provided that the same sub-linearity condition is imposed on the number of switches. To further substantiate the universal optimality of the S-DUDE, we show that when the number of switches is allowed to grow linearly with the size of the data, \emph{any} (sequ
📄 Full Content
Discrete denoising is the problem of reconstructing the components of a finite-alphabet sequence based on the entire observation of its Discrete Memoryless Channel (DMC)-corrupted version. The quality of the reconstruction is evaluated via a user-specified (single-letter) loss function. Universal discrete denoising, in which no statistical or other properties are known a priori about the underlying clean data and the goal is to attain optimum performance, was considered and solved in [1]. The main problem setting there is the "semi-stochastic" one, in which the underlying signal is assumed to be an "individual sequence," and the randomness is due solely to the channel noise. In this setting, it is unreasonable to expect to attain the best performance among all the denoisers in the world, since for every given sequence, there exists a denoiser that recovers all the sequence components perfectly. Thus, [1] limits the comparison class, a.k.a. expert class, and uses the competitive analysis approach. Specifically, it is shown that regardless of what the underlying individual sequence may be, the Discrete Universal DEnoiser (DUDE) essentially attains the performance of the best sliding window denoiser that would be chosen by a genie with access to the underlying clean sequence, in addition to the observed noisy sequence. This semi-stochastic setting result is shown in [1] to imply the stochastic setting result, i.e., that for any underlying stationary signal, the DUDE attains the optimal distribution-dependent performance.
The setting of an arbitrary individual sequence, combined with competitive analysis, has been very popular in many other research areas, especially for problems of sequential decision-making. Examples include universal compression [4], universal prediction [5], universal filtering [2], repeated game playing [6,7,8], universal portfolios [9], online learning [10,11], zero-delay coding [12,13], and much more. A comprehensive account of this line of research can be found in [14]. The beauty of this approach is the fact that it leads to the construction of schemes that perform, on every individual sequence, essentially as well as the best in a class of experts, which is the performance of a genie that had hindsight on the entire sequence before selecting his actions. Moreover, if the expert class is judiciously chosen, the relative sense of such a performance guarantees can, in many cases, imply optimum performance in absolute senses as well.
One extension to this approach is competition with an expert class and a genie that has the freedom to form a compound action, which breaks the sequence into a certain (limited) number of segments, applies different experts in each segment, and achieves an even better performance overall. Note that the optimal segmentation of the sequence and the choice of the best expert in each segment is also determined by hindsight. Clearly, competing with the best compound action is more challenging, since the number of possible compound actions is exponential in the sequence length n, and the brute-force vanilla implementation of the ordinary universal scheme requires prohibitive complexity. However, clever schemes with linear complexity that successfully track the best segments and experts have been devised in many different areas, such as online learning, universal prediction [15,16], universal compression [17,18], online linear regression [19], universal portfolios [20], and zero-delay lossy source coding [22].
In this paper, we expand the idea of compound actions and apply it to the discrete denoising problem. The motivation of this expansion is natural: the characteristics of the underlying data in the denoising problem often tend to be time-or space-varying. In this case, determining the best segmentation and the best expert for each segment requires complete knowledge of both clean and noisy sequences. Therefore, whereas the challenge in sequential decision-making problems is to track the shift of the best expert based on the past, true observation, the challenge in the denoising problem is to learn the shift based on the entire, but noisy, observation. We extend DUDE to meet this challenge and provide results that parallel and strengthen those of [1].
Specifically, we introduce the S-DUDE and show first that, for every underlying noiseless sequence, it attains the performance of the best compound finite-order sliding window denoiser (concretely defined later), both in expectation and in a high probability sense. We develop our scheme in the semi-stochastic setting as in [1]. The toolbox for the construction and analysis of our scheme draws on ideas developed in [2]. We circumvent the difficulty of not knowing the exact true loss by using an observable unbiased estimate of it. This kind of an estimate has proved to be very useful in [2] and [3] to devise schemes for filtering and for denoising with dynamic contexts. Building on this semi-stochastic setting re
…(Full text truncated)…
📸 Image Gallery
Reference
This content is AI-processed based on ArXiv data.
