Double Relief with progressive weighting function

Feature weighting algorithms try to solve a problem of great importance nowadays in machine learning: The search of a relevance measure for the features of a given domain. This relevance is primarily used for feature selection as feature weighting can be seen as a generalization of it, but it is also useful to better understand a problem’s domain or to guide an inductor in its learning process. Relief family of algorithms are proven to be very effective in this task. On previous work, a new extension was proposed that aimed for improving the algorithm’s performance and it was shown that in certain cases it improved the weights’ estimation accuracy. However, it also seemed to be sensible to some characteristics of the data. An improvement of that previously presented extension is presented in this work that aims to make it more robust to problem specific characteristics. An experimental design is proposed to test its performance. Results of the tests prove that it indeed increase the robustness of the previously proposed extension.

💡 Research Summary

The paper addresses the problem of robust feature weighting in the Relief family of algorithms, which are widely used for feature selection and relevance estimation. Standard Relief updates feature weights by comparing each instance with its nearest hit (same class) and nearest miss (different class), using a difference function that handles both nominal and numeric attributes. While effective, Relief’s distance computation becomes unreliable when many irrelevant features are present, because irrelevant dimensions distort the nearest‑neighbor relationships, leading to poor weight estimates.

A previously proposed extension, Double Relief (dReliefF), attempts to improve this by feeding the weight estimates from the current iteration back into the distance calculation for the next iteration. Although this can accelerate convergence, it suffers from a critical drawback: early weight estimates are often inaccurate, and their immediate use can bias the distance metric, degrading performance especially in the early stages of learning.

To overcome this, the authors introduce a progressive weighting scheme called pdReliefF. The core idea is to modulate the influence of the current weight estimate (w) by a function (f(w, t)) that depends on the iteration number (t). The function satisfies three properties: (i) (f(w, 0) = 1) (no weighting at the start), (ii) (f(w, \infty) = w) (full weighting after many iterations), and (iii) monotonic increase with (t). They propose the specific form

\