Discussion of: Treelets--An adaptive multi-scale basis for sparse unordered data

We congratulate Lee, Nadler and Wasserman (henceforth LNW) on a very interesting paper on new methodology and supporting theory [arXiv:0707.0481]. Treelets seem to tackle two important problems of modern data analysis at once. For datasets with many variables, treelets give powerful predictions even if variables are highly correlated and redundant. Maybe more importantly, interpretation of the results is intuitive. Useful insights about relevant groups of variables can be gained. Our comments and questions include: (i) Could the success of treelets be replicated by a combination of hierarchical clustering and PCA? (ii) When choosing a suitable basis, treelets seem to be largely an unsupervised method. Could the results be even more interpretable and powerful if treelets would take into account some supervised response variable? (iii) Interpretability of the result hinges on the sparsity of the final basis. Do we expect that the selected groups of variables will always be sufficiently small to be amenable for interpretation?

💡 Research Summary

The discussion paper offers a critical appraisal of the “Treelets: An Adaptive Multi‑Scale Basis for Sparse Unordered Data” methodology introduced by Lee, Nadler, and Wasserman. Treelets construct a binary tree over variables based on their pairwise correlations and, at each merge, perform a localized principal component analysis to generate a new orthogonal basis vector. This adaptive, multi‑scale approach simultaneously addresses two pervasive challenges in modern high‑dimensional data analysis: (1) handling strong redundancy and correlation among a large number of variables, and (2) providing an interpretable representation by producing a sparse basis that groups related variables together. The authors of the discussion acknowledge these strengths but raise three substantive questions that probe the robustness, extensibility, and practical interpretability of the method.

First, they ask whether the same performance can be achieved by a more conventional pipeline that combines hierarchical clustering with standard PCA. Traditional hierarchical clustering groups variables using a distance metric, after which PCA is applied within each cluster to obtain principal components. Treelets differ because the merging decision is driven by the covariance structure and the orthogonal basis is recomputed after each merge, potentially capturing multi‑scale dependencies that a simple two‑step procedure might miss. The authors suggest systematic empirical comparisons to determine whether the adaptive nature of treelets truly provides a unique advantage over the clustering‑PCA combination.

Second, the discussion points out that treelets are fundamentally unsupervised: the basis selection does not incorporate any response variable. In many applied settings a supervised signal (e.g., a class label or continuous outcome) is available, and leveraging it could improve both interpretability and predictive power. The authors propose integrating supervised criteria—such as the correlation of candidate variable pairs with the response, or the increase in explained variance of the outcome—into the merge decision. This could steer the algorithm toward groups that are most informative for the prediction task, akin to supervised dimensionality‑reduction techniques, while retaining the multi‑scale structure.

Third, the utility of treelets hinges on the sparsity of the final basis. Although each merge initially combines only two variables, repeated merges can produce basis vectors that involve many original variables, potentially undermining interpretability. The authors argue that a quantitative sparsity threshold (e.g., limiting group size to a handful of variables) should be defined, and that regularization mechanisms—such as truncating small loadings or imposing a maximum cluster size—might be necessary to keep the resulting groups amenable to human inspection.

In summary, the discussion acknowledges treelets as an innovative contribution that elegantly merges adaptive hierarchical organization with localized PCA, yielding a sparse, multi‑scale representation. However, it emphasizes three avenues for further research: (i) rigorous benchmarking against a hierarchical‑clustering‑plus‑PCA baseline, (ii) development of supervised extensions that incorporate outcome information into the merging process, and (iii) explicit control of sparsity to guarantee interpretability in practice. Addressing these issues would strengthen the methodological foundations of treelets and broaden their applicability across diverse high‑dimensional domains.