A Unified Algorithmic Framework for Multi-Dimensional Scaling

A Unified Algorithmic Framework for Multi-Dimensional Scaling Arvind Agarwal∗ Jeff M. Phillips† Suresh Venkatasubramanian‡ Abstract In this paper, we propose a unified algorithmic framework for solving many known variants of MDS. Our algorithm is a simple iterative scheme with guaranteed convergence, and is modular; by changing the internals of a single subroutine in the algorithm, we can switch cost functions and target spaces easily. In addition to the formal guarantees of convergence, our algorithms are accurate; in most cases, they converge to better quality solutions than existing methods, in comparable time. We expect that this framework will be useful for a number of MDS variants that have not yet been studied. Our framework extends to embedding high-dimensional points lying on a sphere to points on a lower di- mensional sphere, preserving geodesic distances. As a compliment to this result, we also extend the Johnson- Lindenstrauss Lemma to this spherical setting, where projecting to a random O((1/ϵ2)log n)-dimensional sphere causes ϵ-distortion. 1 Introduction Multidimensional scaling (MDS) [23, 10, 3] is a widely used method for embedding a general distance matrix into a low dimensional Euclidean space, used both as a preprocessing step for many problems, as well as a visualization tool in its own right. MDS has been studied and used in psychology since the 1930s [35, 33, 22] to help visualize and analyze data sets where the only input is a distance matrix. More recently MDS has become a standard dimensionality reduction and embedding technique to manage the complexity of dealing with large high dimensional data sets [8, 9, 31, 6]. In general, the problem of embedding an arbitrary distance matrix into a fixed dimensional Euclidean space with minimum error is nonconvex (because of the dimensionality constraint). Thus, in addition to the stan- dard formulation [12], many variants of MDS have been proposed, based on changing the underlying error function [35, 8]. There are also applications where the target space, rather than being a Euclidean space, is a manifold (e.g. a low dimensional sphere), and various heuristics for MDS in this setting have also been proposed [13, 6]. Each such variant is typically addressed by a different heuristic, including majorization, the singular value de- composition, semidefinite programming, subgradient methods, and standard Lagrange-multipler-based meth- ods (in both primal and dual settings). Some of these heuristics are efficient, and others are not; in general, every new variant of MDS seems to require different ideas for efficient heuristics. 1.1 Our Work In this paper, we present a unified algorithmic framework for solving many variants of MDS. Our approach is based on an iterative local improvement method, and can be summarized as follows: “Pick a point and move it so that the cost function is locally optimal. Repeat this process until convergence.” The improvement step reduces to a well-studied and efficient family of iterative minimization techniques, where the specific algorithm depends on the variant of MDS. A central result of this paper is a single general convergence result for all variants of MDS that we examine. This single result is a direct consequence of the way in which we break down the general problem into an ∗Partially supported by NSF IIS-0712764 †Supported by a subaward to the University of Utah under NSF award 0937060 to the Computing Research Association ‡Partially supported by NSF CCF-0953066 1 arXiv:1003.0529v2 [cs.LG] 30 Mar 2010 iterative algorithm combined with a point-wise optimization scheme. Our approach is generic, efficient, and simple. The high level framework can be written in 10-12 lines of MATLAB code, with individual function- specific subroutines needing only a few more lines each. Further, our approach compares well with the best methods for all the variants of MDS. In each case our method is consistently either the best performer or is close to the best, regardless of the data profile or cost function used, while other approaches have much more variable performance. Another useful feature of our method is that it is parameter-free, requiring no tuning parameters or Lagrange multipliers in order to perform at its best. Spherical MDS. An important application of our approach is the problem of performing spherical MDS. Spherical MDS is the problem of embedding a matrix of distances onto a (low-dimensional) sphere. Spheri- cal MDS has applications in texture mapping and image analysis [6], and is a generalization of the spherical dimensionality reduction problem, where the goal is to map points from a high dimensional sphere onto a low- dimensional sphere. This latter problem is closely related to dimensionality reduction for finite dimensional dis- tributions. A well-known isometric embedding takes a distribution represented as a point on the d-dimensional simplex to the d-dimensional sphere while preserving the Hellinger distance betwee

…(Full text truncated)…

This content is AI-processed based on ArXiv data.