Many problems that arise in machine learning domain deal with nonlinearity and quite often demand users to obtain global optimal solutions rather than local optimal ones. Optimization problems are inherent in machine learning algorithms and hence many methods in machine learning were inherited from the optimization literature. Popularly known as the initialization problem, the ideal set of parameters required will significantly depend on the given initialization values. The recently developed TRUST-TECH (TRansformation Under STability-reTaining Equilibria CHaracterization) methodology systematically explores the subspace of the parameters to obtain a complete set of local optimal solutions. In this thesis work, we propose TRUST-TECH based methods for solving several optimization and machine learning problems. Two stages namely, the local stage and the neighborhood-search stage, are repeated alternatively in the solution space to achieve improvements in the quality of the solutions. Our methods were tested on both synthetic and real datasets and the advantages of using this novel framework are clearly manifested. This framework not only reduces the sensitivity to initialization, but also allows the flexibility for the practitioners to use various global and local methods that work well for a particular problem of interest. Other hierarchical stochastic algorithms like evolutionary algorithms and smoothing algorithms are also studied and frameworks for combining these methods with TRUST-TECH have been proposed and evaluated on several test systems.
Deep Dive into TRUST-TECH based Methods for Optimization and Learning.
Many problems that arise in machine learning domain deal with nonlinearity and quite often demand users to obtain global optimal solutions rather than local optimal ones. Optimization problems are inherent in machine learning algorithms and hence many methods in machine learning were inherited from the optimization literature. Popularly known as the initialization problem, the ideal set of parameters required will significantly depend on the given initialization values. The recently developed TRUST-TECH (TRansformation Under STability-reTaining Equilibria CHaracterization) methodology systematically explores the subspace of the parameters to obtain a complete set of local optimal solutions. In this thesis work, we propose TRUST-TECH based methods for solving several optimization and machine learning problems. Two stages namely, the local stage and the neighborhood-search stage, are repeated alternatively in the solution space to achieve improvements in the quality of the solutions. Our
acteristics of stability boundaries of a nonlinear dynamical system corresponding to the nonlinear function of interest. Basically, our method coalesces the advantages of the traditional local optimizers with that of the dynamic and geometric characteristics of the stability regions of the corresponding nonlinear dynamical system. Two stages namely, the local stage and the neighborhood-search stage, are repeated alternatively in the solution space to achieve improvements in the quality of the solutions. The local stage obtains the local maximum of the nonlinear function and the neighborhood-search stage helps to escape out of the local maximum by moving towards the neighboring stability regions. Our methods were tested on both synthetic and real datasets and the advantages of using this novel framework are clearly manifested. This framework not only reduces the sensitivity to initialization, but also allows the flexibility for the practitioners to use various global and local methods that work well for a particular problem of interest. Other hierarchical stochastic algorithms like evolutionary algorithms and smoothing algorithms are also studied and frameworks for combining these methods with TRUST-TECH have been proposed and evaluated on several test systems.
Chandan Reddy was born in Jammalamadugu, Andhra Pradesh, India on May 11, 1980 A saddle point (x d ) is located between two local minima (x 1 s and x 2 s ).
x 1 m and x 2 m are two local maxima located in the orthogonal direction.
2.2 Phase potrait of a gradient system. The solid lines with solid arrows represent the basin boundary. ∂A p (x 1 s ) = 3 i=1 W s (x i d ). The local minima x 1 s and x 2 s correspond to the stable equilibrium points of the gradient system. The saddle point (x 1 d ) corresponds to the dynamic decomposition point that connects the two stable equilibrium points.
The interval point a 5 can be present either to the left or to the right of the peak. When a 6 is reached, the golden section search method is invoked with a 6 and a 4 as the interval. . . . . . . . . . . . . . . . .
The gradient curve obtained in one of the higher dimensional test cases. MGP indicates the minimum gradient point where the magnitude of the gradient reaches the minimum value. This point is used as a initial guess for a local optimization method to obtain DDP. . .
The problem of finding a global optimal solution arise in many disciplines ranging from science to engineering. In real world applications, multi-dimensional objective functions usually contain a large number of local optimal solutions. Obtaining a global optimal solution is of primary importance in these applications and is a very challenging problem. Some examples of these applications are : molecular confirmation prediction [22], VLSI design in microelectronics [95], resource allocation problems [41], design of wireless networks [69], financial decision making [87],
structural engineering [58] and parameter estimation problems [53]. In this thesis, the primary focus is on the parameter estimation problems that arise in the field of machine learning.
Machine learning algorithms can be broadly classified into two categories [44]: (i) Supervised learning and (ii) Unsupervised learning. The primary goal in supervised learning is to learn a mapping from x to y given a training dataset which consists of pairs (x i , y i ), where x i ∈ X are the data points and y i ∈ Y are the labels (or targets). A standard assumption is that the pairs (x i , y i ) are sampled i.i.d. from some distribution. If y takes values in a finite set (discrete values) then it is a classification problem and if it takes values in a continuous space, then it is a regression problem. Support vector machines [21], artificial neural networks [68] and boosting [61] are the most popular algorithms for supervised learning. All these algorithms will construct a classification (or regression) model based on certain training data available. Usually, the effectiveness of any algorithm is evaluated using testing data which is separate from the training data. In this thesis, we will primarily focus on artificial neural networks and estimating the parameters of its model. Constructing a model using artificial neural network involves estimating the parameters of the model that can effectively exploit the potential of the model. These parameters are usually obtained by finding the global minimum on the error surface. More details on training neural networks will be presented in Chapter 6.
Unsupervised learning, on the other hand, will train models using only the data- In both the models mentioned above (neural networks and mixture models), estimating the parameters correspond to obtaining a global optimal solution on a highly nonlinear surface. The surface can be generated based on a function that
Optimization methods can be broadly classified into two categories: global methods and local methods. Global methods explore the entire
…(Full text truncated)…
This content is AI-processed based on ArXiv data.
