A nonlinear preconditioner for experimental design problems

Noname man uscript No. (will b e inserted b y the editor) A nonlinear preconditioner for exp erimental design problems Mario S. Mommer · Andreas Sommer · Johannes P . Sc hl¨ oder · H. Georg Bo ck Nov em ber 23, 2021 Abstract W e address the slo w conv ergence and po or stabilit y of quasi-newton se- quen tial quadratic programming (SQP) methods that is observ ed when solving exper- imen tal design problems, in particular when they are large. Our findings suggest that this behavior is due to the fact that these problems often hav e bad absolute condition n umbers. T o shed light onto the structure of the problem close to the solution, we for- m ulate a mo del problem (based on the A -criterion), that is defined in terms of a given initial design that is to b e impro v ed. W e prov e that the absolute condition num ber of the mo del problem grows without bounds as the qualit y of the initial design improv es. Additionally , we devise a preconditioner that ensures that the condition num ber will instead sta y uniformly b ounded. Using numerical exp eriments, w e study the effect of this reform ulation on relev an t cases of the general problem, and find that it leads to significan t improv ements in stability and con vergence behavior. Keyw ords Sequential Quadratic Programming · Preconditioning · Design of Exp erimen ts Mathematics Sub ject Classification (2000) 90C30 · 90C55 · 62K99 1 Introduction One of the most imp ortant aspects in model based optimization and in vestigation of real w orld processes is the estimation of parameters app earing in a mo del. T ypically , one estimates these parameters solving a regression problem based on data collected from one or more exp eriments. Optimal exp erimental design is the task of choosing the best exp erimental setup from a set of possible ones, and according to a predefined criterion. As this task is bound to b e constrained in non-trivial w ays, and is form ulated in terms of the optimalit y conditions of an underlying regression problem, it presents a rich class of challenging optimization problems. It is also a practically relev an t class, Mario S. Mommer (  ) · Andreas Sommer · Johannes P . Schl¨ oder · H. Georg Bo ck Interdisciplinary Center for Scientific Computing (IWR), Heidelberg Univ ersity , INF 368, 69120 Heidelb erg, German y . E-mail: { mario.mommer|andreas.sommer|johannes.schloeder|bock } @iwr.uni-heidelberg.de 2 as the solution of these problems leads to imp ortant improv ements in the efficiency of researc h and developmen t [8, 26]. A standard setting is the estimation of parameters using weigh ted least-squares regression. It is then natural to rate the exp eriments according to the quality of the Fisher information matrix, or of its inv erse, the v ariance-cov ariance matrix. A v ariety of criteria to rate this qualit y exists, and they are traditionally named after letters of the alphab et. In this article we will focus mainly on the so called A -criterion , which is defined as the trace of the v ariance-cov ariance matrix of the regression. F or a full list and further discussion w e refer to the classic texts [21, 6]. The experimental design problems we are concerned with here ha ve the following form. W e consider the nonlinear regression problem ˜ p = argmin p m X i =1 | F ( p, q , t i ) − µ i | 2 (1) where F is a nonlinear function depending on p ar ameters p , c ontr ols q , and me asur e- ment p oints t i . Often these points refer to measurement times, but our scop e is not restricted to this case. The v alues µ i represen t the results of measurements, and F rep- resen ts the mo del under study . If the measuremen t errors are indep endent, normally distributed with v ariance one and zero mean, the parameters obtained from (1) will b e a random v ariable whic h, to a first appro ximation, is dra wn from a m ultiv ariate normal distribution cen tered around the true parameter v alues p ∗ , and with v ariance- co v ariance matrix Σ = ( J T J ) − 1 , where the matrix J is giv en by J ij = ∂ ∂ p j F ( p ∗ , q , t i ) . The exp eriment design problem we consider is the problem of finding controls q , and a subset of measurement p oints suc h that the trace of the v ariance-cov ariance matrix is minimal, at least for the parameters p that w e b elieve to b e plausible when designing the exp erimen t. Our basic problem is th us, given a set of feasible controls Θ , and a me asur ement budget m max < m , find a set M ⊂ { 1 , 2 , . . . , m } with cardinalit y # M = m max , and a q ∈ Θ suc h that T r  h J [ M ] ( q ) T J [ M ] ( q ) i − 1  is minimal. Here, w e hav e made explicit the dep endence on q , and hav e denoted b y J [ M ] the matrix J after deleting all ro ws whose index is not in the set M . There are a couple of approac hes in the literature to solve this problem [15, 1, 16, 6, 8]. One imp ortant approach, and the one we will consider here, consists in using a r elaxe d formulation to obtain a constrained minimization problem in contin uous v ariables, and then to apply a mo dern optimization metho d, typically in the form of a sequen tial quadratic programming (SQP) [13, 20, 19] algorithm to solve it. This approac h was pioneered by [16, 1, 15], and the formulation has the follo wing form. Define W : R m → R m × m through W ( w ) := diag ( w ), a diagonal matrix whose en tries are the elemen ts of the v ector of weigh ts w . W e define the set of admissible w eights Ω ( m max , m ) := { w ∈ [0 , 1] m | m X i =1 w i = m max } 3 10 -6 10 -5 10 -4 10 -3 10 -2 10 -1 10 0 10 1 0 50 100 150 200 ||Δ x k || Ite ration Or igina l Pre cond ition ed Fig. 1 Conv ergence behavior of an SQP algorithm solving a nonlinear exp erimental design problem. The problem no w is min w,q T r  h J T ( q ) W ( w ) J ( q ) i − 1  (2a) sub ject to q ∈ Θ , w ∈ Ω ( m max , m ) . (2b) The form of this problem is such that it can b e given to an SQP solver as it is. It has the important adv an tage that the optimization of the controls and of the w eights o ccurs at the same time. On the other hand, it has the disadv an tage of not necessarily yielding integer solutions, although practical exp erience shows that either the solution is in teger, or only a few w eigh ts are not, which can then be remedied b y using a rounding technique [22]. The understanding of this phenomenon is incomplete, but see [24] for an imp ortant contribution on this issue. Alternatively , the weigh ts in (2) can b e in terpreted as a required precision of the measurements, i.e. as a measure of the maxim um v ariance of the error which is acceptable. In practice, it turns out that solving the problem (2) in this wa y leads to po or con vergence. Typical b ehavior can b e seen in Figure 1, where we hav e plotted the length of the searc h direction obtained from the quadratic problem against the iteration n umber (the precise description of the n umerical experiment can be found in section 3). It is important to remark that this difficulty appears also when there are no external con trols, i.e. when the vector of controls q is empty . In what follows, we will attempt to shed some light on the question as to wh y this b eha vior emerges, and in particular onto how to solve it. T o this end, inspired by the “test equation” [3] used in the theory of stiff ordinary differential equations, we tackle the generality of the setting by developing a mo del problem. In this model problem we disco ver arbitrarily bad absolute conditioning under fairly generic conditions. It turns out that this can b e corrected by in tro ducing a simple but nonobvious transformation. This transformation can also be applied to (2), yielding a problem for which the SQP metho d con verges in muc h less iterations than for the original one. While the connection betw een absolute condition n umbers and the difficult y of solv- ing an optimization problem has b een made b efore [27, 28], its role in the conv ergence 4 b eha vior of SQP metho ds has not, to our kno wledge, b een in vestigated. Thus, while w e cannot pro vide a complete theoretical justification for the slow conv ergence of SQP metho ds when solving (2), w e pro vide b elo w what we consider to be strong evidence for the hypothesis that it is due to bad absolute condition n umbers. A t the v ery least we pro vide, in the form of a left preconditioner, an effectiv e wa y to significantly accelerate the n umerical solution of exp erimental design problems. 2 The mo del problem and its conditioning W e are led to our mo del problem b y removing elemen ts of the full problem (2). The first simplification is the remov al of the external con trols. As a consequence, the matrix J is fixed, and we ha ve a problem in a form that is addressed by classical texts (see e.g. [6]). Even this simplified problem is difficult to analyze, as the assumptions imposed on J in this theory are rather w eak. Informally sp eaking, once it has full rank, J can b e any old matrix. Since this class of problems is very general, one can exp ect to find coun terexamples to any generalization of behavior observ ed in practice. Our approac h will thus be to restrict ourselves to a mo del problem that is endow ed with enough structure to understand its badly-conditioned nature, and to devise a metho d of curing it. This problem is as follows. Giv en initial information in the form of a v ariance- co v ariance matrix Σ (which we, for simplicity , will consider to b e a multiple α of the iden tity matrix) for the parameters of interest, our task is to c ho ose b etw een t wo additional observ ations using a relaxed formulation. The idea of this problem is to mo del an adv anced stage of our experimental design, in whic h, for the sake of argumen t, all w eights except for t wo are kno wn to b e either one or zero. The role of the parameter α is to mo del the qualit y of the initial design that is the starting p oint of our mo del problem. The smaller the α , the better the design that is to b e improv ed. The optimization problem w e will study is th us min w 1 ,w 2 T r  h α − 1 I + w 1 v 1 v T 1 + w 2 v 2 v T 2 i − 1  (3a) sub ject to w 1 + w 2 = 1 and 0 ≤ w i ≤ 1 , i = 1 , 2 . (3b) Theorem 1 Supp ose that v T 1 v 2 = 0 and k v 1 k = k v 2 k = 1 . Then the solution of pr oblem (3) is w 1 = w 2 = 1 / 2 , and the absolute c ondition number of the solution is κ abs =  1 + 1 2 α  3 2 α 3 . (4) The use of the absolute condition n umber is due to the fact that we are searching for the zero of a deriv ative. The error in the “data” is thus a p erturbation of the zero, whic h is only meaningful in an absolute sense. Pr o of W e write first w 2 = 1 − w 1 to eliminate the equality constraint, and th us obtain a problem in one v ariable w . The function we w an t to minimize is then f ( w ) := T r  h α − 1 I + wv 1 v T 1 + (1 − w ) v 2 v T 2 i − 1  . (5) 5 The orthogonality of v 1 and v 2 simplifies the application of the Sherman-Morrison form ula to obtain f ( w ) := 2 α − wα 2 1 + wα − (1 − w ) α 2 1 + (1 − w ) α . (6) No w we lo ok for w ∗ suc h that f 0 ( w ∗ ) = 0. Straigh t-forward algebraic arguments yield w ∗ = 1 / 2. T o inv estigate the effect of p erturbations, we choose  > 0, and define the solution mapping g : ( − ,  ) → R b y f 0 ( g (  )) =  . Of course, g (0) = w ∗ , and the condition n umber of the problem f 0 ( w ) = 0 is given b y [4] κ abs = | g 0 (0) | | g (0) | . Using implicit differen tiation on f 0 ( g (  )) −  = 0 w e obtain the expression κ abs = 1 | f 00 ( w ∗ ) || w ∗ | . (7) T o finish the pro of, we only need to compute f 00 ( w ) and verify the expression. Theorem 1 suggests that as the optimization progresses and our exp erimental design b ecomes better and better, that is, α b ecomes smaller, the absolute condition n umber of the problem will increase. It will behav e as α − 3 for small α . Informally speaking, the b ottom of the v alley in which the solution lies will b ecome very flat, making it harder and harder to choose b etw een tw o additional rows of roughly the same size. The mo del problem th us predicts the stagnation in the solution process, whic h is precisely what often o ccurs in practice. A t least for the mo del problem, there is a surprisingly simple remedy in the form of a left pr e c onditioner . A left preconditioner for a giv en problem is a diffeomorphism in the dep endent v ariables that preserves the solution, and at the same time it improv es the condition n umber. Its name comes from the fact that it appears to the left of the function that defines the unpreconditioned problem. The definition w e use here is an extension of the definition that is commonly used in linear algebra (see e.g [23]) to a nonlinear setting. The next theorem suggests a p ossible choice of a left preconditioner for the mo del problem. Theorem 2 The pr oblem min w 1 ,w 2 −  T r  h α − 1 I + w 1 v 1 v T 1 + w 2 v 2 v T 2 i − 1  − 2 (8a) subje ct to w 1 + w 2 = 1 and 0 ≤ w i ≤ 1 , i = 1 , 2 . (8b) has the same minimum as (3). F or every α > 0 the absolute c ondition numb er of the minimum is κ abs = 2 . Pr o of Rep eat, with ˜ f := − f − 2 , the calculation of the condition num b er of Theorem 1 6 In light of the ab ov e, a p ossible wa y to achiev e b etter conv ergence when solving (2) is to tak e inspiration in Theorem 2 and c ho ose the left preconditioner h : (0 , ∞ ) → ( −∞ , 0) h ( z ) := − z − 2 . As a consequence, we prop ose (and recommend) to solve the following problem instead of (2): min w,q −  T r  h J T ( q ) W ( w ) J ( q ) i − 1  − 2 (9a) sub ject to the constrain ts w ∈ Ω ( m max , m ) and q ∈ Θ . (9b) As we will see in the next section, this modification has the desired effect of accelerating the con vergence of SQP metho ds when solving exp erimental design problems. 3 Numerical exp erimen ts In what follo ws, we will compare experimentally ho w w ell the SQP solv er behav es when solving the original problem (2) versus the preconditioned problem (9). In the first exp erimen t, w e tac kle the task of optimizing a design when given a prior information, but without external controls q . In the second exp eriment, w e keep m max constan t and v ary the n umber of candidate measurements to observe ho w the reform ulation affects p erformance as the size of the problem c hanges. The third numerical exp eriment is a full nonlinear experimental design problem with external con trols on a mo del defined through a system of ordinary differential equations. With it, we in tend to illustrate the effect of the reform ulation on practical problems. T o mak e the results representativ e and repro ducible, w e programmed the SQP solv er as it is describ ed in [19], with an augmented Lagrangian p enalty function as describ ed in [25]. This results in a reasonably robust and effectiv e solv er. The quadratic problems are solved using QPOPT [9], and the Hessian is approximated using damp ed BF GS up dates [19]. T o av oid scaling issues as m uch as possible, we use as an initial Hessian appro ximation a diagonal matrix with the absolute v alues of the diagonal of the exact Hessian. This retains the diagonal of the exact Hessian in the unpreconditioned (con vex) case, and assures p ositive definiteness in the preconditioned case. W e dev elop ed a Radau IIa solv er of order 5 based on the ideas in [12] that uses in ternal n umerical differen tiation [2] to compute accurate sensitivities of the differen tial equation (here: 1st to 3rd order). All deriv atives of algebraic functions, including the in version of the information matrix, w ere computed using automatic differen tiation [11, 10], with a Common Lisp AD pac k age [5] extended for higher deriv atives. The in version of the information matrix w as done by directly computing J T W ( w ) J and applying a Cholesky decomposition. T o cope with stability issues, w e used quad-double arithmetic [14]. This also ensures w e can use the full range [0 , 1] for the weigh ts [17]. The problems without external con trols are linear, and th us are defined through the matrix J , which we generate randomly . W e do this by filling a matrix with random en tries uniformly distributed in [ − 1 , 1], performing a singular v alue decomposition, and substituting its singular v alues with exp onentially decaying ones chosen to obtain a condition n umber of 10 4 . 7 5 10 15 20 25 30 35 40 45 10 -6 10 -5 10 -4 10 -3 10 -2 10 -1 10 0 0 0.5 1 1.5 2 2.5 3 3.5 Iterations Error α k p k u e p e u 5 10 15 20 25 30 35 40 45 10 -6 10 -5 10 -4 10 -3 10 -2 10 -1 10 0 0 20 40 60 80 100 Iterations % breakdown α (breakdown) u k p k u Fig. 2 Effect of prior information on stability and av erage iteration counts of the unprecon- ditioned and preconditioned problems (subscripts u and p , respectively). See text for details. 3.1 Effect of prior information In this first exp eriment, we wan t to observe the b eha vior of SQP methods on the sampling design problem for c ho osing 20 out of 50 p ossible measuremen t p oints: min w ∈ Ω (20 , 50) T r  h α − 1 I + J T W ( w ) J i − 1  (10) with and without using the preconditioner. The matrix J ∈ R 50 × 7 is c hosen at random as describ ed before, but we additionally normalize all its rows in the euclidean norm. W e thus obtain a nontrivial problem that is similar to our mo del, and where w e can observ e the effect of prior information αI directly . In Figure 2 we summarize the results of solving problem (10) with and without pre- conditioning for 200 differen t matrices and 11 differen t v alues of α , c hosen equidistan tly on a logarithmic scale in [10 − 6 , 1]. As α decreases, and th us the prior information b e- comes better and b etter, w e see that the iteration count of the preconditioned v ariant stabilizes to around 40, while the sensitivity to the initial guess is about the same for eac h α . This distance w as estimated b y comparing the solution w e obtained using t wo differen t starting guesses for the iteration. One that has the first 20 components equal to 1, and the rest is zero, and one starting guess that is the rev erse. Since the problem is con vex, we can use this sensitivit y as a pro xy for the distance to the exact solution. This v alue tends to b e a lot larger for the unpreconditioned problem, which also b ecomes v ery hard to solv e for small α . On the right of Figure 2 we hav e plotted the percentage of problems that could not b e solv ed because the quadratic solv er reac hed its default iteration limit, something that nev er o ccurred with the preconditioned form ulation. 3.2 Effect of problem size No w we consider matrices J of size 50 n × 7 for n = 1 , 2 , . . . , 10. F or each size w e generated 200 matrices, and solv ed the problem min w ∈ Ω (20 , 50 n ) T r  h J T W ( w ) J i − 1  (11) 8 0 20 40 60 80 100 120 140 160 100 200 300 400 500 iter ation s Ma trix s ize k u (av g.,s. d.) k p (av g.,s. d.) Fig. 3 Average iterations of the SQP method solving the unpreconditioned and preconditioned problem (subscripts u and p , respectively). with and without preconditioner. W e plot a verage iteration counts in Figure 3, with error bars giving the standard deviations. Again we observe a significan t improv ement, on av erage, of the iteration counts for each problem. W e also observe that, for the unpreconditionend problem, the iteration coun t increases with the problem size n , whic h is lik ely due to the fact that the v alue of the minimum b ecomes smaller with matrix size, so that the effect predicted by Theorem 1 increases. Whether the iteration coun ts increase or not for the preconditioned v ariant is not p ossible to tell conclusively from this exp erimen t, but we conjecture that it do es. 3.3 F ull nonlinear problem Our original motiv ation was to find an adv antageous formula tion for full nonlinear exp erimen tal design as it is relev ant for applications in engineering. Thus we consider in our next experiment an experimental design problem defined on a sy stem of nonlinear differen tial equations. F or simplicit y , we choose the FitzHugh-Nagumo [7, 18] mo del, whic h is given by the system ˙ x 1 = x 1 − z x 3 1 − x 2 + I x 1 ( t 0 ) = x 0 , 1 , (12) ˙ x 2 = a · ( x 1 + b + cx 2 ) x 2 ( t 0 ) = x 0 , 2 . (13) The task is to find a exp erimental design to estimate the four fixed parameters z = 0 . 25, a = 0 . 02, b = 0 . 7, and c = − 0 . 8 using the v alues of I , x 0 , 1 , and x 0 , 2 as con trols, whic h are exp ected to satisfy the constrain ts − 5 < x 0 , 1 < 5 , − 5 < x 0 , 2 < 5 , − 1 < I < 0 . 5 . (14) A t most 30 measurements should be tak en. A measurement is the reading of the v alue of an y of the tw o v ariables at any of the times t i = 5 i , i = 1 , 2 , . . . , 100. The initial guess for the weigh ts is that all of them are equal. W e c ho ose the initial guess for the optimal controls randomly to be able to assess t ypical b ehavior as m uch 9 score h k p i h k u i h k u /k p i σ 5:0 46.0 260.8 3.9 1.7 T able 1 Performance statistics for the FitzHugh-Nagumo example. The v alues of k denote iteration counts, the subscripts u and p indicate that they refer to unpreconditioned and preconditioned varian ts. The angled brac kets indicate av erage, and σ stands for the standard deviation of the sp eed-up factor h k u /k p i -3 -2 -1 0 1 2 3 4 5 0 100 200 300 400 500 Concentration Time x 1 x 2 Meas. Fig. 4 Optimal design for the FitzHugh-Nagumo system as p ossible. Since it is p ossible to choose the controls in such a w ay that the least- squares problem is singular, we only use those for which T r([ J T ( q 0 ) J ( q 0 )] − 1 ) ≤ 100. The optimal v alues are usually three to four orders of magnitude low er than this upp er limit. W e repeated the exp eriment five times, and summarize the results in table 1. W e note again that the preconditioned form ulation leads to important gains in conv ergence sp eed. Here we ha ve that the problem is not conv ex for either formulation, and one observ es that there exist many local minima. F or a given starting v alue, th e t wo v ariants ma y or ma y not con verge to the same one. In Figure 1 we can observe the con vergence b eha vior of a t ypical run in terms of the length of the search direction. F or completeness, w e include in Figure 4 a solution of this optimization problem ( I = 0 . 37, x 0 , 1 = 5, x 0 , 2 = 3 . 76), together with the optimal measuremen t p oin ts. 4 Final remarks and outlook In this article, w e hav e studied the con v ergence problems of SQP methods when solv- ing experimental design problems. Our findings suggest that bad absolute condition n umbers are indeed the cause of slo w conv ergence. W e work around the generalit y of the problem by studying a carefully c hosen mo del problem inv olving the A -criterion. F rom the understanding gained ab out the conditioning of this model problem w e deriv e a transformation that guaran tees constan t absolute condition n umbers in this particular case. W e then provide strong experimental evidence that this transformation yields sig- nifican t impro vemen ts in the con vergence behavior of SQP methods when applied to relev ant settings, where w e observ e that the impro vemen t is more significan t for larger 10 problems. The transformation itself does not introduce an y noticeable additional com- putational cost. Ac kno wledgements This work w as supp orted by the German Ministry of Education and Research (BMBF) under grant ID: 03MS649A, and the Helmholtz asso ciation through the SBCancer program. References 1. Bauer, I., Bock, H., K¨ orkel, S., Schl¨ oder, J.: Numerical methods for optim um exp erimental design in D AE systems. J. Comput. Appl. Math. 120 (1-2), 1–15 (2000) 2. Bock, H.: N umerical treatment of inv erse problems in chemical reaction kinetics. In: K. Ebert, P . Deuflhard, W. J¨ ager (eds.) Mo delling of Chemical Reaction Systems, Springer Series in Chemic al Physics , vol. 18, pp. 102–125. Springer, Heidelberg (1981). URL http: //www.iwr.uni- heidelberg.de/groups/agbock/FILES/Bock1981.pdf 3. Dahlquist, G.: A sp ecial stability problem for linear multistep methods. BIT 3 , 27–43 (1963) 4. Demmel, J.W.: On condition num b ers and the distance to the nearest ill-posed problem. Numerische Mathematik 51 , 251–289 (1987) 5. F ateman, R.: Building algebra systems by o verloading lisp: Automatic differentiation. sub- mitted (2006). URL http://www.cs.berkeley.edu/ ~ fateman/papers/overload- small. pdf 6. F edorov, V.: Theory of optimal exp eriments. Academic Press, New Y ork and London (1972) 7. FitzHugh, R.: Impulses and physiological states in theoretical mo dels of nerve membrane. Biophysical Journal 1 (6), 445 – 466 (1961) 8. F ranceschini, G., Macc hietto, S.: Mo del-based design of exp eriments for parameter preci- sion: State of the art. Chemical Engineering Science 63 (19), 4846 – 4872 (2008) 9. Gill, P ., Murray , W., Saunders, M.: User’s Guide F or QPOPT 1.0: A Fortran P ack age F or Quadratic Programming (1995). URL http://www.sbsi- sol- optimize.com/manuals/ QPOPT%20Manual.pdf 10. Griewank, A.: On automatic differentiation. In: Mathematical Programming: Recen t De- velopmen ts and Applications. Kluw er Academic Publishers, Dordrech t, Boston, London (1989) 11. Griewank, A., W alther, A.: Ev aluating Deriv atives: Principles and T echniques of Algorith- mic Differentiation, second edn. SIAM (2008) 12. Hairer, E., W anner, G.: Stiff differen tial equations solv ed by radau methods. J. Comput. Appl. Math. 111 , 93–111 (1999) 13. Han, S.: A globally conv ergent metho d for nonlinear programming. JOT A 22 , 297–310 (1977) 14. Hida, Y., Li, X.S., Bailey , D.H.: Algorithms for quad-double precision floating point arith- metic. In: Proceedings of the 15th Symp osium on Computer Arithmetic, pp. 155–162. IEEE Computer Society Press (2001) 15. K¨ orkel, S.: Numerische Methoden f ¨ ur optimale V ersuchsplan ungsprobleme b ei nich tlin- earen DAE-Modellen. Ph.D. thesis, Univ ersit¨ at Heidelb erg, Heidelb erg (2002) 16. Lohmann, T., Bock, H., Sc hl¨ oder, J.: Numerical metho ds for parameter estimation and optimal experimental design in c hem ical reaction systems. Industrial and Engineering Chemistry Research 31 , 54–57 (1992) 17. Mommer, M.S., Sommer, A., Sc hl¨ oder, J.P ., Bo ck, H.G.: Differentiable ev aluation of ob- jective functions in sampling design with v ariance-cov ariance matrices. submitted (2011) 18. Nagumo, J., Arimoto, S., Y oshizaw a, S.: An active pulse transmission line sim ulating nerve axon. Proceedings of the IRE 50 (10), 2061 –2070 (1962). DOI 10.1109/JRPR OC.1962. 288235 19. Nocedal, J., W right, S.: Numerical Optimization, 2nd edn. Springer V erlag, Berlin Heidel- berg New Y ork (2006). ISBN 0-387-30303-0 (hardco ver) 20. Po well, M.: A fast algorithm for nonlinearly constrained optimization calculations. In: G. W atson (ed.) Numerical Analysis, Dundee 1977, L e ctur e Notes in Mathematics , v ol. 630. Springer, Berlin (1978) 11 21. Pukelsheim, F.: Optimal Design of Exp eriments. Classics in Applied Mathematics 50. SIAM (2006). ISBN 978-0-898716-04-7 22. Pukelsheim, F., Rieder, S.: Efficient rounding of approximate designs. Biometrik a 79 (4), pp. 763–770 (1992). URL http://www.jstor.org/stable/2337232 23. Saad, Y.: Iterativ e Methods for Sparse Linear Systems, 2nd edn. SIAM, Philadelpha, P A (2003) 24. Sager, S.: Sampling decisions in optimum exp erimental design in the light of p ontry agin’s maximum principle (2011). URL http://www.optimization- online.org/DB_HTML/2011/ 05/3037.html . (submitted) 25. Schittk owski, K.: The nonlinear programming method of Wilson, Han, and Po well with an augmented lagrangian type line search function. Numerische Mathematik 38 , 83–114 (1982). URL http://dx.doi.org/10.1007/BF01395810 . 10.1007/BF01395810 26. Sch¨ oneberger, J., Arellano-Garcia, H., Thielert, H., K¨ orkel, S., W ozny , G.: Optimal ex- perimental design of a catalytic fixed bed reactor. In: B. Braunsch weig, X. Joulia (eds.) Proceedings of 18th Europ ean Symposium on Computer Aided Pro cess Engineering - ES- CAPE 18 (2008) 27. Zolezzi, T.: On the distance theorem in quadratic optimization. J. Conv ex. Anal. 9 (2), 693–700 (2002) 28. Zolezzi, T.: Condition num b er theorems in optimization. SIAM J. Optim 14 (2), 507–516 (2003)