Modular Workflow Engine for Distributed Services using Lightweight Java Clients
In this article we introduce the concept and the first implementation of a lightweight client-server-framework as middleware for distributed computing. On the client side an installation without administrative rights or privileged ports can turn any …
Authors: R.-M. Vetter, W. Lennartz, J.-V. Peetz
PREPRINT 1 Modular W or kflow Engine f or Distr ib uted Ser vices using Lightweight J a v a Clients Ralf-Michael V etter , Werner Lennar tz, and J ¨ org-V olk er P eetz Abstract —In this ar ticle we introduce the concept and the first implementation of a lightweight client-ser v er-frame work as middleware f or distributed computing. On the client s ide an installation wit hout administrativ e rights or privileged ports can turn any computer into a worker node. Only a Ja va r untime environment and t he J AR files comprising the workflow c li ent are needed. T o connect all clients t o the engine one open server por t is sufficient. The engine submits data to the clients and orchestrates their work by workflow descr iptions from a central database. Cli ents request new task descriptions per iodically , thus the system i s robust against network failures . In the basic set-up, data up- and downloads are handled via HTTP communication with the ser v er. The performance of the modul ar system could additionally be improv ed using dedicated file ser v ers or distributed network file systems . We demonstrate the d esign feat ures of the proposed engine in real-world applications from mechanical engineer ing. We have used this system on a compute cluster in design-of-e xperiment studies, parameter opt imisations and robustnes s v ali dations of finite element structures. Index T erms —Jav a grid engine, workflow automation, distributed services, parameter optimisation, design-of-e xperiments ✦ 1 I N T R O D U C T I O N N O WA D AYS not only engineers and computer scien- tists use various simulation code s and data p re- and post-processing tools f or almost every ta sk during product development or experimental design. In the financial sector automated data analysis and predictions from numerical models are common. Integrating consec- utive tasks from distributed systems in workflows f rees the user from the need to transfer data and to invoke applications in d ue time. During the last de cade middleware such a s Globus T oo lkit [ 1], [2], [3], Unicore [4 ], [5], and gLite [6], [7] was developed in grid projects. These tools mainly f ocus on service-oriented grid infra structur es with seamless and secure ac c e ss to data sources for users without experience in grid computing. The middlewares expose applications via web services [8] to workflow ma nagers like T averna wo rkbench [9], [10] enabling users to design and exe cute complex workflows in graphical interfa ces and monitor the operations r emotely . Due to the i nherent complexity o f abstract database interfaces and multi- ple middleware layers specialists are needed to install, configure, and te st these tools. The specialist must be supported by the local administrators of the worker nodes as various tasks require administrative rights. Users without programming experience cannot d eploy • Dr . R .-M . V e tter is wi th the D epartment of Mechanical Engineering, Un i- versity of Applied Sciences Bonn-Rhein-Sieg, Sankt Augustin, Germany . E-mail: post@rmvetter .de • Dr . W . Lennartz is head of software development at Business - Acts Information and Data Systems, Cologne, Germany. E-mail: see http://businessacts.de • Dr . J. -V . P eetz is with the Department of Simulation Engineering, Fraun- hofer Institute for Algo rithms and Scientific Computing (SCAI), Sankt Augustin, Germany. E-mail: see http://scai.fraunhofer .de data streams in graphical interfa ces unless the a ccess to da ta and applica tions is encapsulated by pa rticular service wrappers. In our experience with scenarios like this developers spent most of the time setting up and d e- bugging the diverse tools. The construction of workflows via graphical user interfa ces, however , is not flexible or efficient enough to save a reasonable amount of time. As a typical example from mechanical engineering we have chosen numerical simulations where input decks containing finite element models have to pass through several workflow stages. At each stage a n applica tion processes the resulting data from a previous stage to become the input data of the subsequent one. In real- world scenarios we face many different applications and various restrictions, e.g. a CAD program to create a modified model only runs on a dedica ted W indows system, while the meshing generator uses an old but reliable Unix system, and the finite element simulation code runs on a Linux c luster without sufficient licences for all nodes being ava ila ble. T o expose a single ser vice it would no t make sense to install Globus T oolkit, Uni- core, or even gLite on a ll these nodes. But how can a bunch of lega c y software in a highly heterogeneous and distributed environment be orchestrated without having administrative rights for the machines? 2 T H E E N G I N E In the framework we cover below , we have inverte d the standard scena rio of grid computing where users act as clients contacting services on distributed hosts. Instead of wrapping ser vices on every host by com- plex midd le wa re f or accounting, workflow mana gement, and job control we dev e loped a lightweight grid client that can be deployed without installation. This client 2 PREPRINT Webserver Java Client 2 Server Engine User HTTP & workflows Client 1 Java ANSYS submits data Database JDBC Firewall Gnuplot HTTP Fig. 1. The frame work in a simple scenario: the ser ver engine utilises a database f or workfl ow descr iptions and a w eb ser ver for the communication with the clients . Applications are wr apped by lig htweight Ja va client s per i- odically query ing the server engine for ne w tas ks. program turns applications on different hosts into re- mote controllable ser vices. For each application a generic wrapper is e xtended to deal with the app lication-specific parameters and to c reate and start suitable batch scripts. New wrappers can be added to the syst em running without restart. At the central host a servlet container , i.e. Apache T omcat, processes the communication of all clients [11], [12]. A relational database maintains work- flow descriptions and par ameters for the applications [13], [14], see Figur e 1 . 2.1 Distributed w orkflows In an ex e mplary workflow the client downloads d a ta and processes it, i.e. it starts an a pplication editing the data. The client monitors the ongoing process a nd uploads the results afte r the application has finished. Further processing of the data resulting can be d elegated to a pplications on other clients. The clients communicate with a servlet container on the central host accessing an URL with their logical client name as par ameter , e.g.: http://server: 8180/engine/Tasks?Cl ient=node-01 The underlying servlet Tasks queries the serve r databa se for open tasks f or this client node-01 and generates an XML web page [1 5] with the workflow description of the tasks. In our simulation exa mple the requesting client receives the detailed description below as response: The engine stores parameters for each type of job in a separate table, in our example Download , Upl oad , JobAnsys , and JobParseAnsysEige nfreq . C lient a p pli- cations ca n be supplied with an arbitrary number of parameters. For ea ch p a rameter a column in the cor- responding job ta ble for this type of application has to be created. The table columns are the attributes of the embedded Parameter elements. The database table Download contains paths to the input decks for the Ansys simulations in the ser ver file system: select * from Download; +-----+--------+-- -----------------+------- -+----------+---------+ | Job | Server | | | | | | ID | Dir | File | Length | MD5Sum | LastMod.| +-----+--------+-- -----------------+------- -+----------+---------+ | 11 | models | inputde ck_001.dat | 20592 | fd13fb.. | 2009-.. | | 12 | models | inputde ck_002.dat | 20592 | dbf2e9.. | 2009-.. | +-----+--------+-- -----------------+------- -+----------+---------+ If available, the client uses the file length, the last- modified time stamp, and the MD5Sum [ 16] to verify the integrity of downloaded files. Repe ated downloads of id e ntical files c a n be avoided by comparing them to a local file cache. T able JobAnsys contains the para meters for a wrap p e r class with the identical name starting an Ansys simulation. The job wrappers will be d iscussed later in se c tion 2.2 . Here the input deck filename is the only paramete r to the Ansys job wrapper: select * from JobAnsys; +-------+--------- ----------+---------+ | JobID | Input | Timeout | +-------+--------- ----------+---------+ | 11 | inputdeck_001.dat | 1000 | | 12 | inputd eck_002.dat | 1000 | +-------+--------- ----------+---------+ The generic p a rameter Timeout is availa b le f or all kind of jobs, it is particularly useful for tests during the construction of new workflows. Once set, running jobs a re a utomatically a borted a f ter a previo usly de- fined period, here after 1000 seconds. The next job JobParseAnsys Eigenfreq parses the Ansys output and creates two files F reqfile and Mo defile for natura l fre- quencies and mode shapes. The Ansys output filename is not used a s a third pa r ameter because the de fault name is already hardcoded in the parser script: select * from JobParseAnsysEigenfreq; +-------+--------- ------+---------------+-- -------+ | JobID | Modefile | Freqfile | Timeout | +-------+--------- ------+---------------+-- -------+ | 11 | eigenm ode. asc | eigenfre q.asc | 120 | | 12 | eigenm ode. asc | eigenfre q.asc | 120 | +-------+--------- ------+---------------+-- -------+ Finally , the table Upload con tains the names of the two result files. The files are created on the client by the p a rser script, thus their length and MD5sum are unknown before. They a re uploaded to the server , col- umn ServerDir creates separate result directories in the server file system: select * from Upload; +-----+----------- ------+---------------+-- ------+-----+------+ | Job | | | | MD5 | Last | | ID | ServerD ir | File | Length | Sum | Mod. | +-----+----------- ------+---------------+-- ------+-----+------+ | 11 | results/sim_001 | eigenm ode.asc | | | | | 12 | results/sim_001 | eigenf req.asc | | | | | 13 | results/sim_002 | eigenm ode.asc | | | | | 14 | results/sim_002 | eigenf req.asc | | | | +-----+----------- ------+---------------+-- ------+-----+------+ In the previous tables each line describes a single job for a client. One ore more jobs can be combined to ta sks with unique task IDs. T asks a re the basic workflow elements of our engine. Clients either being on hold or active on the jobs of a single task. The next table shows an excerpt of the Tasks ’ table structure. The column Job specifies VETTER et al. : MODULAR WORKFLO W ENGINE FOR DISTRIBUTED SER VICES USING LIGHTWEIGHT JA V A CLIENTS 3 the na mes of the tables containing the job parameters while column Job ID indicates the corresponding line numbers. T he ta ble contains two similar tasks, each with five jobs for clients from the clu ster group. The first task ( TaskID=1 ) consists of a download job, a n Ansys simulation, a ca ll of the script to parse the result data, and two upload jobs : select TaskID, Job, JobID, Client, ClientGroup, Status from Tasks; +------+---------- --------------+-----+---- -+---------+---------+ | Task | | Job | Cli | Client | | | ID | Job | ID | ent | Group | Status | +------+---------- --------------+-----+---- -+---------+---------+ | 1 | Downloa d | 11 | | cluster | waiting | | 1 | JobAnsy s | 11 | | cluster | waiting | | 1 | JobPars eAnsysEigenfreq | 11 | | cluster | waiting | | 1 | Upload | 11 | | cluster | waiting | | 1 | Upload | 12 | | cluster | waiting | | 2 | Downloa d | 12 | | cluster | waiting | | 2 | JobAnsy s | 12 | | cluster | waiting | | 2 | JobPars eAnsysEigenfreq | 12 | | cluster | waiting | | 2 | Upload | 13 | | cluster | waiting | | 2 | Upload | 14 | | cluster | waiting | +------+---------- --------------+-----+---- -+---------+---------+ A task can be a ssigned to a specific client or to a client group, here cluster . In the mapping ta ble ClientGroups each client can b e associated with several groups, here some nodes of the cluster group are shown: select * from ClientGroups; +-------------+--- ------+ | ClientGroup | Client | +-------------+--- ------+ | cluster | node-01 | | cluster | node-02 | | cluster | node-03 | | cluster | node-04 | | cluster | ... | +-------------+--- ------+ T asks are assigned to the clients sorted by lowest ava il- able ID and status waiting . T o block the ta sk for other clients the Tas ks ser v le t changes the status to a ctive with the following SQL command [13], [1 4]: UPDATE Tasks SET Status= ’ a c t i v e ’ , Client= ’ $ c l i e n t ’ WHERE TaskID= ’ $ t a s k I D ’ ; In SQL statements we use ’$’-notation to indicate the values of va riables. The servlet assembles the SQL state- ments in string buffers replacing va riables by their val- ues, here it substitutes the logical name of the requesting client and the recent task ID. Further SQL statements process a ll jobs of this task by subsequent queries to the tables Download , JobAnsys , JobParseAn sysEigenfreq etc., i.e. they select the lines from the column Jo bID a nd generate the XML document with workflow instructions for the client. When a client is initially started, the server ’s IP a d - dress, a unique logical name for the client (e.g. node- 01 ), a local directory ( e .g. /tmp/node-01/ ), and optionally a client group can be specified. Downloads, uploads, and all a pplications only act on the da ta in this client direc- tory . The client in our example creates a working direc- tory Client Dir="ansys_001" for the Ansys task within /tmp/node-01/ . The optional para mete r S erverDir is used to relocate files in the directory structure during the up- and downloads, e. g. the input decks come from a special model library directory on the server . The A nsys results from different clients are a lso collected on the server in a joint directory res ults/ . Clients and server automatically create a ll directories from the workflow descriptions. Several clients with different client directories can run on the same ma chine without interfering. Independent clients with distinct logical names provide a convenient way to exploit the computing power of multicore and multiprocesso r systems. In our e x ample the models/ directory on the server comprises a set of input decks. The file system is exposed to the clients by the web server via HTTP . Clients can download files requesting an URL c onsisti ng of the server ’s IP address, the web server subdirectory , and the name of the file. The path information is URL- encoded to deal with potential whitespaces in directory or file names. For file uploads the clients send HTTP-POST forms and SOAP-with-atta chment requests [17] to the server . On the server side incoming multipart/form- data is handled by a particular up load servlet and the web se r vice exte nsion A xis2 [18]. These ser vices directly write uploade d files to the file system of the web server making the data immediately available for other clients. The input decks in the serv e r ’s m odels/ directory are part of different Ansys simulation runs. The first requesting client downloads inputde ck_ 001 .dat to its local working directory /tmp/node- 01/ansys_ 001/ , performs an Ansys simulation, and starts a script that parses the results to the files eigenm ode.asc and eigenfreq.asc in the same dir ectory . T able TasksWorkingD ir allows to allocate each task to a dif- ferent working directory on the clients: select * from TasksWorkingDir; +--------+-------- ---+-------------+ | TaskID | Dir | EraseOn Exit | +--------+-------- ---+-------------+ | 1 | ansys_001 | false | | 2 | ansys_002 | false | +--------+-------- ---+-------------+ Once the task is finished, the flag EraseOnExit au- tomatically d e letes the subdirectory . Subsequent ta sks can operate on the result data ca ched in the working directory , if set to false. 2.2 Job wrappers f or local applications Basic jobs for downloading and uploading data are integrants of the grid client. A workflow engine must be ca pable of wrapping further applications runn ing on the clients. In our framework a generic class Job is ex- tended to wrap new a pplications. The class name of the extended job wrapper corresponds to the job name in the Tasks table. The grid client automatically starts the job wrapper and supplies it with the job par ameters of the XML description. The extended wrapper class employs these parameters and assembles a valid script calling the application. The script is temporarily generated in an internal string buffer . A method of the generic job class replaces all aliases (see below) and writes it to the working d irectory of the c lient. The grid client uses Ja va Native Interface (JNI) to sta r t the script and monitor the activity of the applica tion [1 9]. In our example, the string buffer created by Job Ansys looks like this: #!/bin/bash cd /tmp/node-01/ansys_001/ 4 PREPRINT export LM_LICENSE_FIL E =ANSYS_ LICENSE ANSYS -b -i inputdeck_001.dat The shell header and the cha nge-the-directory command in line 1 and 2 are selected by the gener ic job class according to the operating system of the client. Fur- ther lines are created by the extended wrappe r class, ANSYS_LICENSE here defines a so-called a lias that can be understood as an internal envir onment var iable. Be fore the script is written to the working directory the generic job wrapper contacts a n Aliases servlet on the server and requests a list of known aliases for the given client. It then replaces all aliases by machine-specific parameters. The highst priority have aliases directly attached to a client’s na me, e.g. JAVA on client nod e-01 is mapped to /opt/ jdk1.6.0_12/bin/java . T hen the aliases f or entire groups of clients are e xchanged. Finally , global aliases not assigned to a client or client group (e.g. JAVA is mapped to java ) are a pplied: select * from Aliases; +---------------+- ------------------------- ---+---------+---------+ | | | Client | | | ServiceName | Service | Group | Client | +---------------+- ------------------------- ---+---------+---------+ | ANSYS | /opt/ansys/v11/bin/ansys11 | cluster | | | ANSYS_LICENSE | /opt/ansys/shrd/license.lic | cluster | | | JAVA | /usr/lib/jdk1.6.0/bin/java | linux | | | JAVA | /opt/jre-6-solaris/bin/java | solaris | | | JAVA | /opt/jdk1.6.0_12/bin/java | | node-01 | | JAVA | /opt/jdk1.6.0_12/bin/java | | server | | JAVA | java | | | | GNUPLOT | /us r/bin/gnuplot | linux | | | GNUPLOT | /op t/gnu/bin/gnuplot | solaris | | +---------------+- ------------------------- ---+---------+---------+ The next listing shows the extended job wrapper class for Ansys. The grid client creates a job class instance, assigns it with the logical client name, the work- ing directory dir , a nd Ansys-specific parameters, and starts the run() method in a separate thread. The call of replaceAl iases() ex c ha nges the a liases ANSYS and ANSYS_LICENSE in the generic wra pper script by machine-specific values from the Aliase s table. Based on the client’s name the Aliases servlet returns a list of all known aliases as key-value-pairs. Function createScript( ) writes the completed script to a file script_ansys in the client working directory . The thread waits at execu teScript() until the Ansys simu- lation has finished: public class JobAnsys extends Job { String inputdeck; // Parameter for Ansys public void setParameters(Element parameter) { inputdeck = getAttribute(parameter, ” I n p u t ” ); } public void run() { log( ” S t a r t i n g J o b A n s y s f o r ” + di r + ” / ” + inpu tdeck); StringBuffer t = new StringBuffer(); t.append( ” e x p o r t LM LI CEN SE F ILE= AN SY S LI CE NS E” + ” \ n ” ); t.append( ”A N S Y S − b − i ” + inputdeck + ” \ n ” ); Script s = new Script(dir, ” s c r i p t a n s y s ” ); s.createScript( getShellHeader() + replaceAliases(t.toString()) ); s.executeScript(); if (! ( new File(dir, ” mo de4 . png ” )).exists() ) { log( ” − J o b f a i l e d − ” ); setStatus(Job.JOB_ANSYS_FAILED); } flag_done = true ; // Signal to the main thread } } At the same time the main thr ea d mo nitors process load and disk space of the client in a loop. A boolean flag_done notifies the main thread that the Ansys job is finished. In our ex ample the Ansys input decks contain ad ditional instructions to create images of the simulated finite element models. The wrapper uses im- age mode4.png to a scertain whether the simulation was successfully finished and data for further processing is available. The job wrapper a utomatically redirects standard out- put and error streams to a log file in the working directory . W ith some applica tions running separately in a ba c kgro und thread, e. g. PamCrash impact simulations, only the content of the lo g file can be analysed to monitor the progress and determine when the job will be finished. The job wrapper passes the final sta tus of the job exe cution by an integer number to the grid client, indicating whether the simulation run has be e n successfully or not. After all jobs of a task are completed, the grid client contacts the servlet TaskCompleted on the central host and sends task ID, status information, and additional information (total runtime of the task, the recent p rocess load and free disk space) to the server . The status information allows the engine to decide whether to redistribute the same task, e.g. to a different client, or to tick it off and unlock further tasks depending on it (see section 2.5 , T a sk chains). If a client issues a request f or new tasks without having completed the last active task by calling TaskCo mpleted , e.g. be c ause the process was interrupted or the client was switched off, the status of the former task is again set back to waiting . The Tasks servlet implements this by an S QL update: UPDATE Tasks SET Status= ’ w a i t i n g ’ WHERE Client= ’ $ c l i e n t ’ AND Status= ’ a c t i v e ’ ; The interrupted client then resumes its work and restarts the task. 2.3 P arameter temp late files Sometimes command line arguments are not sufficient and special pa rameter files are needed to contr ol an application. Bef ore the a pplication is started parameter files can be downloaded from the central ser ver or a particular job can directly create them on the client. For the d ownload ca se we have developed a generic job that substitutes embedded tags in a par a meter template file by numerical values from a list. Parameter template files are useful in design-of-experiment studies, numerical optimisations, or analysis of the robustness of a solution, where many different sets of parameters have to be tested. Therefore the pre-defined template file is deployed on the ser ver and the job ReplaceTags genera tes a valid parameter file for a simulation run from the downloaded template. This jo b parses the template a nd looks for VETTER et al. : MODULAR WORKFLO W ENGINE FOR DISTRIBUTED SER VICES USING LIGHTWEIGHT JA V A CLIENTS 5 placeholders in XML-element form. The names of the el- ements ca n be arbitrarily chosen to avoid collisions with keywords of the parameter file. The manda tory attribute ID ser ves as a key to replace the entire tag by a numerical value from the parameter list of the R eplaceTag job. T o handle fixed- format files with strict column widths a n optional attribute L en specifies the number of characters, the numerical values hav e to be accordingly rounded down or extended with whitespaces. The XML ta gs can be extended b y further a ttributes, i.e. we have determined minimum and maximum bounds (or mean value and deviation) f or each pa rameter . T his e nables ReplaceTag to substitute the pla ceholders by random values from the defined range (or values from a nor- mal distribution around the mean value). This fe ature allows the user to select and alter ar b itra ry parameters in de sign-of-e xperiment studies on the fly without any changes to the workfl ow . The next listin g provides an exce r pt of an input deck template for Ansys [2 0] that we use in tests with finite element simulations. The first line defines a square solid plate with edge length a , the next three lines c onstruct solid cylinders at fixed positions inside the plate. In the input deck we substitute the radii of the cylinders by tags with identifiers r1 , r2 , and r3 . The la st line d rills holes into the structure subtracting the cylinder shapes from the plate (cf. Fig. 2): RECTNG,0,a,0,a CYL4,32e-3,32e-3, ; ; The last task is assigned to a grid client on the server with access to the local da tabase via JDBC [14], c on- fer Fig. 1 . Like every other client it first downloads the results to a local working directory temp / before a second job inserts them to the ta ble AnsysResult s . The directory extension ” _001 ” is incremented in each simulation run, in the result table it acts as a unique ID. The workflow submission package provides a generic way to assemble the XML description and convey it to the server . A workflow submission client is designed to work within the loop of an optimisation a lgorithm. It is triggered by the subroutine that usua lly evaluates the objective function. During the first call J NI Invocation [19] initialises a persistent instance of a virtual machine within the C/C++ or Fortran subroutine to hold the Java submission client. Parameter guesses from the optimisa- tion libra ry are tra nsparently passed to the client without restarting the virtual mac hine at each call. The client program inserts the guesses to the XML template and sends the description to a workflow submission servlet. The c lient waits until the workflow is processed and the value of the objective function is a v a ilable on the web server as a r esult file. It returns the value to the calling function of the optim isation library . Insertion of tasks and jobs in the engine ta - bles and linkage of the dependencies are estab- lished by the submission servlet controlled by at- tributes in the XML workflow description. The attributes start/closeTa skChain and start/closeTaskG roup mark the beginning and end of task chains and groups. The attribute startTask Chain establishes a linear chain: every ta sk except the v e ry first is blocked by its direct predecessor (cf . section 2.5). The flag startDependsOnGro upNode indicates that all tasks of a chain have to b e execute d on the same node of a client group. It allows to distribute task chains to client groups where the client receiving the first task will also process all consecutive tasks of the chain. This allows more flexiblity than assigning all pending jobs to one single task, since client monitoring and job repetition can only be applied on a task-wide sca le. The flag is stored in the column DependsOn GroupNode of the Tasks table, and the servlet Task Completed updates the dependencies after a task f rom a chain has been completed: UPDATE Tasks SET Clien t = ’ $ c l i e n t ’ , ClientGrou p = NU LL WHERE Depend sOnTask = ’ $ t a s k I D ’ AND Depends OnGroupNod e = ’ t r u e ’ ; Attribute Type a nnotates the types of da ta for the database ta ble c orresponding to the job’s name. If not in place, the workflow engine automatically constructs 10 PREPRINT the table JobReplaceTag with columns r1 , r2 , r3 of type DOUBLE for the pa r ameter values and Input , Output of type VARCHAR for the names of the input or output files respectively . The attribute startTaskGroup is used to split the workflow and create pa rallel branches. Each branch of a task group constitutes an independent chain, the initial tasks of all chains are coupled to the same ID of a preceding task. T his preceding task unblocks all branches simultaneous ly . Conversely close TaskGroup acts as a thread b a rrier waiting until all tasks of a group are completed. It employes the column TaskGroup of the Tas ks table and a dummy task as special job MonitorTaskGr oup moni toring the status of all tasks with a c e rtain group ID. In the following exa mple two independent branches (tasks 3 78 and 384) are a lready processed by nodes 04 and 02. Nodes 01 - 04 are now active on further tasks (cf. dotted lines): select * from Tasks; +-----+---------------+ -----+--------+---------+----- ----+------+-------+---------+ | Task| | Job | Dep’s | | Client | DepOn| Task | | | ID | Job | ID | OnTask | Client | Group | GrNod| Group | Status | +-----+---------------+ -----+--------+---------+----- ----+------+-------+---------+ |.376.|.Simulation....|..56.|........|.node-03.|.cluster.|......|...375.|.active..| | 377 | Upload | 86 | 376 | | cluster | true | 375 | waiting | | 378 | Simulation | 57 | | node-04 | cluster | | 375 | done | | 379 | Upload | 87 | | node-04 | cluster | true | 375 | done | |.380.|.Simulation....|..58.|........|.node-01.|.cluster.|......|...375.|.active..| | 381 | Upload | 88 | 380 | | cluster | true | 375 | waiting | |.382.|.Simulation....|..59.|........|.node-04.|.cluster.|......|...375.|.active..| | 383 | Upload | 89 | 382 | | cluster | true | 375 | waiting | | 384 | Simulation | 60 | | node-02 | cluster | | 375 | done | | 385 | Upload | 90 | | node-02 | cluster | true | 375 | done | |.386.|.Simulation....|..61.|........|.node-02.|.cluster.|......|...375.|.active..| | 387 | Upload | 91 | 386 | | cluster | true | 375 | waiting | | 388 | Simulation | 62 | | | cluster | | 375 | waiting | | 389 | Upload | 92 | 388 | | cluster | true | 375 | waiting | |:390:|:MonitorTaskGr:|:375:|::::::::|:::::: :::|:::::::::|::::::|:::::::|: active::| | 391 | CreateReport | 13 | 390 | server | | | | waiting | +-----+---------------+ -----+--------+---------+----- ----+------+-------+---------+ The monitor is implemented in the T asks servlet. If the number of ta sks with group ID 375 and status unequal to don e turns zero, the servlet sets the monitor task to done and unblocks dependent tasks, here task 391 that generates a report of all performed simulation runs. Using the workflow submission package we have interfaced the e ngine with the Adaptive Simulated An- nealing optimisation library [32], [33]. Simulated An- nealing attempts to find the global minim um of the objective function in a stochastic process [34]. In Fig- ures 4 and 5 the bla ck dots indicate the progress of the optimisation procedure. The final para meter set r ∗ = (3 . 521 , 3 . 755 , 5 . 937) mm yields a plate with the exact frequency spectrum demanded. In the next section we analyse how robust the solution will be if the pa rameters differ f rom this optimal setting. 3.3 Robustness of solutions The data from the previous sections is employed in a visualisation workflo w analysing the robustness of a design against para meter fluctuations. The workflow automatically creates diagrams with the interactive d ata plotting utility Gnuplot [35] and the R package for statis- tical computing [36]. T he para meter sets from the DOE study and the optimisation process are reused to inter- polate the frequency response f ( r ) around the optimum value r ∗ . V arious simulated points in the immediate vicinity allow a de ta iled prediction for f ur ther parameter sets without a dditional numerica l simulations. A ser vice supplied with a se t of measured points and a list of parameters with unknown va lues executes the interpolation. T he ser vice tessellates the measured points using Qhull [37], [ 38], in the 3-dimensional parameter space, i.e. it constructs tetrahedrons between every four points with no further point inside. The va lues of ar- bitrary paramete r points are then interpolated by the corner values of the tetrahedron containing the new point. W e have e mployed the interpolated frequency response to examine different aspects of the robustness of the optimal design: • T o get frequency contour lines in the parameter space as depicted in Figure 6 we have evaluate d the frequency respons e function on regular grids in three perpendicular pla nes. The contour lines show how the value of the objective function varies for pa- rameter sets next to the optimal point r ∗ (black d ot). Large distances b e tween c ontour lines indicate areas of robust solutio ns with the frequency value being only slightly a ffected by parameter inacc ur acies. The reliability of this prediction depends on the distance between an interpolated parameter point r and the next simulated point, cf . the grey background scale in Fig. 6. Since part of the da ta stems from the optimisation run many simulated points a re close to r ∗ . • W e have also interpolated the f unction on a cloud of random points r around the optimum. Figure 7 shows the resulting f requencies depending on the distance d = | r − r ∗ | to the optimal point r ∗ . Deviations from the optimal parameter set result in enlarged frequency spectra, the solid lines c onfine the spec tr a a nd specify worst case frequency values for parameter inaccura c ies. • Ca pability study techniques tr ea t inevitable inacc u- racies in the manufacturing processes by specific parameter distributions, e .g. the realisations r are assumed to be normal distributed a round their op- timum value r ∗ . Histograms or box plots of the resulting frequency spe c trum are used in Six Sigma analysis to determine how the manufa cturing pro- cess meets specification limits [39]. For the last case we have generate d 10,0 00 paramete r sets r = ( r 1 , r 2 , r 3 ) follow ing a multivaria te normal probability de nsity with mean r ∗ and standard deviation σ = 0 . 01 mm f or all D = 3 par ameters: p ( r ) = 1 (2 π ) D/ 2 σ exp " − 1 2 r − r ∗ σ 2 # (2) T o de a l with independent or correlated d e viations ( σ 1 , σ 2 , σ 3 ) a more general form of (2) can be applied, see [40]. The results are summarised in Figure 8. For the given manufacturing precision almost 95% of the plates have natural frequencies in the range f Mo de4 = [27 . 97 3; 28 . 02 9] , f Mo de5 = [29 . 964; 3 0 . 039] , and f Mo de6 = [32 . 983; 3 3 . 013] kHz. The diagrams a re created by R a nd Gnuplot jobs in VETTER et al. : MODULAR WORKFLO W ENGINE FOR DISTRIBUTED SER VICES USING LIGHTWEIGHT JA V A CLIENTS 11 Mode 4: 28 kHz 3.45 3.5 3.55 3.6 Radius 1 [mm] 3.7 3.75 3.8 3.85 Radius 2 [mm] Mode 5: 30 kHz 3.45 3.5 3.55 3.6 Radius 1 [mm] Mode 6: 33 kHz 3.45 3.5 3.55 3.6 Radius 1 [mm] 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.6 0.5 0.4 0.3 0.2 0.1 3.7 3.75 3.8 3.85 Radius 2 [mm] 5.84 5.86 5.88 5.9 5.92 5.94 5.96 5.98 6 6.02 Radius 3 [mm] 3.7 3.75 3.8 3.85 Radius 2 [mm] 3.7 3.75 3.8 3.85 Radius 2 [mm] 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.6 0.5 0.4 0.3 0.2 0.1 3.45 3.5 3.55 3.6 Radius 1 [mm] 5.84 5.86 5.88 5.9 5.92 5.94 5.96 5.98 6 6.02 Radius 3 [mm] 3.45 3.5 3.55 3.6 Radius 1 [mm] 3.45 3.5 3.55 3.6 Radius 1 [mm] 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.6 0.5 0.4 0.3 0.2 0.1 Fig. 6. Robustness of the solution: Hole radi i r ∗ 1 = 3 . 5 21 , r ∗ 2 = 3 . 755 and r ∗ 3 = 5 . 937 mm (point in the center) re veal e xac tly the desired mode frequencies. Contour lines indicate the frequency va r iation of deviatio ns from this optimal point, denoted in per mil le of the or iginal value, i.e. fo r mode 5 (30 kHz) the first line ” 0.1 —” c orresponds to 29.997 and 30.003 kHz. The c ontour line s derive from the inter polatio n of the scattered simulation data. Their reliability is hea vily dependent of the distance to the closest simulation point in the parameter space. The distance from interpolated points to the nearest simulated point is depicted as grey scale in the background in [mm]. In white areas a simulated parameter set ( r 1 , r 2 , r 3 ) is alwa ys within the close distance of maximal 0.03 mm. the workfl ow chain. W e have manually assembled a prototype of each dia gr a m and used it as a template for wrappers of the visualisation programs. W orkflows then produce all diagrams supplying the scripts with alternat- ing pair s of column numbers and the corresponding axis labels. The listing shows a wrapper for Gnuplot creating the contour plots f rom Figure 6. The file s1 contains the Gnuplo t controls from the te mpla te, script s2 starts Gnuplot with these c ontr ols: public class JobGnuplot extends Job { String inputfile; String outputfile; String colX; ... public void setParameters(Element parameter) { inputfile = getAt tribute(parameter, ” I n p u t ” ); outputfile = getAttribute(parameter, ” Ou t pu t ” ); colX = getAttrib ute (param eter, ” Colu mn X” ); ... } public void run() { StringBuffer t = new StringBuffer(); t.append( ” s e t t e r m p o s t c o l o r s o l i d e n h a n c e d 8 ” + ” \ n ” ); t.append( ” s e t o u t ’ ” + outputfile + ” ’ ” + ” \ n” ); t.append( ” s e t t i t l e ’ ” + title + ” ’ ” + ” \ n ” ); t.append( ” s e t g r i d ” + ” \ n” ); t.append( ” u n s e t s u r f a c e ” + ” \ n” ); t.append( ” s e t c o n t o u r b a s e ” + ” \ n” ); t.append( ” s e t c n t r p a r a m l e v e l s i n c r e m 0 . 0 , 0 . 1 ” + ” \ n ” ); t.append( ” s e t v i ew 0 , 9 0 ” + ” \ n” ); t.append( ” s e t x l a b e l ’ ” + xla bel + ” ’ ” + ” \ n” ); t.append( ” s e t y l a b e l ’ ” + ylabel + ” ’ ” + ” \ n ” ); t.append( ” s e t x r a n g e [ ” + x1 + ” : ” + x2 + ” ] ” + ” \ n ” ); t.append( ” s e t y r a n g e [ ” + y1 + ” : ” + y2 + ” ] ” + ” \ n ” ); t.append( ” s p l o t ’ ” + input file + ” ’ u s i n g ” + colX + ” : ” + colY + ” : ” + colZ + ” w i t h l i n e s ” + ” \ n ” ); String controlfile = ” g n u p l o t . c o n t r o l ” ; Script s1 = new Script(dir, controlfile); s1.createScript( t.toString() ); String text = ”G N U P L O T ” + co ntrolfile; Script s2 = new Script(dir, ” s t a r t g n u p l o t ” ); s2.createScript(getShellHeader() + replaceAliase s (text)); s2.executeScript(); flag_done = true ; } } Figures 6 - 8 show the results of the visualisation work- flows. In the diagrams only the influen ce of design 12 PREPRINT 27.8 27.85 27.9 27.95 28 28.05 28.1 28.15 28.2 Frequency [kHz] Mode 4 29.8 29.85 29.9 29.95 30 30.05 30.1 30.15 30.2 Frequency [kHz] Mode 5 32.8 32.85 32.9 32.95 33 33.05 33.1 33.15 33.2 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 Frequency [kHz] Distance d to optimal parameter set r [mm] * Mode 6 Fig. 7. V ar iation of mode frequency with parameters ( r 1 , r 2 , r 3 ) = r differing form their optim um v alue r ∗ . The Distance from the optimal v alue is defined by d = ( r 1 − r ∗ 1 ) 2 + ( r 2 − r ∗ 2 ) 2 + ( r 3 − r ∗ 3 ) 2 1 / 2 , r ed cr osses mar k data points from simulatio ns runs, blac k dots are inter po- lated. Solid lines indicate w ors t expected values , i.e. the small ap er tur e a ngle indicates that mode 6 is most robust against parameter v ar iation. Mode 4 Frequency [kHz] Count 27.96 28 28.04 0 500 1000 1500 Mode 5 Frequency [kHz] 29.96 30 30.04 Mode 6 Frequency [kHz] 32.96 33 33.04 Fig. 8. Distribution s of natural frequencies if hole radii r fo llow normal distrib utions around their optimal value r ∗ with standard de viation σ = 0 . 01 mm, i.e . appro ximately 95 % of the samples are within range r ∗ ± 2 σ . 10,00 0 inter- polated simulations were used to create the histog rams and the boxplots at the bottom, green bo xes mark 50 % quar tiles for the results. parameter inaccura cies are examined. Other parameters, not subject to the design or optimisation procedure, may have more impact on the robustness of a solution, e.g. the thickness of the plate, hom ogeneity of the material, positions of the holes, or frequency shifts by temper- ature. Their infl uence can be evaluated in additional DOE studies applying the same methods and automated workflows discussed bef ore. 3.4 Fur ther aspects 3.4.1 Stateful ser vices The interpolation service from the previous section ap- plies a list of simulated parameters with known f re- quencies a s input. Every call of the service results in a new tessellation of all p oints from the list. This re- dundancy can b e avoided by a state f ul service, where the behaviour in consecutive ca lls depends on the actua l state of the service. In the case of the interpolation service, the tessellated models are stored in the local working directory . The MD5 algorithm [1 6] c alculates hash keys f or the content of the input files. T o label the tessellated models keys with the length of 3 2 char acters are used as file names. In every call the service first checks whether a model file f or the input da ta exists. When identical input d ata occur , as many times during the generation of the diagrams, the service works with the already tessellated models from the repository . An implementation of the interpolation as a web service further increases the performance, since the model data can be held in the main memory of the application server . 3.4.2 Billi ng As the runtime of all processed tasks is logged in the table TasksCompleted it ca n be used to implement a billing system for expensive simulatio n ta sks. In the database we have selecte d the r untimes of, e.g., all Ansys simulations and grouped them by users. T he run- times are multiplied with pe rformance f a ctors from the Clients table describing the individual spee d of a node. If more processes than available processors (or cores) a re active the runtimes are divided by load factor . The grid client monitors the load during the job execution and periodically sends it to the serve r . The time interval is customisable, the default is every two minutes. For the billing system a cron job logs the va rying load s from table ClientsAvai lable for fur ther evaluations. W e developed standard benchmark jobs to automati- cally get presets for the pe r formance factors, e.g. we use a program to calculate the Mandelbrot set [4 1] for d if- ferent resolutio ns and iteration d epths. The benchmark jobs are e x ecuted on the clients a t different load levels. The runtimes of the jobs serve as a relative measure for the preformance of the clients. VETTER et al. : MODULAR WORKFLO W ENGINE FOR DISTRIBUTED SER VICES USING LIGHTWEIGHT JA V A CLIENTS 13 3.4.3 Dedicated ser vers and distributed file systems In the ba sic set-up, the web server for file transfers, the servlet c ontainer for communication with the clients, and also the data base engine are active on the same host. Especially during the transmission of extensive data the web server imposes a bottleneck and slows down the concurrent processes. The central file repository can be relocated to a dedicated web server to enhance the performance. In the default configuration the grid c lient contacts ser vlet container and web server at the same address. The concept of a server file system on the engine also allows to a pply more sophisticated methods for data exchange. The system was originally designed to deal with clients behind fir ewa lls interac ting only via HTTP with the centra l server . If the clients share common directories via network file systems, e.g. Samba [42], NFS [43], or AFS with ca ching facilities [44], [45], the central file repository can be replaced by such a resource. In this scenario download and upload jobs are simplified to native methods that copy da ta be tween local directories. Even now the network file system is responsible to share the data between the clients, this change does not effect the abstract workflow descriptions. Also peer-to-peer file systems, which are highly op- timised for data throughput, c an be embedded as a transport layer , e.g. XtreemFS [46]. They organise d ata in chunks and distribute them to all network nodes. Ac- cess to remote da ta initiates pa rallel data transfers from different hosts. Since the data chunks are replicated on several locations, the system is robust against fa ilures of single nodes. In ea ch case the workflow engine remains responsible for the synchronisation of the data transfers. A consecutive download process of a client will not be initiated bef ore the upload of the data f rom an other client has been finished. 4 C O N C L U S I O N The proposed platf orm indepe ndent client-server-system quickly turns a large number of heterogeneous comput- ers into a flexible computing grid without expensive installations or administrator rights. T he system is ro- bust against unreliable network connections or firewall restrictions as the clients periodically contact the server engine via stateless HTTP . The engine utilises d a tabase tables to implement the logic of c omplex parallel work- flows. It is based on a table for ta sks and servlets to communicate with the clients. Further tables and ser vlets extend the modular system to exe c ute customised a ppli- cations, pla tform independent services, workflow sub- mission, and status information of the system itself. All tables’ structure is self-explanatory . The data is processed and combined by SQL statements from the servlets. The XML workflow descriptions a nd the database ta bles provide generic interfaces to couple further a pplications, such as e.g. optimis ation libraries with the syst em. The engine distinguishes job execution from data transmission. Client applications only work on data in local directories while the system is responsible to transfer the data. T e sting and d ebugging of complex workflows benefit from the fact that a ll intermediate data is accessible in the client’s working directory . W orkflow chains can be tested step-by-step as job wrappe rs for applications can be started manually using their main() method. Data transfer can be substituted by a lter native methods, e. g. network file systems or distributed peer- to-peer networks. The development of the engine was motivated by ac- tual requirements of engineering and scientific comput- ing. W e have demonstrated how to accomplish design- of-experiment studies and par ameter optimisations in finite element applications. W e want to stress that pre- and post-processing and data visualisation routines can easily be integrate d in the workflow chains. The hi- erarchical description allows to extend and recombine existing workflow s. The data ma nagement is extremely simplified as a ll application pa rameters and results are available in the d atabase tables of the e ngine. Currently , we are workin g on a consolidated version of the system that we plan to release as open source project. Besides a consistent renaming of the cla sses a nd a complete refactoring of the code, we aim to integrate a database a b stra ction layer to simplify the integration of client applications, e. g. Hiberna te /JP A [47]. Hibernate encapsulates da tabase-specific implementations and cre- ates table schemes directly from job wrapper entities. In addition, we are developing a web interface to control the engine by browser . W e plan to enhance data trans- fer f or cluster computing, where clients are mutually connected and data transfer via a central server is an redundant detour . A C K N OW L E D G M E N T S The authors would like to thank Y vonne Ha vertz from University of Cologne, Dr . Annette Kuhlmann from Shell Germany , and Dr . Detlef Billep from Fra unhofer Institute ENAS. R E F E R E N C E S [1] S. Brunett, K. Czajkowski, S. Fitzgerald, I. Foster , A . John son, C. Kesselman, J. L e igh, and S. T uecke, “Application experiences with t he Globus t oolkit,” in H PDC , 1998, pp. 81–89. [2] L. C. B orja Sot om ayor , Globus T oolkit 4: Programming Java Services . Morgan Kaufmann, 2006. [3] Globus T oolkit for building grid systems and applications. [Online]. A vailable: http: //www .globus.org/toolkit [4] Un icor e grid system. [Online]. A v ailable: http: //www .unicore.eu [5] B. Bierbaum, C. Clauss, T . Eickermann, L. Kirtchakova, A. Krechel, S. Springstubbe, O. W aeldrich, and W . Ziegler , “Re- liable orc he st ration of distributed mp i-applications in a unicore- based grid with metampich and metascheduling,” ser . Lecture Notes in Comp ut er Science, vol. 4192. Springer , 2006. [6] S. Campana, D. Rebatto, and A. S ciaba, “Experience with the glite workload man agement system in atlas monte carlo production on lcg,” J. Phys. Conf. Ser . , v ol. 119, p. 052009, 2008. [7] GL ite middleware for grid computing. [Online]. A v ailable: http://glite.web.cern.ch 14 PREPRINT [8] M. P . Papazoglou, Web Services: Principles and T echnology . Pr entice Hall, 2007. [9] T av erna workbench for d e signing and executing workflows. [Online]. A vailable: http: //tav erna.sourc eforge.net [10] D. Hull, K. W olstencroft, R. Stev ens, C. Goble, M. R. Pocock, P . Li, and T . Oinn, “T averna: a tool for building and running workflows of services,” Nucl. Acids Res. , vol. 34, p p. 729–732, 2006. [11] J. Hunter and W . Crawfor d, Java Servlet Programming . O’Reilly Media, 2003. [12] J. Brittain and I. F . Darwin, T omcat: T he Definitive Guide . O’Reilly Media, 2003. [13] M . Kofler and D. Kramer , The Definitive Guide to M ySQL 5 . Apress, 2005. [14] G . Reese, Data base Programming with JDB C and Java . O’Reilly Media, 2000. [15] B . McLaughlin, Jav a and XML: Solutions to Real -World Problems . O’Reilly Media, 2001. [16] M D5Sum calculates 128-bit h ashes. [Online]. A vailable: http://en.wikipedia.org/wiki/Md5sum [17] O. Zimmermann , M . R. T omlinson, and S. Peuser , Perspectives on Web Services: Applying SO AP , WSDL and U DDI to Real-W orld Projects . Springer , 2005. [18] T . K. I. Kent , Developing Web Ser vices with Apache Axis2 . Book- surge, 2008. [19] R. Gordon, E ssential JNI: Java Native Interface . Prentice Hall, 1998. [20] Y . Nakasone, S. Y oshimoto, and T . A. Stolarski, Engineering Anal- ysis with ANSYS Software . B utterworth Heinemann, 2007. [21] D. J. Barrett and R. Silverman, SS H , T he Secure Shell: T he Definitive Guide . O’Reilly , 2001. [22] Jav a Secur e Channel (JSch) implementat ion of SSH2. [Online]. A vailable: http://www .jcraft.com/jsch [23] E . Pitt and K. McNiff, Java.rmi: T he Rem ote Method Invocatio n Guide . Addison–W esley , 2001. [24] E . Siever , S. Spainhour , J. P . Hekman, and S. Figgins, Linux in a Nutshell . O’Reilly , 2005. [25] Cron time-based job scheduler for unix-like systems. [Online]. A vailable: https://help.ub untu.com/community/CronHow to [26] Z. Uykan, “A temporal round robin scheduler ,” in IEEE 68th V ehicular T echnology Conference Proceedings , Calgary , Canada, 2008. [27] Y . Ito, S. T asaka, and Y . Ishibashi, “V ariably weighte d round robin queueing for core ip routers,” ser . Performance, Computing, and Communications Conference, 2002. 21st IEEE Inte rnational, 2002, pp. 159–166. [28] D. C. Montgomery , Design and Analy sis of Experiments . W iley , 2004. [29] D. C. Montgomery and R. H. M y e rs, Response Surface Methodol- ogy: Process and Product Optimizatio n Using Designed Experiments . W iley-Interscience, 2002. [30] J. T ate, B. W oolford-Lim, I. Bate , and X. Y ao, “Comparing d esign of experiments and evolutionary appro aches to multi-objective optimisation of sensornet protoc ols,” in Proceedings of the 10th IEEE Congress on Evolutionary Computation , May 2009, pp. 1137– 1144. [31] J. Goupy and L . Creighton, Introduction to Design of Experiments with JMP Examples . SAS Press, 2007. [32] L . Ingb er , “Ad aptive simulated ann ealing (asa): Le ssons learned, ” Contro l and Cybernetics , vol. 25, pp. 33–54, 1996. [Online]. A vailable: http://arxiv .org/abs/cs /0001018 [33] A daptive Simulated Annealing (ASA) for global optimisation. [Online]. A vailable: http: //www .ingber .com/ [34] S . Kirkpatrick, C. D. Gelatt Jr . , and M. P . V ecchi, “Optimization by simulated annealing,” S cience. New Series , vol. 220 No. (4598), pp. 671–680, 1983. [35] G nuplot - portable comman d -line driven graphing utility . [Online]. A vailable: http: //www .gnuplot.info [36] T he R Project for statistical computing. [Onli ne ]. A vailable: http://www .r-pro ject. org [37] Qh ull - computing convex hull an d Delaunay triangulation. [Online]. A vailable: http: //www .qhull.org [38] C. B . Barber , D. P . Dobkin, and H. Huhdanpaa, “The quickhull algorithm for convex hulls,” ACM T rans. Math. Soft. , v ol. 22, pp. 469–483, 1995. [39] T . T . Allen, Introduction to Engineering Statistics and Six SIGMA: Statistical Quality Contr ol and Design of Experiments and Systems . Springer , 2006. [40] R. O. Duda, P . E. Hart, and D. G. Stork, P attern Classification . W iley-Interscience, 2000. [41] B . B. Mande lbrot, The Fractal Geometry of Nature . Henry Holt, 2000. [42] R. W . Smith, Linux Samba Server Administration . Sybex, 2000. [43] B . Callaghan, NFS Illustrated . Addison-W esley Professional, 2000. [44] R. Campbell and A. Campbell, Managi ng AFS: The Andr ew File System . Addison-W esley Professio nal, 2000. [45] Ope n AFS. [Online]. A vailable: h t tp://www .openafs.org [46] Xt reemFS distributed and replicated file system. [Online]. A vailable: http://www .xtreemfs.org [47] C. B auer and G. King, Java P e rsistence with Hibernate . M an ning Publications, 2006.
Original Paper
Loading high-quality paper...
Comments & Academic Discussion
Loading comments...
Leave a Comment