Utility Optimal Scheduling in Processing Networks

1 Utility Optimal Scheduling in Processing Networks Longbo Huang, Michael J. Neely Abstract —W e consider the problem of utility optimal schedul- ing in general processing networks with random arriv als and network conditi ons. These are generalizations of traditional d ata networks wh ere commodities in one or more qu eues can be combined to produce new commodities that are deliver ed to other parts of the network. Th is can b e used to model p roblems such as in-network data fusion, stream processing, and grid computing. Schedulin g actions ar e complicated by the un derflo w problem that arises when so me queues with required com ponents go empty . In this paper , we dev elop the Perturbed Max-W eight a lgorithm (PMW) to achieve optimal util ity . The idea of P MW is to perturb the weights used by the usual Max-W eight algorithm to “push” queue lev els towards non-zero v alues (av oiding underflows). W e show that wh en the p erturbations are carefully chosen, PMW is able to achiev e a util ity th at is with in O (1 /V ) of the optimal value f or any V ≥ 1 , while ensuring an av erage n etwork backlog of O ( V ) . Index T erms —Dynamic Control, Processing Networks, Data Fusion, L yapunov Analysis, Stochastic Optimization I . I N T R O D U C T I O N Recently , there h as been much attention on developing opti- mal schedu ling algorithms for the class of pr ocessing networks e.g., [1], [2], [3], [ 4], [5]. These networks are g eneralization s of trad itional data n etworks. Contents in these n etworks can represent information , data pac kets, or cer tain raw ma terials, that n eed to go through m ultiple pro cessing stages in the network b efore they can be utilized. One example o f su ch processing networks is the Fork a nd Join network co nsidered in [4], which models, e.g., stream processing [6] [7] and grid computin g [8]. In the stream processing ca se, the conten ts in the network repre sent d ifferent types of da ta, say voice and video, that need to be combin ed or jointly com pressed, and the network top ology represen ts a pa rticular sequenc e of oper ations that needs to be conducted durin g pro cessing. Another example of a processing network is a sensor network that perfo rms da ta fusion [9], in which case sensor data must first be fused before it is d eliv ered. Finally , th ese processing networks also contain the class o f manufactur ing networks, where raw materials ar e assembled into p roducts [3], [5]. In this pap er , we develop o ptimal schedulin g algorithm s for th e f ollowing gener al utility maximization problem in processing networks. W e are g iv en a discrete time stochastic processing n etwork. The network state, which descr ibes the network random ness (such as ran dom channel conditio ns or commod ity arrivals ), is time varying accordin g to some p rob- ability la w . A network controller performs some action at every Longbo Huang (web: http:// www-scf.usc.edu/ ∼ long bohu ) and Michael J. Neely (web: http:/ /www-rcf.usc.edu/ ∼ mjn eely) are with the Dept. of Elect ri- cal Eng., Uni versity of Southern Califor nia, Los Angele s, CA 90089, USA. This material is supported in p art by one or more of the follo wing: the D ARP A IT -MANET program grant W911NF-07-0028, the NSF grant OCE 0520324, the NSF Career grant CCF-0747525. time slot, based on the observed network state, and subject to the constraint that the network queu es must ha ve enough contents to support the action . The ch osen action g enerates some u tility , but also consumes so me am ount o f co ntents f rom some qu eues, and possibly generate s new co ntents for so me other qu eues. These con tents cause congestio n, and thus lead to backlogs at qu eues in the network. The g oal of the con troller is to maximize its time average u tility subject to the constraint that the time average to tal backlog in th e network is finite. Many of the utility m aximization pr oblems in data n etworks fall into this general framework. For instan ce, [10], [ 11], [12] [13], [14], can be viewed as spe cial cases of the above framework which allow scheduling actions to be indepe ndent of the con tent le vel in the queue s (see [15] for a sur vey of problem s in data n etworks). By comparin g the pr ocessing networks with the data networks, we note th at the main difficulty in performin g utility op timal sch eduling in these processing networks is that we nee d to build an optimal scheduling algorithm on to p of a mechanism tha t pr events queue un derflows . Such schedulin g problem s with und erflow constraints are usually formulated as dynamic program s, e.g., [16], wh ich req uire substantial statistical knowledge of the network random ness, and are u sually very difficult to solve. In this paper, we d ev elop th e Perturb ed Max -W eigh t a l- gorithm (PMW) for ach ieving optimal utility in p rocessing networks. PMW is a greedy algo rithm tha t makes decisions ev ery tim e slot, without r equiring any statistical kno wledge of the network random ness . PMW is based on the Max-W eight algorithm developed in the data ne twork context [17] [18]. There, Max-W eight has been shown to b e ab le to ac hieve a time average u tility that is within O (1 / V ) of the optimal network utility for any V ≥ 1 , while ensurin g th at the average network delay is O ( V ) , when the network dyn amics are i.i.d. [18]. Th e idea of PMW is to perturb the weigh ts u sed in the Max-W e ight algorithm so as to “pu sh” the queue sizes towards some no nzero values. Doing so pr operly , we can en sure that the queues always ha ve enou gh contents for the schedu ling actions. Once this is accom plished, we then do schedu ling a s in the usual Max-W eigh t algorithm with the per turbed weights. In this way , we simultaneo usly av oid qu eue underflows and achieve go od utility perfo rmance, and also elimin ate the need to solve complex dynamic prog rams. The PM W algorithm is quite dif ferent fr om the ap proache s used in th e processing network literatu re. [1] analyzes ma nu- facturing n etworks usin g Brownian ap proxim ations. [2] applies the Max-W eigh t algorithm to do sched uling in m anufacturing networks, assumin g all the que ues a lw ays have eno ugh con - tents. [3] d ev elops th e Deficit Max -W eigh t algor ithm ( DMW), by u sing Max-W e ight based o n an altern ativ e contr ol pro - cess fo r decision makin g. [4] fo rmulates the p roblem as a conv ex optimizatio n p roblem to match the in put and o utput 2 rates of the qu eues, without considering the queu eing level dynamics. PMW instead provides a way to exp licitly av oid queue un derflows, and allow us to compu te explicit back log bound s. Ou r algorithm is per haps mo st similar to the DMW algorithm in [ 3]. DM W ach ie ves the d esired perfo rmance by bound ing th e “deficit” incu rred by th e alg orithm and app lies to bo th stability and utility max imization problem s. Whereas PMW uses per turbation s to av oid d eficits en tirely and allows for more ge neral time varying system dynam ics, e.g., random arriv als and rando m costs. The p aper is organ ized as fo llows: In Section II we set u p our notations. In Section III, we p resent a study on a data fusion example to demo nstrate the main id ea of th e p aper . In Section IV we state the ge neral network mode l and the scheduling pro blem. In Section V we character ize op timality , and in Sections VI we dev elop the PMW algorith m and sh ow its utility can a pproach the optimu m. Section VI I c onstructs a PM W algorithm for a mo re specific yet general network. Simulation results are presen ted in Section VIII. I I . N O TA T I O N S Here we first set up the notation s used in this paper: R represents the set of real number s. R + (or R − ) denotes the set o f nonnegative ( or n on-po siti ve) real n umbers. R n (or R n + ) is the set of n d imensional column vectors, w ith each element being in R (o r R + ). Bold symbo ls a and a T represent a column vector an d its transpose. a  b means vector a is entrywise no less th an vector b . || a − b || is th e Euc lidean distance o f a and b . 0 and 1 d enote colu mn vectors with all elements bein g 0 and 1 . For any two vectors a = ( a 1 , ..., a n ) T and b = ( b 1 , ..., b n ) T , th e vector a ⊗ b = ( a 1 b 1 , ..., a n b n ) T . Finally [ a ] + = max[ a, 0] . I I I . A DAT A P RO C E S S I N G E X A M P L E In this sectio n, we study a d ata f usion example and d ev elop the Per turbed Max- W eight a lgorithm (PMW) in th is c ase. Th is example dem onstrates the main idea of this paper . W e will later present our general mode l in Section IV. A. Network S ettings W e consider a network shown in Fig. 1, where the network perfor ms a 2 -stage da ta proc essing for the data en tering into the network . q 1 P 1 P 2 q 3 q 2 Output R 1 (t) R 2 (t) ! 1 (t) ! 3 (t) ! 2 (t) Fig. 1. An example network consisti ng of three queues q 1 , q 2 , q 3 and two processors P 1 , P 2 . In this network, there are two rand om data streams R 1 ( t ) , R 2 ( t ) , which r epresent, e. g., sensed data th at com e into sensors, or vid eo and voice d ata that ne ed to be m ixed. W e assume that R i ( t ) = 1 or 0 , equa lly likely , fo r i = 1 , 2 . At ev ery time slo t, the network c ontroller first decides whether or n ot to adm it the ne w arriv als, given that accepting any one new arriv al u nit incu rs a c ost of 1 . Th e contr oller then h as to decide h ow to activate the two proce ssors P 1 , P 2 for d ata processing. W e assume that b oth proc essors ca n be activated simultaneou sly . When activ ated, P 1 consumes on e u nit of data from both q 1 and q 2 , and generates one un it of fused data into q 3 . This data n eeds furthe r pr ocessing that is do ne by P 2 . Wh en P 2 is activated, it consum es one un it of data fro m q 3 , and generates one unit of processed data. W e assum e that each u nit of successfu lly fused an d processed d ata gene rates a p rofit o f p ( t ) , wher e p ( t ) is i.i.d. and takes value 3 or 1 with equal probabilities. Th e network con troller’ s objecti ve is to maximize the average utility , i.e., profit minu s cost, sub ject to queue stability . For the ease o f presenting the g eneral mod el later , we define a network state S ( t ) = ( R 1 ( t ) , R 2 ( t )) , 1 which d escribes the current n etwork random ness. W e also d enote the contr oller’ s action at time t to be x ( t ) = ( D 1 ( t ) , D 2 ( t ) , I 1 ( t ) , I 2 ( t )) , where D j ( t ) = 1 ( D j ( t ) = 0 ) means to adm it (reject) the new arr i vals into q ueue j , an d I i ( t ) = 1 ( I i ( t ) = 0 ) mean s processor P i is activated (tu rned off). W e no te the f ollowing no-un derflow co nstraints must be met for all time when we activ ate pr ocessors P 1 , P 2 : I 1 ( t ) ≤ q 1 ( t ) , I 1 ( t ) ≤ q 2 ( t ) , I 2 ( t ) ≤ q 3 ( t ) . (1) That is, I 1 ( t ) = 1 only when q 1 and q 2 are both no nempty , and I 2 ( t ) = 1 only if q 3 is non empty . Note that [3] is th e first to identify such no-un derflow constrain ts and propo se explicit solution to the qu eue u nderflow prob lems f or the con text of a processing network. Sub ject to (1), we ca n th en write th e amount o f arrivals into q 1 , q 2 , q 3 , a nd th e service rates of the queues a t time t as fu nctions of the network state S ( t ) and the action x ( t ) , i.e., A j ( t ) = A j ( S ( t ) , x ( t )) = D j ( t ) R j ( t ) , j = 1 , 2 , A 3 ( t ) = A 3 ( S ( t ) , x ( t )) = I 1 ( t ) . (2) µ j ( t ) = µ j ( S ( t ) , x ( t )) = I 1 ( t ) , j = 1 , 2 , µ 3 ( t ) = µ 3 ( S ( t ) , x ( t )) = I 2 ( t ) . (3) Then we see that the queues ev o lve according to the following: q j ( t + 1 ) = q j ( t ) − µ j ( t ) + A j ( t ) , j = 1 , 2 , 3 , ∀ t. ( 4) The instanta neous utility is given by: f ( t ) = f ( S ( t ) , x ( t )) = p ( t ) I 2 ( t ) − D 1 ( t ) R 1 ( t ) − D 2 ( t ) R 2 ( t ) . (5) The go al is to max imize the time average value of f ( t ) subjec t to network stability . Note that the constraint (1) g reatly complicates the design of an o ptimal schedu ling algorithm . This is because th e dec ision made at time t may affect the q ueue states in future time slots, which can in turn af f ect th e set of possible actions in the future. 1 The network state here contai ns just R 1 ( t ) and R 2 ( t ) . More complica ted setting s, where the amount consumed from queues may also depe nd on the random link condit ions between queues and processors can also be modeled by incorporat ing the link components into the network state , e.g., [19]. 3 In the following, we will develop the Pertu rbed Max- W eight algorithm ( PMW) for th is examp le. The idea of PMW is use the usual Max- W eight algor ithm, but to p erturb the weigh ts so as to push the queue sizes tow ards certain non zero values. By carefully designin g the perturba tion, we can simultane ously ensure tha t the qu eues always have enough data fo r proc essing and the ach ie ved utility is close to optimal. B. The P ertu rbed Max-W eigh t algorithm (PMW) W e now present the constru ction of the PMW algo rithm for th is simple example (this is extended to general network models in Section VI). T o start, we first define a perturbatio n vector θ = ( θ 1 , θ 2 , θ 3 ) T and th e L y apunov fun ction L ( t ) = 1 2 P 3 j =1 [ q j ( t ) − θ j ] 2 . W e th en define the one-slo t conditiona l drift as: ∆( t ) = E  L ( t + 1) − L ( t ) | q ( t )  , (6) where the expec tation is taken over the random n etwork state S ( t ) and the ran domness over the actions. Using the que ueing dynamics (4), it is ea sy to ob tain that: ∆( t ) ≤ B − 3 X j =1 E  ( q j ( t ) − θ j )[ µ j ( t ) − A j ( t )] | q ( t )  , where B = 3 . Now we use the “dr ift-plus-pen alty” approach in [1 8] to design our algorithm for this problem. T o do so, we define a co ntrol parameter V ≥ 1 , which will affect our utility-backlog tradeoff, and add to b oth side s the ter m − V E  f ( t ) | q ( t )  to get: ∆( t ) − V E  f ( t ) | q ( t )  (7) ≤ B − V E  f ( t ) | q ( t )  − 3 X j =1 E  ( q j ( t ) − θ j )[ µ j ( t ) − A j ( t )] | q ( t )  . Denote ∆ V ( t ) = ∆( t ) − V E  f ( t ) | q ( t )  , a nd plu g (2), (3) and (5) in to the ab ove, to g et: ∆ V ( t ) ≤ B + E  D 1 ( t ) R 1 ( t )[ q 1 ( t ) − θ 1 + V ] | q ( t )  (8) + E  D 2 ( t ) R 2 ( t )[ q 2 ( t ) − θ 2 + V ] | q ( t )  − E  I 2 ( t )[ q 3 ( t ) − θ 3 + p ( t ) V ] | q ( t )  − E  I 1 ( t )[ q 1 ( t ) − θ 1 + q 2 ( t ) − θ 2 − ( q 3 ( t ) − θ 3 )] | q ( t )  . W e now develop ou r PMW algorith m by choo sing an actio n at every time slot to minimize the right-ha nd side ( RHS) of (8) subject to (1). The algorith m th en works as f ollows: PMW : At ev ery time slot, observe S ( t ) and q ( t ) , and do the fo llowing: 1) Data Adm ission: Choose D j ( t ) = 1 , i.e., ad mit the new arriv als to q j if: q j ( t ) − θ j + V < 0 , j = 1 , 2 , (9) else set D j ( t ) = 0 and reject the ar riv als. 2) Processor Activ ation: Choose I 1 ( t ) = 1 , i.e., ac ti vate processor P 1 , if q 1 ( t ) ≥ 1 , q 2 ( t ) ≥ 1 , and that: q 1 ( t ) − θ 1 + q 2 ( t ) − θ 2 − ( q 3 ( t ) − θ 3 ) > 0 , (10) else choose I 1 ( t ) = 0 . Similarly , choose I 2 ( t ) = 1 , i.e., activ ate pr ocessor P 2 , if q 3 ( t ) ≥ 1 , and that: q 3 ( t ) − θ 3 + p ( t ) V > 0 , (11) else choo se I 2 ( t ) = 0 . 3) Queueing up date: Update q j ( t ) , ∀ j , acco rding to (4). C. P e rformance of PMW Here we analyze the performa nce of PMW . W e will first prove the following imp ortant claim: under a pr oper θ vector , PMW min imizes th e R HS of (8) over all po ssible policies of arrival ad mission and pr oc essor activation , including those that choose ac tions r egar d less o f the co nstraint (1) . W e then u se this claim to p rove the perf ormance of PMW , by comparin g the value of the RHS of ( 8) und er PMW versus that under an alternate po licy . T o prove the claim, we first see that the policy th at minimizes the RHS o f (8) witho ut the con straint (1) differs from PMW only in the pr ocessor a cti vation p art, where PMW also considers the co nstraints q 1 ( t ) ≥ 1 , q 2 ( t ) ≥ 1 and q 3 ( t ) ≥ 1 . Thu s if one can show that these c onstraints are indeed red undan t in th e PMW algorith m under a p roper θ vector , i.e. , one can acti vate the processors without considering them b ut still en sure th em, th en PMW minimizes the RHS of (8) over all p ossible po licies. In the following, we will use the following θ j values: θ 1 = 2 V , θ 2 = 2 V , θ 3 = 3 V . (12) Let us now look at the qu eue sizes q j ( t ) , j = 1 , 2 , 3 . From (11), we see that P 2 is activ ate d if and only if: q 3 ( t ) ≥ θ 3 − p ( t ) V + 1 , and q 3 ( t ) ≥ 1 . (13) Hence I 2 ( t ) = 1 whenever q 3 ( t ) ≥ θ 3 − V + 1 , but I 2 ( t ) = 0 unless q 3 ( t ) ≥ θ 3 − 3 V + 1 . Since q 3 can receive and d eliv er at most on e unit of data at a time, we g et: θ 3 − V + 1 ≥ q 3 ( t ) ≥ θ 3 − 3 V , ∀ t. (14) Using θ 3 = 3 V , this implies: 2 V + 1 ≥ q 3 ( t ) ≥ 0 , ∀ t. (15) This shows that with θ 3 = 3 V , the activ ation s of P 2 are always feasible even if we do n ot consider th e constraint q 3 ( t ) ≥ 1 . W e now look at q 1 ( t ) and q 2 ( t ) . W e see from ( 9) that fo r θ 1 , θ 2 ≥ V , we have: q j ( t ) ≤ θ j − V , j = 1 , 2 . (16) Also, using (10) and (1 4), it is easy to see that when I 1 ( t ) = 1 , i.e., when P 1 is turned on , we have: q 1 ( t ) − θ 1 + q 2 ( t ) − θ 2 > q 3 ( t ) − θ 3 ≥ − 3 V . (17) Combining (17) with (16), we see that if I 1 ( t ) = 1 , we hav e: q j ( t ) ≥ 1 , j = 1 , 2 . (18) This is so because, e.g., if q 1 ( t ) = 0 , then q 1 ( t ) − θ 1 = − θ 1 = − 2 V . Since q 2 ( t ) − θ 2 ≤ − V by ( 16), we thu s hav e: q 1 ( t ) − θ 1 + q 2 ( t ) − θ 2 ≤ − 2 V − V = − 3 V , 4 which cannot be greater than − 3 V in (17). Thu s by (15) and (18), we h av e: q j ( t ) ≥ 0 , j = 1 , 2 , 3 , ∀ t. (19) This sh ows that b y using the θ j values in (12), PMW auto - matically ensur es th at no q ueue un derflow happens, and hence PMW minimizes the RHS of (8) over all possible po licies. Giv en th e above observation, the utility per forman ce o f PMW can now be an alyzed as the usual Max-W eigh t algo- rithm. Spec ifically , using a sim ilar argum ent as in [5], we can compare th e drift under PMW with a stationary ran domized algorithm which cho oses scheduling actions pu rely as a func- tion of S ( t ) , and achieves E  µ j ( t ) − A j ( t ) | q ( t )  = 0 for all j an d E  f ( t ) | q ( t )  = f ∗ av = 1 2 , where f ∗ av is the optimal av erage utility . Note th at this co mparison will not h ave been possible here witho ut using the perturba tion to e nsur e (19) . Now p lugging this p olicy into (7), we o btain: ∆( t ) − V E  f ( t ) | q ( t )  ≤ B − V f ∗ av . (20) T ak ing expectatio ns over q ( t ) on bo th sides an d summ ing it over t = 0 , 1 , ..., T − 1 , we ge t: E  L ( T ) − L (0)  − V T − 1 X t =0 E  f ( t )  ≤ T B − V T f ∗ av . (21) Now rear ranging the terms, dividing both sides by V T , a nd using the fact tha t L ( t ) ≥ 0 , we g et: 1 T T − 1 X t =0 E  f ( t )  ≥ f ∗ av − B V − E  L (0)  T V . (22) T ak ing a liminf as T → ∞ , an d using E  L (0)  < ∞ , we get: f P M W av = lim inf T →∞ 1 T T − 1 X t =0 E  f ( t )  ≥ f ∗ av − B V , (23) where f P M W av denotes the time average utility a chieved by PMW . Th is thu s shows that PMW is able to achieve a time av erage utility that is within O (1 /V ) o f the optimal value, and gua rantees q j ( t ) ≤ O ( V ) fo r all time. Note that PMW is similar to the D MW algo rithm developed in [3] . Howe ver , DMW allows the queues to be empty when activating p roces- sors, which may lead to “deficit, ” wher eas PMW effecti vely av oid s this by using a pertu rbation vector . In the fo llowing, we will pre sent the gene ral proce ssing net- work utility op timization model, and analyze the p erforma nce of th e general PMW algorith m unde r this general m odel. Our analysis uses a duality argument, an d will be different from that in [5]. As we will see, o ur app roach allows on e to an alyze the alg orithm p erform ance without proving the existence of an optimal stationar y and rando mized algorithm. I V . G E N E R A L S Y S T E M M O D E L In this section, we present the general network model. W e consider a network controller that operates a general network with the goal of ma ximizing the time average utility , subject to the network stability . The network is assumed to operate in slotted time, i.e., t ∈ { 0 , 1 , 2 , ... } . W e assume there are r ≥ 1 queues in the network. A. Network S tate In every slot t , we use S ( t ) to d enote the curren t network state, which ind icates the c urrent network param eters, such as a vector of ch annel co nditions fo r each link, or a collection of o ther re le vant inform ation abou t the curren t network link s and a rriv als. W e assume that S ( t ) is i.i.d. every time slot, with a total of M d ifferent ra ndom network states denoted by S = { s 1 , s 2 , . . . , s M } . 2 W e let π s i = P r { S ( t ) = s i } . Th e network controller can observe S ( t ) at the beginning of every slot t , but the π s i probab ilities ar e not n ecessarily known. B. The Utility , T raffic, and Service At ea ch time t , after obser ving S ( t ) = s i and th e n etwork backlog vector, the controller will p erform an a ction x ( t ) . This action rep resents the aggregate d ecisions made by the controller at time t , which can include, e.g., in the previous example, the set o f p rocessors to turn on, or the amo unt of arriving contents to accept, or b oth, etc. W e d enote X ( s i ) the set of all feasib le actions for network state s i , a ssuming all th e queu es co ntain eno ugh contents to meet th e scheduling requir emen ts . Note th at we always h av e x ( t ) = x ( s i ) for some x ( s i ) ∈ X ( s i ) whenever S ( t ) = s i . The set X ( s i ) is assumed to be time-inv ariant and compact for all s i ∈ S . I f the ch osen action x ( t ) = x ( s i ) at time t can be perfor med, i.e., it is feasible an d all the qu eues have enoug h contents, then the utility , tr affic, and service generated by x ( t ) are as follows: (a) The cho sen action has an associated utility g i ven by the utility functio n f ( t ) = f ( s i , x ( s i ) ) : X ( s i ) 7→ R ; (b) The amount of conten ts genera ted by the action to queue j is determined by th e tr affic function A j ( t ) = A j ( s i , x ( s i ) ) : X ( s i ) 7→ R + , in un its of con tents; (c) The amou nt o f con tents co nsumed from qu eue j by the action is given by the rate fun ction µ j ( t ) = µ j ( s i , x ( s i ) ) : X ( s i ) 7→ R + , in units of contents; Note that A j ( t ) includes both th e exogenou s arri vals from out- side the network to queue j , a nd th e end ogenou s arriv als f rom other queu es, i.e., the n ewly g enerated conten ts by pro cessing contents in som e o ther queues, to q ueue j . W e assum e th e function s f ( s i , · ) , µ j ( s i , · ) and A j ( s i , · ) are continuo us, time- in variant, their m agnitude s are unifor mly upper b ounded by some c onstant δ max ∈ (0 , ∞ ) for all s i , j , and th ey are k nown to the n etwork operator . In any ac tual algo rithm implem entation, howev er , we see that not all actions in the set X ( s i ) can be p erform ed when S ( t ) = s i , due to the fact that some qu eues may not ha ve enoug h co ntents for the action. W e say th at an action x ( s i ) ∈ X ( s i ) is feasible at time t with S ( t ) = s i only when the following general no-und erflow co nstraint is satisfied: q j ( t ) ≥ µ j ( s i , x ( s i ) ) , ∀ j. (24) That is, all the q ueues must have c ontents greater tha n or equal to what will be consumed . In the following, we assum e there exists a set o f ac tions { x ( s i ) k } k =1 , 2 ,... , r +2 i =1 ,...,M with x ( s i ) k ∈ 2 Note that all our results can easily be exte nded to the case when S ( t ) e volv es according to a finite state aperiod ic and irred ucible Marko v chain , by using the results de velop ed in [20]. 5 X ( s i ) and so me variables ϑ ( s i ) k ≥ 0 for all s i and k with P r +2 k =1 ϑ ( s i ) k = 1 f or all s i , such th at: X s i π s i  r +2 X k =1 ϑ ( s i ) k [ A j ( s i , x ( s i ) k ) − µ j ( s i , x ( s i ) k )]  ≤ − η , (25 ) for some η > 0 for all j . T hat is, the “stability constraints” are feasible with η -slackne ss. 3 In the f ollowing, we use: A ( t ) = ( A 1 ( t ) , ..., A r ( t )) T , µ ( t ) = ( µ 1 ( t ) , ..., µ r ( t )) T , (26) to den ote the arrival and service vecto rs at time t . C. Qu eueing, A v erag e Cost, a nd the Objec tive Let q ( t ) = ( q 1 ( t ) , ..., q r ( t )) T ∈ R r + , t = 0 , 1 , 2 , ... be the qu eue backlo g vector pr ocess of the n etwork, in un its of contents. Due to the feasibility co ndition (24) of the actions, we see that the queu es ev o lve accordin g to th e f ollowing dynamics: q j ( t + 1 ) = q j ( t ) − µ j ( t ) + A j ( t ) , ∀ j, t ≥ 0 , (27) with some || q (0) || < ∞ . No te that using a non zero q j (0) can be v iewed as placing a n “initial stock” in the que ues to facilitate algorithm imp lementation. In th is paper, we a dopt the fo llowing notion o f queue stability: q , lim sup t →∞ 1 t t − 1 X τ =0 r X j =1 E  q j ( τ )  < ∞ . (28) W e also use f Π av to de note the tim e average u tility ind uced by an action-ch oosing policy Π , defined as: f Π av , lim inf t →∞ 1 t t − 1 X τ =0 E  f Π ( τ )  , (29) where f Π ( τ ) is the utility incurr ed at time τ b y policy Π . W e call a n action -choo sing policy feasible if at every time slot t it only chooses action s f rom the feasib le action set X ( S ( t )) that satisfy (24). W e then call a f easible action-ch oosing policy under which (28) h olds a stable p olicy , an d u se f ∗ av to de note the op timal time average utility over all stable policies. In every slot, the n etwork contro ller observes the curren t network state and the queue backlog vector , and choo ses a feasible contro l action that ensu res (24), with the ob jectiv e of max imizing the time average utility subject to network stability . Note that if con dition (24) can b e ig nored , and if any processor only req uires conten ts from a single q ueue, th en this problem falls into the general stochastic network optimiza tion framework considered in [18], in which case it can be solved by using th e usual Max -W eigh t algorithm to achieve a utility that is within O (1 /V ) of the op timal while ensurin g that the av erage network back log is O ( V ) . 3 The use of r + 2 actions here is due to the use of Caratheodory’ s theore m [21] in the proof of Theorem 1. V . U P P E R B O U N D I N G T H E O P T I M A L U T I L I T Y In this section, we first ob tain an upper bound of the optimal utility that th e n etwork controller can achieve. Th is up per bound will later be used to analy ze the perf ormance o f our algorithm . Th e result is summarized in the following theor em. Theorem 1: Suppose the initial queue backlog q ( t ) satisfies E  q j (0)  < ∞ for all j = 1 , ..., r . Th en we have: V f ∗ av ≤ φ ∗ , (30) where φ ∗ is the o ptimal value of the fo llowing pro blem: max : φ = X s i π s i V r +2 X k =1 a ( s i ) k f ( s i , x ( s i ) k ) (31) s.t. X s i π s i r +2 X k =1 a ( s i ) k A j ( s i , x ( s i ) k ) (32) = X s i π s i r +2 X k =1 a ( s i ) k µ j ( s i , x ( s i ) k ) x ( s i ) k ∈ X ( s i ) , ∀ s i , k (33) a ( s i ) k ≥ 0 , ∀ s i , k , X k a ( s i ) k = 1 , ∀ s i . (34) Pr oof: See Appen dix A. Note that the p roblem ( 31) only r equires th at the time a verage input rate in to a q ueue is e qual to its time average o utput rate. This requir ement igno res the a ction feasibility constraint (24), and m akes (31) easier to solve than th e sch eduling prob lem. W e now look at the dua l pr oblem of the p roblem (3 1). T he following lemma shows that the d ual p roblem of (3 1) do es no t have to include the v ariables { a ( s i ) k } k =1 ,..., r + 2 i =1 ,...,M . This lemma will also b e useful for o ur later analy sis. Lemma 1: T he dual pr oblem of (31) is given by : min : g ( γ ) , s.t. γ ∈ R r , (35) where the fu nction g ( γ ) is d efined: g ( γ ) = sup x ( s i ) ∈X ( s i ) X s i π s i  V f ( s i , x ( s i ) ) (36) − X j γ j  A j ( s i , x ( s i ) ) − µ j ( s i , x ( s i ) )   . Moreover , let γ ∗ be any optimal solu tion of (35), we h av e: g ( γ ∗ ) ≥ φ ∗ . (37) Pr oof: (Lem ma 1) It is easy to see fr om (31) th at the d ual function is given by: ˆ g ( γ ) = sup x ( s i ) k ,a ( s i ) k X s i π s i  r +2 X k =1 a ( s i ) k V f ( s i , x ( s i ) k ) (38) − X j γ j r +2 X k =1 a ( s i ) k  A j ( s i , x ( s i ) k ) − µ j ( s i , x ( s i ) k )   . Due to the use o f the { a ( s i ) k } k =1 ,..., r + 2 i =1 ,...,M variables, it is easy to see that ˆ g ( γ ) ≥ g ( γ ) . Howe ver, if { x ( s i ) } M i =1 is a set of maximizers of g ( γ ) , then the set of variables { x ( s i ) k , a ( s i ) k } k =1 ,..., r + 2 i =1 ,...,M where for each s i , x ( s i ) k = x ( s i ) for 6 all k , an d a ( s i ) 1 = 1 with a ( s i ) k = 0 for all k ≥ 2 , will also be maximizer s of ˆ g ( γ ) . Thus g ( γ ) ≥ ˆ g ( γ ) . This shows that g ( γ ) = ˆ g ( γ ) , an d he nce g ( γ ) is the d ual fu nction of (31). (37) follows from weak du ality [21]. In the following, it is useful to define the f ollowing fun ction: g s i ( γ ) = sup x ( s i ) ∈X ( s i )  V f ( s i , x ( s i ) ) (39) − X j γ j  A j ( s i , x ( s i ) ) − µ j ( s i , x ( s i ) )   . That is, g s i ( γ ) is the du al fu nction of ( 31) when there is a single network state s i . W e can see fro m (36) and (3 9) that: g ( γ ) = X s i π s i g s i ( γ ) . (40) In th e following, we w ill u se γ ∗ = ( γ ∗ 1 , ..., γ ∗ r ) T to de note an optimal solution of the pr oblem (35). V I . T H E P E RT U R B E D M A X - W E I G H T A L G O R I T H M A N D I T S P E R F O R M A N C E In this section, we d ev elop the general Perturbe d M ax- W eight algorithm (PMW) to solve o ur schedu ling pro blem. T o start, we first ch oose a perturbatio n vector θ = ( θ 1 , ..., θ r ) T . Then we define the following weighted per turbed L y apunov function with some positive con stants { w j } r j =1 : L ( t ) = 1 2 r X j =1 w j  q j ( t ) − θ j  2 . (41) W e then define the one-slot cond itional drift as in (7), i.e., ∆( t ) = E  L ( t + 1) − L ( t ) | q ( t )  . W e will sim ilarly use the “drift-plus-p enalty” appro ach in Section III to constru ct the alg orithm. Specifically , we first use the qu eueing dyn amic equation (27), an d have th e following lemma: Lemma 2: U nder any feasible co ntrol policy th at ca n be implemented at time t , we have: ∆( t ) − V E  f ( t ) | q ( t )  ≤ B − V E  f ( t ) | q ( t )  (42) − r X j =1 w j  q j ( t ) − θ j  E  [ µ j ( t ) − A j ( t )] | q ( t )  , where B = δ 2 max P r j =1 w j . Pr oof: See Appen dix B. The gen eral Perturbed M ax-W eig ht a lgorithm (PMW) is then o btained b y cho osing an action x ( t ) fr om X ( S ( t )) at tim e t to minimize the righ t-hand side (RHS) o f (42) subject to (2 4). Specifically , define the function D ( s i ) θ , q ( t ) ( x ) as: D ( s i ) θ , q ( t ) ( x ) (43) , V f ( s i , x ) + r X j =1 w j  q j ( t ) − θ j  µ j ( s i , x ) − A j ( s i , x )  . W e see that th e function D ( s i ) θ , q ( t ) ( x ) is ind eed the term in side the conditio nal expe ctation on the RHS of (42). W e now also d efine D ( s i ) ∗ θ , q ( t ) to b e th e optim al value of the fo llowing problem : max : D ( s i ) θ , q ( t ) ( x ) , s.t., x ( s i ) ∈ X ( s i ) . (44) Hence D ( s i ) ∗ θ , q ( t ) is the maximu m value o f D ( s i ) θ , q ( t ) over all possible policies, including those that may not consider t he no- underflow constraint (24). Th e gen eral Perturbed Max- W eight algorithm (PMW) then work s as fo llows: PMW : Initialize the perturb ation vector θ . At every time slot t , observe the curre nt network state S ( t ) and the back log q ( t ) . If S ( t ) = s i , c hoose x ( s i ) ∈ X ( s i ) subject to (24) th at makes the value of D ( s i ) θ , q ( t ) ( x ) close to D ( s i ) ∗ θ , q ( t ) . Note that dependin g on the pro blem stru cture, the PMW algorithm can usu ally be implemen ted easily , e.g., [5], [11]. Now we analy ze the perform ance o f th e PMW algorithm. W e will prove our result und er the following condition: Condition 1: T here exists some finite con stant C ≥ 0 , such that at every time slot t w ith a n etwork state S ( t ) , the value of D ( S ( t )) θ , q ( t ) ( x ) und er PMW is at least D ( S ( t )) ∗ θ , q ( t ) − C . The immediate consequ ence of Condition 1 is that PMW also minimize s the RHS of (4 2), i.e., the co nditional expec- tation, to within C of its minimum value over all possible policies. If C = 0 , then PMW simultaneo usly ensures (24) and minimizes the RHS of (4 2), e.g., as in the example in Section III. Howe ver , we note tha t Con dition 1 d oes not require the value of D ( S ( t )) θ , q ( t ) ( x ) to b e exactly the same as D ( S ( t )) ∗ θ , q ( t ) . This allows for more flexibility in c onstructing the PMW algorithm (See Section VII for an example) . W e also n ote that Condition 1 can be ensured , e.g., by carefully cho osing the θ j values to ensur e q j ( t ) ≥ δ max for all time [5]. W e w ill show tha t, under Conditio n 1, PMW achieves a time average utility that is within O (1 /V ) of f ∗ av , while gu aranteeing that the tim e av erage network queu e size is O ( V ) + P j w j θ j , which is O ( V ) if θ = Θ ( V ) and w j = O (1) , ∀ j . Th e following theorem summar izes PMW’ s perfor mance results. Theorem 2: Suppose that (2 5) holds, that Condition 1 holds, and th at E  q j (0)  < ∞ for all j = 1 , ..., r . Then under PMW , we have: 4 f P M W av ≥ f ∗ av − B + C V , (45) q P M W ≤ B + C + 2 V δ max η + r X j =1 w j θ j . (46) Here B = δ 2 max P r j =1 w j , η is th e slackness param eter in Section I V -B, f P M W av is de fined in (29) to b e th e time average expected utility of PMW , and q P M W is the time average expected weighted network backlog und er PMW , de fined: q P M W , lim sup t →∞ 1 t t − 1 X τ =0 r X j =1 w j E  q j ( τ )  . Pr oof: See Appen dix C. Theorem 2 shows tha t if Condition 1 hold s, then PMW can be used as in p revious n etworking pro blems, e.g., [11], [12], to obtain explicit utility-b acklog tradeoffs. W e n ote that a condi- tion similar to Cond ition 1 was assumed in [2]. Howe ver, [2] only considers the u sual Max-W eight alg orithm, under which case ( 24) may no t b e satisfied for all time. Whereas PMW resolves this problem by carefully ch oosing the pertu rbation vector . One such example of PMW is the recent work [5], 4 Easy to see that (46) ensures (28), hence the network is stable under PMW . 7 which ap plies PMW to an assembly lin e sch eduling p roblem and achieves an [ O (1 /V ) , O ( V )] u tility-backlo g trad eoff. V I I . C O N S T RU C T I N G P M W F O R N E T W O R K S W I T H O U T P U T R E W A R D In th is sectio n, we look at a specific y et gen eral pro cessing network mod el, and explicitly con struct a PMW alg orithm, including finding the proper θ vector and choo sing actio ns at each time slot. A. Network Mod el W e assume that the network is modeled b y an acyclic directed gr aph G = ( Q , P , L ) . Here Q = Q s ∪ Q in is the set o f queues, consisting o f the set o f sour ce que ues Q s where arrivals enter the network, and the set of internal queues Q in where c ontents ar e sto red fo r fu rther pro cessing. P = P in ∪ P o is the set of pro cessors, consisting of a set of internal pro cessors P in , which g enerate pa rtially processed contents for furth er processing at othe r processors, and outp ut processors P o , which generate fully p rocessed con tents and deliver them to the ou tput. L is the set of directed link s that connects Q an d P . Note that a link only exists between a queue in Q and a processor in P . W e d enote N in p = |P in | , N o p = |P o | and N p = N in p + N o p . W e also denote N s q = |Q s | , N in q = |Q in | and N q = N s q + N in q . Each pro cessor P n , when activ ated, consumes a certain amount o f c ontents fr om a set o f supp ly queu es, den oted by Q S n , and g enerates some am ount of n ew contents. These new contents either go to a set of demand q ueues, denoted by Q D n , if P n ∈ P in , o r are de li vered to the o utput if P n ∈ P o . For any queue q j ∈ Q , we use P S j to denote the set of processors that q j serves as a sup ply q ueue, and u se P D j to deno te the set of p rocessors th at q j serves as a deman d queu e. An examp le of such a n etwork is shown in Fig. 2. In the fo llowing, we assume that for each p rocessor P i ∈ P in , | Q D i | = 1 , i.e., each processor only generates contents fo r a sing le demand queue. W e use β nj to den ote the amoun t pro cessor P n consumes from a queue q j in Q S n when it is acti vated. For each P i ∈ P in , we also use α ih to d enote the amount P i generates into the queue q h if q h = Q D i , wh en it is activated. For a pro cessor P k ∈ P o , we use α ko to denote the amo unt of o utput generated by it when it is turn ed on . 5 W e d enote β max = max i,j β ij , β min = min i,j β ij and α max = max i,j, [ α ij , α io ] . W e assum e that β min , β max , α max > 0 . W e also d efine M p to be the maximum numbe r of supply queues that any p rocessor can have, define M d q to be the maximu m n umber o f pr ocessors that any qu eue can serve as a dem and queue, an d defin e M s q to be the maximum numbe r of pro cessors tha t any q ueue can serve as a supply qu eue. W e use R j ( t ) to d enote the amoun t of co ntents arriving to a sou rce queu e q j ∈ Q s at time t . W e assume R j ( t ) is i.i.d . ev ery slot, and that R j ( t ) ≤ R max for all q j ∈ Q s and all t . W e assume th at there are no exog enous arriv als into the qu eues in Q in . 5 Note that here we only conside r binary actions of processors. Our results can also be generaliz ed into the case when there are multiple operation le vels under which dif ferent amount of content s will be consumed and generate d. R 1 R 2 R 4 R 3 Output 1 q 1 q 2 q 3 q 4 q 5 q 6 P 2 P 1 P 3 P 4 P 5 Output 2 Fig. 2. A genera l processing network. W e assume that in every slot t , admitting any unit amoun t of R j ( t ) arriv al incu rs a cost of c j ( t ) , and th at acti vating any internal pro cessor P i ∈ P in incurs a cost o f C i ( t ) , whe reas activ ating any output processor P k ∈ P o generates a profit of p k ( t ) per unit ou tput conte nt. 6 W e assume c j ( t ) , C i ( t ) , p k ( t ) are all i.i.d . every time slot. In th e following, we also assum e that p min ≤ p k ( t ) ≤ p max , and that c min ≤ c j ( t ) ≤ c max and C min ≤ C i ( t ) ≤ C max for all k , j, i and fo r all time. Below , we use I n ( t ) = 1 to denote the a cti vation decision of P n , i. e., I n ( t ) = 1 ( I n ( t ) = 0 ) mean s that P n is activated (turned off). W e also use D j ( t ) ∈ [0 , 1] to denote the portion of arriv als fr om R j ( t ) that are a dmitted into q j . W e assume there exist some gen eral constraint on h ow the pro cessors can be ac ti vated, which can b e due to, e.g., r esource sharin g among p rocessors. W e m odel this con straint by defining an activ ation vector I ( t ) = ( I 1 ( t ) , ..., I N p ( t )) , and th en assume that I ( t ) ∈ I for all tim e, wher e I denotes the set o f all feasible p rocessor activ ation decision vectors, assuming a ll the queues have enough conten ts for p rocessing. W e assume that if a vector I ∈ I , then by cha nging one eleme nt of I f rom one to z ero, the newly o btained vector I ′ satisfies I ′ ∈ I . No te that the c hosen vector I ( t ) must always e nsure the co nstraint (24), which in this case implies that I ( t ) has to satisfy the following constraint: q j ( t ) ≥ X n ∈ P S j I n ( t ) β nj , ∀ j = 1 , ..., r. (47) Under this constrain t, we see th at the queues evolve according to the fo llowing q ueueing dynamics: q j ( t + 1 ) = q j ( t ) − X n ∈ P S j I n ( t ) β nj + D j ( t ) R j ( t ) , ∀ j ∈ Q s , q j ( t + 1 ) = q j ( t ) − X n ∈ P S j I n ( t ) β nj + X n ∈ P D j I n ( t ) α nj , ∀ j ∈ Q in . Note that we have used j ∈ Q to represent q j ∈ Q , and use n ∈ P to represent P n ∈ P in the above fo r notation simplicity . Th e o bjective is to max imize the time average of the following utility func tion: f ( t ) , X k ∈P o I k ( t ) p k ( t ) α ko − X j ∈Q s D j ( t ) R j ( t ) c j ( t ) (48) − X i ∈P in I i ( t ) C i ( t ) . 6 This ca n be vie wed as the diff erence betwee n profit and cost associated with these processors. 8 (48) can be used to mo del ap plications wh ere generatin g completely pro cessed contents is the p rimary target, e.g., [5]. B. Relatio n to the gen eral mod el W e see that in th is network m odel, the network state, the action, and the traffic and service fu nctions are given by : • The network state is given b y: S ( t ) = ( c j ( t ) , j ∈ Q s , C i ( t ) , i ∈ P in , p k ( t ) , k ∈ P o ) . • The action x ( t ) = ( D j ( t ) , j ∈ Q s , I n ( t ) , n ∈ P ) . • The arrival fu nctions are given by : A j ( t ) = A j ( S ( t ) , x ( t )) = D j ( t ) R j ( t ) , ∀ q j ∈ Q s , A j ( t ) = A j ( S ( t ) , x ( t )) = X n ∈ P D j I n ( t ) α nj , ∀ q j ∈ Q in . • The service f unctions are g iv en by: µ j ( t ) = µ j ( S ( t ) , x ( t )) = X n ∈ P S j I n ( t ) β nj , ∀ j. Thus, we see that this network model falls into the gener al processing network f ramework in Section I V, and Th eorem 2 will apply in this ca se. Th erefore, in the fo llowing, we will construct our PMW algorith m to ensure that Con dition 1 holds. C. The P MW algorithm W e n ow obtain the PMW algo rithm for th is gene ral network in the following. W e will look for a perturb ation vector th at is the same in all entries, i.e ., θ = θ 1 . W e first comp ute the “drift-plu s-penalty” expre ssion using th e weigh ted perturbed L yapun ov fun ction defined in (4 1) under some given positiv e constants { w j } r j =1 and some no nzero constant θ : ∆( t ) − V E  f ( t ) | q ( t )  ≤ B (49) − X j ∈Q s E  w j  q j ( t ) − θ  X n ∈ P S j I n ( t ) β nj − R j ( t ) D j ( t )  | q ( t )  − X j ∈Q in E  w j  q j ( t ) − θ  X n ∈ P S j I n ( t ) β nj − X n ∈ P D j I n ( t ) α nj  | q ( t )  − V E  X k ∈P o I k ( t ) p k ( t ) α ko − X j ∈Q s D j ( t ) R j ( t ) c j ( t ) − X i ∈P in I i ( t ) C i ( t ) | q ( t )  . Here B = w max  N q ( M s q β max ) 2 + N s q R 2 max + N in q ( M d q α max ) 2 2  , where w max = max j w j . W e also d enote w min = min j w j . Rearrangin g the terms, we get th e following: ∆( t ) − V E  f ( t ) | q ( t )  ≤ B (50) + X j ∈Q s E  V c j ( t ) + w j ( q j ( t ) − θ )  D j ( t ) R j ( t ) | q ( t )  − X k ∈P o E  I k ( t )  X j ∈ Q S k w j ( q j ( t ) − θ ) β kj + V p k ( t ) α ko  | q ( t )  − X i ∈P in E  I i ( t )  X j ∈ Q S i w j ( q j ( t ) − θ ) β ij − w h ( q h ( t ) − θ ) α ih − V C i ( t )  | q ( t )  . Here in the last ter m q h = Q D i . W e now presen t the PMW algorithm . W e see that in th is case the D ( S ( t )) θ , q ( t ) ( x ) fun ction is giv en by: D ( S ( t )) θ , q ( t ) ( x ) = − X j ∈Q s  V c j ( t ) + w j ( q j ( t ) − θ )  D j ( t ) R j ( t ) + X k ∈P o I k ( t )  X j ∈ Q S k w j ( q j ( t ) − θ ) β kj + V p k ( t ) α ko  + X i ∈P in I i ( t )  X j ∈ Q S i w j ( q j ( t ) − θ ) β ij (51) − w h ( q h ( t ) − θ ) α ih − V C i ( t )  . Our goa l is to design PMW in a way such that under any network state S ( t ) , the value of D ( S ( t )) θ , q ( t ) ( x ) is clo se to D ( S ( t )) ∗ θ , q ( t ) ( x ) , which is th e maximum value of D ( S ( t )) θ , q ( t ) ( x ) without the un derflow co nstraint (47), i.e., D ( S ( t )) ∗ θ , q ( t ) ( x ) = max D j ( t ) ∈ [0 , 1] , I ( t ) ∈I D ( S ( t )) θ , q ( t ) ( x ) . Specifically , PMW works as f ollows: PMW : Initialize θ . At every time slot t , ob serve S ( t ) and q ( t ) , and do the following: 1) Content Adm ission: Choose D j ( t ) = 1 , i.e., admit all new ar riv als to q j ∈ Q s if: V c j ( t ) + w j ( q j ( t ) − θ ) < 0 , (52) else set D j ( t ) = 0 . 2) Processor Activ ation: For each P i ∈ P in , d efine its weight W ( in ) i ( t ) as: W ( in ) i ( t ) =  X q j ∈ Q S i w j [ q j ( t ) − θ ] β ij (53) − w h [ q h ( t ) − θ ] α ih − V C i ( t )  + , where q h = Q D i . Similarly , for each P k ∈ P o , define its weight W ( o ) k ( t ) as: W ( o ) k ( t ) =  X q j ∈ Q S k w j [ q j ( t ) − θ ] β kj + V p k ( t ) α ko  + . (54) Then, ch oose an activ ation vector I ( t ) from I to max- imize: X i ∈P in I i ( t ) W ( in ) i ( t ) + X k ∈P o I k ( t ) W ( o ) k ( t ) , (55) subject to th e following queue edge constraints: a) For each P i ∈ P in , set I i ( t ) = 1 , i.e., activate processor P i , only if: • q j ( t ) ≥ M s q β max for all q j ∈ Q S i , • q h ( t ) ≤ θ , where q h = Q D i . b) For each P k ∈ P o , cho ose I k ( t ) = 1 on ly if: • q j ( t ) ≥ M s q β max for all q j ∈ Q S k . The ap proach of imposing the q ueue edge constrain ts was inspired b y th e work [22], where similar constrain ts are imposed for routing problems. Note that if without th ese queue edge constrain ts, then PMW will be the same as the ac tion that maximizes D ( s i ) θ , q ( t ) ( x ) with out the un derflow constrain t (47). 9 D. P erformance Here we show that PMW indeed en sures that the value of D ( S ( t )) θ , q ( t ) ( x ) is with in some additiv e co nstant o f D ( S ( t )) ∗ θ , q ( t ) ( x ) . In the fo llowing, we assum e that: θ ≥ max  V α max p max w min β min , V c min w min + M s q β max  . (56) W e also assume that the { w j } r j =1 values ar e cho sen such that for any pro cessor P i ∈ P in with the demand queue q h , we have fo r any supply queu e q j ∈ Q S i that: w j β ij ≥ w h α ih . (57) W e n ote that (56) can easily be satisfied and on ly requires θ = Θ( V ) . A way o f choosin g the { w j } r j =1 values to satisfy (57) is given in Sectio n VII-E. Note tha t in the special case when β ij = α ij = 1 for a ll i, j , simply u sing w j = 1 , ∀ j meets the co ndition (57). W e first look at the queu eing boun ds. By (52), q j admits new ar riv als on ly when q j ( t ) < θ − V c min /w j . Thu s: q j ( t ) ≤ θ − V c min /w j + R max , ∀ q j ∈ Q s , t. (58) Now b y the p rocessor activ ation ru le, we also see that: 0 ≤ q j ( t ) ≤ θ + M d q α max , ∀ q j ∈ Q in , t. (59) This is because under th e PMW alg orithm, a processor is activ ated only when all its supply qu eues have at least M s q β max units of contents, a nd when its d emand queue h as at most θ un its o f co ntents. The first r equiremen t e nsures th at q j ( t ) ≥ 0 for a ll time, wh ile the seco nd requ irement ensu res that q j ( t ) ≤ θ + M d q α max . Below , by defining: ν max , max  M d q α max , R max , M s q β max  , (60) we can co mpactly write (5 8) and (59) as: 0 ≤ q j ( t ) ≤ θ + ν max , ∀ q j ∈ Q , t. (61) T o p rove the p erforma nce of th e PMW algorithm, it suffices to prove the fo llowing lemm a, which shows that Condition 1 holds for so me finite con stant C under the PM W algorithm . Lemma 3: Su ppose (56) and (57) hold. Then under PMW , D ( S ( t )) θ , q ( t ) ( x ) ≥ D ( S ( t )) ∗ θ , q ( t ) ( x ) − C , where C = N p w max M p ν max β max . Pr oof: See Appen dix D. W e can now directly use The orem 2 to h av e the following corollary con cerning the perfor mance of PMW in this case: Corollary 1: Su ppose (25), (5 6) an d (5 7) hold . Th en PMW achieves the fo llowing: f P M W av ≥ f ∗ av − B + C V , (62) q P M W ≤ B + C + 2 V δ max η + θ r X j =1 w j , (63) where C = N p w max M p ν max β max , f P M W av and q P M W are the time av erage expec ted utility and time av erage exp ected weighted backlo g under PMW , r espectively .  Note that h ere δ max can be cho sen to be: δ max = max  ν max , N o p p max α max , N s q R max c max + N in p C max  . Also, since (56) only requires θ = Θ( V ) , an d w j = Θ(1 ) for all j , we see th at PMW in deed achieves an [ O (1 /V ) , O ( V )] utility-back log tr adeoff in this case. E. Choo sing the { w j } r j =1 values Here we describe how to cho ose the { w j } r j =1 values to satisfy (57). W e first let K be the maxim um n umber of processors that any path going from a queue to an ou tput processor can have. It is easy to see that K ≤ | N p | sinc e there is n o cycle in the network . The fo llowing algorithm termin ates in K iteration s. W e use w j ( k ) to den ote the value of w j at the k th iteration. In th e following, we u se q h n to den ote the demand que ue of a pro cessor P n . 1) At Iteration 1 , denote the set of queues that serve a s supply queue s for any output pro cessor as Q l 1 , i.e., Q l 1 = { q j : P S j ∩ P o 6 = φ } . Then set w j (1) = 1 for each q j ∈ Q l 1 . Also, set w j (1) = 0 for all other q j / ∈ Q l 1 . 2) At Iteratio n k = 2 , ..., K , den ote Q l k to be the set of queues that serve as supply queues for any p rocessor whose deman d queue is in Q l k − 1 , i.e., Q l k = { q j : ∃ P n ∈ P S j s.t. Q D n ∈ Q l k − 1 } . Then set: w j ( k ) = max  w j ( k − 1) , max n ∈ P S j w h n ( k − 1) α nh n β nj  , (64) where α nh n is the amo unt P n generates into q h n , which is the dem and queu e of P n . Also, set w j ( k ) = w j ( k − 1) for all q j / ∈ Q l k . 3) Output the { w j } r j =1 values. The following lemma shows that the above algorithm outputs a set of { w j } r j =1 values th at satisfy (57). Lemma 4: T he { w j } r j =1 values gen erated b y th e above algorithm satisfy ( 57). Pr oof: See Appen dix E. As a concrete example, we con sider the exam ple in Fig. 2, with the assump tion that each p rocessor, when acti vated, consumes one un it of content from each of its su pply queues and generates two units of contents in to its demand queue. In this examp le, we see th at K = 3 . Th us the algo rithm works as follows: 1) Iteration 1 , denote Q l 1 = { q 4 , q 5 , q 6 } , set w 4 (1) = w 5 (1) = w 6 (1) = 1 . For all other queu es, set w j (1) = 0 . 2) Iteration 2 , den ote Q l 2 = { q 1 , q 2 , q 3 , q 4 , q 5 } , set w 1 (2) = w 2 (2) = w 3 (2) = w 4 (2) = w 5 (2) = 2 . Set w 6 (2) = 1 . 3) Iteration 3 , den ote Q l 3 = { q 2 , q 3 } , set w 2 (3) = w 3 (3) = 4 . Set w 1 (3) = w 4 (3) = w 5 (3) = 2 , w 6 (3) = 1 . 4) T erminate an d ou tput w 1 = w 4 = w 5 = 2 , w 2 = w 3 = 4 , w 6 = 1 . V I I I . S I M U L A T I O N In th is sectio n, we simulate th e example giv en in Fig. 2. In this example, we assume each R j ( t ) is Bern oulli being 0 or 2 with equal p robab ilities. For eac h P i ∈ P in , i. e., P 1 , P 2 , P 3 , C i ( t ) is assum ed to be 1 o r 10 with probab ilities 0 . 3 and 10 0 . 7 , r espectively . For the o utput pro cessors P k ∈ P o , i.e., P 4 and P 5 , we assume that p k ( t ) = 1 or 3 with pr obabilities 0 . 6 and 0 . 4 , respectiv ely . W e assume that e ach proce ssor , when activ ated , takes o ne unit of content from each of its supply queues and generates two un its of con tents into its demand queue ( or to the output if it is an output p rocessor). W e f urther assume that all proce ssors can be tu rned on without af fecting others. Note th at in th is case, w e h av e c j ( t ) = 0 for a ll sou rce queues q j . It is easy to see th at in this case M p = M s q = M d q = 2 , β max = β min = 1 , and α max = 2 . Using the results in the above, we cho ose w 6 = 1 , w 1 = w 4 = w 5 = 2 , w 2 = w 3 = 4 . W e also use θ = 6 V a ccording to (56). W e simulate the PMW algorithm f or V ∈ { 5 , 7 , 10 , 1 5 , 20 , 50 , 100 } . Each simulatio n is run over 5 × 10 6 slots. Fig. 3 shows the utility an d back log perfor mance of th e PMW algo rithm. W e see that as V increases, the a verage utility per forman ce qu ickly conv erges to the op timal value. The average b acklog size also only gr ows linear in V . 0 20 40 60 80 100 2.04 2.06 2.08 2.1 2.12 2.14 2.16 2.18 2.2 2.22 V 0 20 40 60 80 100 0 500 1000 1500 2000 2500 3000 3500 V Utility Backlog Fig. 3. Utility and backlog performance of PMW . Fig. 4 also shows three sam ple path qu eue pro cesses in the first 10 4 slots under V = 10 0 . W e see that n o qu eue h as an underflow . This shows that all the acti vation decisions of P MW are f easible. I t is also easy to verify tha t the queueing bo unds (58) and (5 9) hold for all time. 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 0 100 200 300 400 500 600 700 Time Queue Size q 1 q 3 q 5 Fig. 4. Sample path backlog processes with V = 100 . W e observe in Fig. 4 that the queu e sizes usua lly fluctuate around certain fixed values. Similar “exponen tial attr action” pheno menon has been obser ved in prior work [ 19]. Hen ce ou r results can also be extended, using the results developed in [19], to ac hieve an average utility that is within O (1 /V ) of the op timal with only Θ([log ( V )] 2 ) average backlog size. In this c ase, we c an also imp lement the PMW algorithm with finite buffers using the idea o f floatin g queues in [23], which works as fo llows: F or each q j , we associate with it an actual buf fer of size Θ([log( V )] 2 ) a nd a c ounter . Wh en contents ar e sent into th e queue an d the buffer is not full, we store the contents in the actual buffer . Howe ver, whe n the buf fer is full and c ontents are sent to q j , these contents ar e dropp ed but the counter is incremented . Wher eas if contents are consumed from q j but q j does not have enough contents, then the cou nter is decr emented, and the action in th at slot is a ssumed to be null. Und er this metho d, it can b e shown that the d roppin g and underflow events h appen on ly with a very small p robab ility . Hence almost all action s are valid. Thus we lose a tiny fractio n in the utility pe rforman ce, but red uce th e average back log size from O ( V ) to O ([lo g ( V )] 2 ) . I X . C O N C L U S I O N In this paper, we d ev elop th e Perturb ed Max -W eigh t a l- gorithm (PMW) f or utility optim ization p roblems in gen eral processing n etworks. PMW is based on the usu al Max- W eight alg orithm for data networks. It has two main func- tionalities: queue under flow prevention and utility o ptimal scheduling . PMW simultane ously ach iev es both objectives by ca refully pe rturbing the weights used in the usual Max- W eight algorith m. W e show that PMW is a ble to achieve an [ O (1 / V ) , O ( V )] u tility-backlog tradeo ff. The PMW algo rithm developed here can be applied to pro blems in the area s o f data fusion, stream p rocessing and clou d computing . A P P E N D I X A – P R O O F O F T H E O R E M 1 W e p rove The orem 1 in this section, using an argumen t similar to the one u sed in [12]. Pr oof: (Theor em 1) Consider any stable scheduling policy Π , i.e., the cond itions (24) and (28) are satisfied unde r Π . W e let { ( f (0 ) , A (0) , µ (0)) , ( f (1 ) , A (1) , µ (1)) , ... } be a seq uence of (u tility , arriv al, ser vice) triple generated by Π . Then ther e exists a sub sequence of times { T i } i =1 , 2 ,... such th at T i → ∞ and tha t the limiting time average utility over time s T i is equal to the liminf average utility un der Π (defined by (29)). Now define the condition al average o f utility , an d arri val minu s service over T slots to be: ( φ ( s i ) ( T ); ǫ ( s i ) 1 ( T ); ... ; ǫ ( s i ) r ( T )) , (65) 1 T T − 1 X t =0 E  f ( t ); ǫ 1 ( t ); ... ; ǫ r ( t ) | S ( t ) = s i  , where ǫ j ( t ) = A j ( t ) − µ j ( t ) . Using Caratheo dory’ s the orem, it c an b e shown, as in [12] that, there exists a set of variables { a ( s i ) k ( T ) } r +2 k =1 and a set of action s { x ( s i ) k ( T ) } r +2 k =1 such tha t: φ ( s i ) ( T ) = r +2 X k =1 a ( s i ) k ( T ) f ( s i , x ( s i ) k ( T )) , and for a ll j = 1 , ..., r that: ǫ ( s i ) j ( T ) = r +2 X k =1 a ( s i ) k ( T )[ A j ( s i , x ( s i ) k ( T )) − µ j ( s i , x ( s i ) k ( T ))] . 11 Now using the continuity of f ( s i , · ) , A j ( s i , · ) , µ j ( s i , · ) , a nd the compactness of all the actions sets X ( s i ) , we can thu s fin d a sub-su bsequence ˜ T i → ∞ o f { T i } i =1 , 2 ,... that: a ( s i ) k ( ˜ T i ) → a ( s i ) k , x ( s i ) k ( ˜ T i )) → x ( s i ) k , (66) φ ( s i ) ( ˜ T i ) → φ ( s i ) , ǫ ( s i ) j ( ˜ T i ) → ǫ ( s i ) j , ∀ j = 1 , ..., r. (67 ) Therefo re the time average utility unde r the po licy Π ca n be expressed as: f Π av = X s i π s i φ ( s i ) = X s i π s i r +2 X k =1 a ( s i ) k f ( s i , x ( s i ) k ) . (68) Similarly , the a vera ge arrival rate min us th e average serv ice rate un der Π can be written as: ǫ j = X s i π s i ǫ ( s i ) j (69) = X s i π s i r +2 X k =1 a ( s i ) k [ A j ( s i , x ( s i ) k ) − µ j ( s i , x ( s i ) k )] ≤ 0 . The last ineq uality is due to the fact that Π is a stable p olicy and that E  q j (0)  < ∞ , hence the average ar riv al ra te to any q j must be no more than the av erage service rate of the queue [24]. Howe ver, by (2 4) we see that what is con sumed f rom a queue is always no more that wh at is gener ated into the queue. This implies that the inpu t rate into a queue is always no less than its output ra te. Thu s, ǫ j ≥ 0 f or all j . Ther efore we conclude tha t ǫ j = 0 for all j . Using this fact and (68), we see that V f Π av ≤ φ ∗ , wh ere φ ∗ is given in (31). This proves Theorem 1. A P P E N D I X B – P R O O F O F L E M M A 2 Here we pr ove Lem ma 2. Pr oof: Using the q ueueing equation (27), we have: [ q j ( t + 1 ) − θ j ] 2 = [( q j ( t ) − µ j ( t ) + A j ( t )) − θ j ] 2 = [ q j ( t ) − θ j ] 2 + ( µ j ( t ) − A j ( t )) 2 − 2  q j ( t ) − θ j  [ µ j ( t ) − A j ( t )] ≤ [ q j ( t ) − θ j ] 2 + 2 δ 2 max − 2  q j ( t ) − θ j  [ µ j ( t ) − A j ( t )] . Multiplying both sides with w j 2 and summin g the above over j = 1 , ..., r , we see th at: L ( t + 1) − L ( t ) ≤ B − r X j =1 w j  q j ( t ) − θ j  [ µ j ( t ) − A j ( t )] , where B = δ 2 max P r j =1 w j . Now add to b oth sides the term − V f ( t ) , we g et: L ( t + 1) − L ( t ) − V f ( t ) ≤ B − V f ( t ) (70) − r X j =1 w j  q j ( t ) − θ j  [ µ j ( t ) − A j ( t )] . T ak ing expectations over the ran dom ne twork state S ( t ) on both sides c onditionin g o n q ( t ) p roves the lemma. A P P E N D I X C – P R O O F O F T H E O R E M 2 Here we prove Th eorem 2. W e first have the following simple lemma. Lemma 5: For any ne twork state s i , we have: D ( s i ) ∗ θ , q ( t ) = g s i (( q ( t ) − θ ) ⊗ w ) , (71) where w = ( w 1 , ..., w r ) T . Pr oof: By comparing ( 44) with (39), we see that th e lemma follows. Pr oof: (T heorem 2) W e first recall the equ ation ( 70) as follows: L ( t + 1) − L ( t ) − V f ( t ) ≤ B − V f ( t ) (72) − r X j =1 w j  q j ( t ) − θ j  [ µ j ( t ) − A j ( t )] . Using D ( s i ) θ , q ( t ) ( x ) define d in (43), this c an be wr itten as: L ( t + 1) − L ( t ) − V f ( t ) ≤ B − D ( S ( t )) θ , q ( t ) ( x ( t )) . Here x ( t ) is PMW’ s action a t tim e t . Accordin g to Condition 1, we see that fo r any n etwork state S ( t ) = s i , PMW ensur es (24), and th at: D ( s i ) θ , q ( t ) ( x ) ≥ D ( s i ) ∗ θ , q ( t ) − C . Using (71), this implies that u nder PMW , L ( t + 1) − L ( t ) − V f ( t ) ≤ B − g s i (( q ( t ) − θ ) ⊗ w ) + C . T ak ing expectations over the random n etwork state on both sides co nditionin g on q ( t ) , and using ( 40), i.e., g ( γ ) = P s i π s i g s i ( γ ) , we get: ∆( t ) − V E  f ( t ) | q ( t )  ≤ B + C − g (( q ( t ) − θ ) ⊗ w ) . (73 ) Now u sing Theor em 1 and Lemma 1 , we have: V f ∗ av ≤ φ ∗ ≤ g ( γ ∗ ) ≤ g (( q ( t ) − θ ) ⊗ w ) . Therefo re, ∆( t ) − V E  f ( t ) | q ( t )  ≤ B + C − V f ∗ av . (74) T ak ing expectations over q ( t ) on both sides an d summin g the above over t = 0 , ..., T − 1 , we get: E  L ( T ) − L (0)  − T − 1 X t =0 V E  f ( t )  ≤ T ( B + C ) − T V f ∗ av . Rearrangin g term s, dividing bo th sides by V T , using the facts that L ( t ) ≥ 0 and E  L (0)  < ∞ , a nd taking th e liminf as T → ∞ , we get: f P M W av ≥ f ∗ av − ( B + C ) /V . (7 5) This proves (45). Now we pr ove (46). First, by us- ing the definition of ˆ g ( γ ) in (38), and plugg ing in the { x ( s i ) k , ϑ ( s i ) k } k =1 ,..., r + 2 i =1 ,...,M variables in the η -slackness assum p- tion (25) in Section IV -B, we see that: ˆ g (( q ( t ) − θ ) ⊗ w ) ≥ η r X j =1 w j [ q j ( t ) − θ j ] − V δ max . (76) 12 This by L emma 1 imp lies that: g (( q ( t ) − θ ) ⊗ w ) ≥ η r X j =1 w j [ q j ( t ) − θ j ] − V δ max . Using this in (73), we get: ∆( t ) − V E  f ( t ) | q ( t )  ≤ B + C + V δ max − η r X j =1 w j [ q j ( t ) − θ j ] . W e can now use a similar argument as above to get: η T − 1 X t =0 r X j =1 w j E  [ q j ( t ) − θ j ]  ≤ T ( B + C ) + 2 T V δ max + E  L (0)  . Dividing both sides by η T an d taking th e limsup as T → ∞ , we get: q P M W ≤ B + C + 2 V δ max η + r X j =1 w j θ j . This com pletes the proo f the theorem. A P P E N D I X D – P RO O F O F L E M M A 3 Here we prove Lem ma 3 b y co mparin g th e values of the three ter ms in D ( S ( t )) θ , q ( t ) ( x ) in (51) under PMW versus th eir values unde r the action that maximizes D ( S ( t )) θ , q ( t ) ( x ) in (51) subject to only the constraints D j ( t ) ∈ [0 , 1 ] , ∀ j ∈ Q s and I ( t ) ∈ I , called th e max-actio n. That i s, unde r the max-action, D ( S ( t )) θ , q ( t ) ( x ) = D ( S ( t )) ∗ θ , q ( t ) ( x ) . Note th at the max -action differs from PMW o nly in that it does not consider the q ueue edge constraint. Pr oof: (A) W e see that the first term, i.e. , − P j ∈Q s  V c j ( t ) + w j ( q j ( t ) − θ )  D j ( t ) R j ( t ) is m aximized under PMW . Thu s its value is th e same a s that under th e max-action . (B) W e now show that for any processor P n ∈ P , if it violates th e qu eue edg e con straint, then its weight is bo unded by M p w max ν max β max . This will then be used in Part (C) below to show that the value of D ( S ( t )) θ , q ( t ) ( x ) under PMW is within a co nstant of D ( S ( t )) ∗ θ , q ( t ) ( x ) und er the max- action. (B-I) For any P i ∈ P in , the following are the on ly two cases und er which P i violates the q ueue edge con straint. 1) Its d emand qu eue q h ( t ) ≥ θ . In this case, it is easy to see from (53) and (6 1) that: W ( in ) i ( t ) ≤ X j ∈ Q S i w j ν max β ij ≤ M p w max ν max β max . (77) 2) One of P i ’ s supply qu eue has a queue size less than M s q β max . In this ca se, we denote ˆ Q S i = { q j ∈ Q S i : q j ( t ) ≥ M s q β max } . Then we see that: W ( in ) i ( t ) = X j ∈ ˆ Q S i w j ( q j ( t ) − θ ) β ij − w h ( q h ( t ) − θ ) α ih + X j ∈ Q S i / ˆ Q S i w j ( q j ( t ) − θ ) β ij − V C i ( t ) ≤ X j ∈ ˆ Q S i w j ν max β ij + w h θα ih + X j ∈ Q S i / ˆ Q S i w j ( M s q β max − θ ) β ij . Here q h = Q D i . Now b y our selection of { w j } r j =1 , w j β ij ≥ w h α ih for any q j ∈ Q S i . Also u sing ν max ≥ M s q β max , we have: W ( in ) i ( t ) ≤ M p w max ν max β max . (78) (B - II) F or any P k ∈ P o , we see that it vio lates th e queue edge c onstraint only when one of its supply qu eues has size less than M s q β max . In th is case, we see that: W ( o ) k ( t ) ≤ X j ∈ ˆ Q S k w j ( q j ( t ) − θ ) β kj + V p k ( t ) α ko + X j ∈ Q S k / ˆ Q S k w j ( M s q β max − θ ) β ij ≤ M p w max ν max β max + V α max p max − w min θβ min . This by ( 56) implies that: W ( o ) k ( t ) ≤ M p w max ν max β max . (79) Using (7 7), ( 78) and (79), we see th at whenever a proc essor violates the queue edge constra int, its weigh t is at most M p w max ν max β max . (C) W e now show that the value of D ( S ( t )) θ , q ( t ) ( x ) un der PMW satisfies D ( S ( t )) θ , q ( t ) ( x ) ≥ D ( S ( t )) ∗ θ , q ( t ) ( x ) − C , where C = N p M p w max ν max β max . T o see this, let I ∗ ( t ) b e the activ ation vector obtained by the max-action , and let W ∗ ( t ) be the value of (55) u nder I ∗ ( t ) . W e also u se I P M W ( t ) an d W P M W ( t ) to denote the activ ation vector chosen by the PMW algorithm a nd the value of (55) under I P M W ( t ) . W e now constru ct an alter nate activ atio n vector ˆ I ( t ) by changing all elements in I ∗ ( t ) correspon ding to the p rocessors that vio late the queu e edge constraints to zero . Note then ˆ I ( t ) ∈ I is a feasible activation vector at tim e t , under which no pr ocessor violates the q ueue edge c onstraint. By Part ( B) ab ove, we see that the value of ( 55) under ˆ I ( t ) , denoted by ˆ W ( t ) , satisfies: ˆ W ( t ) ≥ W ∗ ( t ) − N p M p w max ν max β max . Now since I P M W ( t ) maximizes the value of ( 55) und er the queue edge co nstraints, we have: W P M W ( t ) ≥ ˆ W ( t ) ≥ W ∗ ( t ) − N p w max M p ν max β max . Thus, by co mbining the above and Part (A), we see that PMW maximizes the D ( S ( t )) θ , q ( t ) ( x ) to within C = N p M p w max ν max β max of the m aximum. 13 A P P E N D I X E – P RO O F O F L E M M A 4 Pr oof: (Proo f of Lemma 4) The proof consists of two main s teps. In the first step, we show th at the algorithm updates each w j value at least on ce. This shows that all the w j values for all the qu eues that serve as demand q ueues are upd ated at least o nce. In the secon d step , we show that if q h is the demand queue o f a p rocessor P i ∈ P in , then e very time af ter w h is upd ated, th e algorithm will also update w j for any q j ∈ Q S i before it term inates. This ensures that (57) h olds fo r any P i ∈ P in and henc e proves the lemma. First we see that af ter K iterations, we must have Q ⊂ ∪ K τ =1 Q l τ . Th is is becau se at Iteration k , we include in ∪ k τ =1 Q l τ all the queue s starting fro m which th ere exists a path to an output processor th at co ntains k pr ocessors. Thus all the w j values a re up dated at least on ce. Now co nsider a queue q h . Suppo se q h is the de mand queue of a pr ocessor P i ∈ P in . W e see that there exists a time ˆ k ≤ K a t which w h is last modified . Sup pose w h is last modified at Iter ation ˆ k < K , in wh ich case q h ∈ Q l ˆ k . Then all the q ueues q j ∈ Q S i will be in Q l ˆ k +1 . Thus their w j values will b e mod ified at Iteration ˆ k + 1 ≤ K . Th is implies that at Iteration ˆ k + 1 , we will have w j ( ˆ k + 1) β ij ≥ w h ( ˆ k ) α ih . Since q h / ∈ Q l k for k ≥ ˆ k + 1 , we have w h ( k ) = w h ( ˆ k ) fo r all k ≥ ˆ k + 1 . There fore w j ( k ) β ij ≥ w h ( k ) α ih ∀ ˆ k + 1 ≤ k ≤ K , because w j ( k ) is not decr easing. Therefo re the on ly case when th e alg orithm can fail is when w h is upda ted at Iteration k = K , in which case w h may increase but the w j values f or q j ∈ Q S i are no t modified accordin gly . Ho wev er , since w h is u pdated a t Iteratio n k = K , this implies that ther e exists a p ath from q h to an outp ut processor that has K processor s. T his in turn imp lies that starting from any q j ∈ Q S i , th ere exists a path to an o utput processor that co ntains K + 1 processors. This con tradicts the definition of K . Thus the lemma f ollows. R E F E R E N C E S [1] J. M. Harison. A broader view of brownian network s. Ann. Appl. Pr obab . , 2003. [2] J. G. Dai and W . L in. Maxi mum pressure policies in stochastic processing networks. Operati ons Researc h, V ol 53, 197-218 , 2005. [3] L. Jian g and J. W alra nd. Stable and utilit y-maximizin g scheduling for stochasti c processing networks. A llerton Confer ence on Communicatio n, Contr ol, and Computing , 2009. [4] H. Zhao, C. H. Xia, Z. Liu, and D. T owsle y . A unified modeling frame work for distribu ted resource allocation of general fork and join processing networks. Pro c. of AC M Sigmetrics , 2010. [5] M. J. Neely and L. Huang. Dynamic product assembly and in ventory control for maximum profit. IEEE Confere nce on Decisi on and Contr ol (CDC), Atlanta, Geor gia , Dec. 2010. [6] L. Amini, N. Jain, A. Sehgal, J. Silber , and O. V ersche ure. Adapti ve con- trol of ex treme-scal e stream processing systems. Proc . of Internat ional Confer ence on Distrib uted Computing Systems (ICDCS) , 2006. [7] A. Biem, E. Bouillet, H. Feng, A. Ranganathan , A. Riabov , O. V er - scheure, H. Ko utsopoulos, and C. Moran . Ibm infosph ere streams for scalabl e, real-time , intell igent transportati on services. P r oceedings of the international confer ence on Mana geme nt of data , 2010. [8] J. Cao, S. A. Jarvis, S. Saini, and G. R. Nudd. Gridflow: workflow management for grid computing. Intl. Symposium on Cluster Computing and the Grid (CCGrid) , 2003. [9] S. Eswaran , M. P . Johnson, A. Misra, and T . La Porta. Adaptiv e in-netw ork processing for bandwidth and ener gy constrained mission- oriente d multi-hop wireless network s. Pro c. of DCOSS , 2009. [10] A. Eryilmaz and R. Srikant. Fair re source alloca tion in wirele ss networks using queue -length-b ased scheduling and congestio n control. IE EE/ACM T rans. Netw . , 15(6):1 333–1344, 2007. [11] M. J. Neely . Energy optimal control for time-v arying wirele ss netw orks. IEEE T ransactions on Informatio n Theory 52(7): 2915-2934 , July 2006. [12] L. Huang and M. J. N eely . The optimality of two prices: Maximizing re venu e in a stochast ic netw ork. Pro c. of 45th Annual Allerton Confer- ence on Communicatio n, Contr ol, and Computing (in vited paper) , Sept. 2007. [13] R. Urg aonkar and M. J. Neely . Opportunistic schedul ing with reliabil ity guarant ees in cogniti ve radio networks. IEEE INFOCOM Pr oceedi ngs , April 2008. [14] M. J. Neely . Super-fast delay tradeof fs for utilit y opti mal fair scheduling in wirele ss networks. IE EE Journal on Selected Areas in Communica- tions (JSAC), Speci al Issue on Nonlinear Optimization of Communic a- tion Systems , 24(8), Aug. 2006. [15] Y . Y i and M. Chiang . Stochast ic netw ork utilit y m aximiza tion: A tribute to kell y’ s paper published in this journal a decade ago. E ur opean T ransaction s on T ele communicat ions , vol . 19, no. 4, pp. 421-442, June 2008. [16] D. I. Shuman and M. Liu. Energy-ef ficient transmission scheduling for wireless media streaming with strict underflo w constrain ts. W iOpt , 2008. [17] L. T assiulas and A. Ephremides. Stability properti es of constrained queuein g systems and s chedul ing poli cies for maximum throughput in multihop radio networks. IEEE T rans. on Aut omatic Contr ol, vol. 37, no. 12, pp. 1936-1949 , Dec. 1992. [18] L. Georgiad is, M. J. Neely , and L. T assiulas. Resource Allocation and Cr oss-Layer Contro l in W ireless Network s . Found ations and Tre nds in Networ king V ol. 1, no. 1, pp. 1-144, 2006. [19] L. Huang and M. J. Neely . Delay reduct ion via lagrange multipli ers in stochasti c network optimizati on. P r oc. of W iOpt, Seoul , June 2009. [20] L. Huang an d M. J. Neely . Max-weight achi e ves the e xact [ o (1 /v ) , o ( v )] utilit y-delay tradeo ff under markov dynami cs. arXiv:1008 .0200v1 , 2010. [21] D. P . Bertsekas, A. Nedic, and A. E. Ozdaglar . Con vex Analysis and Optimizati on . Boston: Athena Scienti fic, 2003. [22] M. J. Neely . Uni versa l schedulin g for networ ks with arbitrary traf fic, channe ls, and mobility . arXiv:1001.0960v1 , Jan 2010. [23] S. Moeller , A. Sridharan, B. Krishnamachari , and O. Gnawal i. Routing without routes: The backpressure collec tion protocol. 9th ACM /IEEE Internati onal Conferen ce on Informatio n Proce ssing in Sensor Networks (IPSN) , 2010. [24] M. J. Neely . Dynamic P ower Allocation and Routing for Sate llite and W ireless Networks with T ime V arying Channels . PhD thesis, Massachuset ts Institute of T echnology , L aborat ory for Information and Decision Systems (LIDS), 2003.