Lessons from the Failure and Subsequent Success of a Complex Healthcare Sector IT Project

Lessons from the Failure and S ubsequent Succes s of a Complex Health care S ector IT Project David Greenwo od , Ali K hajeh - Hosseini, Ian Sommerv ille Dependabl e Socio -T echnical Systems Engineering Group, School of Computer Science, Univ ersity of St Andrews, UK {dsg22, akh, ifs}@cs.st - andrews.ac.uk Abstract. T hi s p a p e r a r g u e s that IT f ai l u r e s d i ag n o s e d as e r r o r s at t h e t e c h ni c a l or project management level are often mistakenly point ing to symptoms of failure rather than a project’s underlying socio - complexity ( co mplexit y resulting from the interactions of people and gro ups ) whic h i s usually t he ac tu al source of failure . We pro pose a n o v e l method that adopts a socio - complexity lens, Stakeholder Impact A nalysis, that can be used to identify ris ks associated with soc io - comp lexity as it i s g r o u n d e d i n i n s i g h t s f r o m t h e s o c i a l s c i e n c e s , psychology and management science . This paper demonstr ates the eff ectivenes s of Stakeholder Impact Analysis by using the 1 9 9 2 L o n d o n A m b u l a n c e S e r v i c e Computer Aided Di spatch proj ect as a case study, and shows that had our method been used to identif y the risks and had they been mitigated , i t w o u l d have reduced the risk of project failure. This paper’s o r i g i n a l c o n t r i b u t i o n comprises adopting a socio - complexity lens and expanding upon exi sting accounts of failure by examining fai lures at a level of granularity not seen elsewhere that e n a b l e s t h e u n d e r l y i n g s o c i o - co mplexity sources of risk to be identified. Keywords: IT Failure, Socio - Technical Sys tems Engineer ing, Stakeh older Impact Analys is, O rganisational An alysis, Socio - c omplexity 1 Introduction IT failures have been plaguing th e implementation of IT systems sin ce their introduction in to organisatio ns in the 1960s. To day’s econ omic practice s are more dependent upon IT than ever before and therefore understanding and preventing IT failures is even more worthy of academic study and industrial investment. There is a disciplinary history of analysing IT failures as technical failures and in more recent times broad ening this to include pro ject manag ement and organisatio nal factors. This paper’s o r i g i n a l c o n t r i b u t i o n e x p a n d s u p o n t h e e x i s t i n g b o d y o f I T f a i l u r e s t u d i e s b y arguing that IT failures diagnosed as erro rs at the technical or pro ject management level are often mis takenly poin ting to s ymptoms of failure rather than a project’s underlying socio - complexity which is r egu larly the ac tual source o f failure. This paper demonstra tes the effecti veness of Stakehol der Impact Analysis by using the 1 9 9 2 /1996 L o n d o n A m b u l a n c e S e r v i c e C o m p u t e r A i d e d Dispatc h ( LASCAD92 /96 ) p r o j e c t s as a case study, and shows that had our method been used to identif y the risks and had they been mitigated , i t w o u l d h a v e r e d u c e d t h e r i s k o f project failure . The Stakehold er Impact Analysis me thod is ba sed on a fr amework th at examines failures at a level of g r a n u l a r i t y n o t s e e n e l s e w h e r e , w h i c h e n a b l e s t h e underlying socio - complexity sources of risk to be identified rather than its symptoms at the technical and project management level. The paper is structur ed such that: S ec tio n 2 intro duces the case - study, and studies of IT f ailure and their d eficiencies; S ection 3 describes s ocio - complexity and the Stakeholder Conflict Matrix; Section 4 details the paper ’ s case - study methodology; Section 5 summari s es the results; Section 6 describes the lessons lea rned; and Sec tion 7 concludes the paper and disc usses future work . 2 Background 2.1 Studies of IT Failu re Information System Developm ent ( ISD ) fail ure is an ongo ing t heme in t he st udy o f large scale c omplex IT systems and is e specially well researched in the Information Systems (IS) literature. Failure is, by nature, not a well - defined concept as it is the consequence of an evaluation that a stakeholder applies to a system, or project, at a particular time with respect to their e x p e c t a t i o n s [1] . It has been demonstrate d that over - time a stakeh older’s perception of an ISD p roject changes with time and is dependent upon their perspective and the legitimacy of other voices over time [2] . In consequence the literature identifies many broad types of failure e. g . C o r re s p on d e nc e failure; Process failure; and Interaction failure [3, 4] . Corresponden ce failure whe re the ‘w rong’ thing is delivered but to the ‘correct’ budg et an d sch edule, process failur e is w here the ‘correct’ thing is delivere d b ut to the w rong budget or sched ule, and interaction failure where the ‘correct’ thing is de livered to the ‘co rrect’ budget an d schedule but stakeholders do not use the system ISD failure is studie d using tw o dominant approache s , i d e n t i f y i n g f a i l u r e f a c t o r s , and i d e n t i f y i n g f a i l u r e p r o c e s s e s / d y n a m i c s [1] . Both appro aches are based o n t h e hypothesis that if the causes of failure can be id entified , w e c a n m o n i t o r a n d c o n t r o l those ris ks. Ea rly stud ies foc used o n sim ple cau ses an d often identif ied the cause as the shortcom ings of individual IT pra ctitioners or IT mana gers. Howe ver [5] identified that most of the d ifficulties were social a nd behavioural rather than technical. T his result wa s confirme d and gen eralised by [6] an d later by [7] . The bul k of th e re search has therefo re i dentif ied social and behaviour fac tors associated with failure and these factors have typical ly been t reated as causes of failure d espite the fact that few large - scale statistical studies exist to i l l u s t r a t e h o w risk factors correlat e with ISD project performance. 2.2 Risk Iden tification & Man ageme nt Meth ods Risks are events or situati ons that may adversely affec t project performance. Accordi ng to [8] r i s k e x p o s u r e ( R E ) i s a f u n c t i o n o f t h e probability of an adverse outcome P(OU) and the loss to parties if the adverse outcome occurs L(OU ) ; therefore, R E = P ( O U ) * L ( U O ) . R i s k i s m a n a g e d i n I T p r o j e c t s u s i n g a t w o - step procedure: risk assessment , and r i s k c o n t r o l . R i s k a s s e s s m e n t c o m p r i s e s identification , an alysis and prioritiz ation, which m eans possible so urces of events th at may cause adverse outcomes are elicit ed, their likelih ood of occurrence and their consequences are estimated and then finally the risks are ranked for importance. Risk con trol comp rises plannin g and manage ment, resolution an d monitoring, w hich means plans are m ade documenting how each risk will be dealt with (e.g. avoidance, transfer, reduction ), these plans are executed and then the effectiv eness of the plans are monitored and correctiv e action is tak en where appropriate . Project risks are normally identified by a ris k analyst using their personal experience, project files, historical data and supplementary data co llected on the basis of one - to - one interviews or analyst led wo rk i n g g r o up s. I n a l l t h e s e s i t u a t i o n s s u p po rt tools su ch as checklists, mode ls or prompts may be use d to focus atte ntion o n a particular area of risk [9 , 10] . Ther e ex ist m any check lists a nd models wh ich focus on different aspects of projects [8, 11 - 14] e . g . a s p e c t s t h a t a u t h o r s b e l i e v e s i g n i f i c a n t l y contribute t o process failure, correspondence fai lure, or interaction failure. These identification methods are incorporated into boarder risk management approac hes such a s ‘RiskM an’, ‘R ISKIT’ and S ODIS [15 - 17] . We argue that the weakness tha t these exis ting appro aches su ffer from is that th ey poin t to sym ptoms of failure rather than a pro ject’s unde rlying soc io - complexity (complexity resulting from the interactions of peop le and groups), which is u sually the actual source o f failure and thus the risk th at needs to b e addresse d. 2. 3 The Case stud y – The Lo ndon A mbula nce Service & LA SCAD The London Ambulance Service (LAS) is an NHS Trust that provides emergency ambulance service to the whole of the L ondon area (approx 620 square miles). Similar t o o t h e r p u b l i c s e c t o r o r g a n i s a t i o n s , the L AS exhibits significant socio - complexity. In 1990 it was involved in a protracted pay and conditions dispute with its wo rkers’ union. In 19 91, th e 268 senio r and mid dle m anagem ent po sts in th e LAS were cut to 53. T h e o fficial r eport of the e nquiry into the LAS a l s o c o m m e n t e d t h a t this restructur ing caused a great de al of anxiety to work ers creating the perception of continual pressure to down - size and improve performance [18] . The London Ambulance Service Computer Aided Despatch ( LASCAD ) p r o j e c t i s an ex emplar o f an IT enabled work - transfo rmation p roject. It com prised the automation of the dispatch of ambulances from call taki ng to ambulance dispatch. Th e need for the automation project was identified in the m id 1980s when the government perceived the London Ambulance Service to be failing to modernise and generally invest in its work force. After a failed modernisat ion attempt in 1987, a sec ond attempt was initiated i n October 1990. The planned implementation date was 8 th o f January 1992 but by M arch 1992, the second pha se of live trials was suspended due to the users o f the sys tem not h aving co nfidence in the system resulting in the Nation U nion of Public Employees to get involved [8]. Until the 26 th O c t o b e r 1 9 9 2 , t h e automated system struggled to satisf y i t s o b j e c t i v e s r e s u l t i n g i n a m b u l a n c e s b e i n g scheduled inefficiently and the system was finally switched off in the following 48 hours. Fol lowing this f ailure a pu blic enquiry was pe rformed as the failure ha d becom e one of the highest profile IT failures in the UK. The project w as revitalised by the newly appointed management after its failure in 92, however rather than pursuing the same appr oach as the LASCAD92 failure they opted for a radically different approach. The new project, called LA SCAD96 in this paper, com prised a non - time pressured in - house development project. A custom off - th e - shelf ( COTS ) so l u t i o n wa s evaluated, but rejected, and a part ici pat ive a ppr oach u til isi ng pr oto typi ng wa s ado pted to gene rate use r participa tion an d own ership [ 4]. The initial system was e xtremely simple and improvemen ts w ere released in small increments where by Sep tember 1996 more radical enhancements were being accepted by the user - base resulting in a jump in pro ductivity fro m 38% - 60% of calls being despat ched in 3minutes [4]. 3 Socio-c omplexity and the Stakeholder Conflict Matrix Complexit y is the quality or propert y of being complicated such that an acto r is unable to make predictions about the consequences of an action as the variables or their interactions are not we ll understood. Soc io - complexity comprises the complicatedness associated with interactions between people and groups of people. This is obs erved in an organisation, or project organisation, in terms of pluralities of perspectives and pluralities of behaviour. Pluralities of perspective can be caused by ambiguity, uncertainty, un - surfaced assum ptions, d ifferences in interpretive norm s , or group processes [19] . Plurali ties of behaviour can be caused by differen ces in expectations, intentions and behavioural norms [19] . Stakeholder resistan ce is a f orm of confl ict and a s such is the embodiment of socio - complexity and a consequent of the tensions between the pluralities of stakeholders’ perspectives and behaviours. Stakeholder resistance can be viewed as a feedback mechani sm between stakehol ders about the goodness of fit between their local environment and the intended project. It is for this reason that the Stakeholder Impact Analys is metho d appe ars to be an interesting risk analysis method as by understanding the k inds of thin g that would mak e specific stakeholder s , or stakeholder group s resist , man y spe cif ic ri sks can b e ide nti fie d upf ront in a syst emat ic manner. The Stakeholder Impact Analysis method used in this case s tudy is an operationalisat ion of the Stakeholder Conflict Matrix [20 ] . T h e S t a k e h o l d e r C o n f l i c t Matr ix is the product of a multi - disciplinary li terature sur vey of the sources of conflict in social settings. It c o m p r i s e d a r e v iew of over 50 article s in jour nals covering Applie d Psychol ogy, Administr ativ e Science , Ma nagemen t S cience, Social Science, Computer Science and Informati on Systems. The Stakeholder Conflict Matrix helps address t he l imitations o f exi sting studies b y a p p r oaching failure from a conflict perspective thus allo wing r i g o r o u s l y d e s i g n e d s t u d i e s f r o m diverse fields to b e brought to bear upon our understanding of failure thus p r o v i d i n g m u c h n e e d e d granulari ty and cor roborating findings within C S/Informatics dom ain . The liter ature review findings are condensed into the S takeho lder C onflict M atrix (Table 1) . Table 1 - Stakeholder Conflict M atrix Conflict Individual Intra - group Inter - group Task N/A Disagree ment ove r what to do t o achi eve aim s Process N/A Disagree ment ove r how t o accompl ish ta sk Relational (Procedural / Distrib utive Injusti ce) N/A Unjust d elegat ions of status, duty or resources within the group Unjust delegations of status, duty or resources between groups Role (Time, Resou rces and Capabilities) Incompatibility between required ac tivity and practical constraints Role (Valu es, Status and Satisfaction ) Incompatibility between required ac tivity and an individual’s va lues, status or satisfaction Incompatibility between required activity and group values, status and s atisfaction Role (Mult iple) An individual is assigne d multiple roles with in compatib le act ivitie s or assessment metrics A group is assigned multiple roles with incompatible activities or assessment metric s Relation al (Incompatibility between actors) N/A Negative emotio nality between individuals Negative emotionality between groups 4 Methodology A case study methodolog y was adopted to evalu ate the eff ectiveness of th e Stakeholder Impact Analysis . Data on the LASC A D project was collected from five separate sources: [18] [21] [2 2] [23] [24] an d it was verified that each account broadly corrobora ted one another o ther to ascertain the reliability of the data. The data was analysed using Stakeholder Impact A nalysis (SIA) which is an operationalisation of the Sta keholder Conflict Matrix . SIA c o m p r i s e s: 1 . Identifyi ng key stakeholders; 2 . Identifying changes in what t a s k s stakeh older w o u l d b e r e q u i r e d t o p e r f o r m a n d how they we re to per form the m; 3 . H y p o t h e s i s i n g w h a t t h e l i k e l y c o n s e q u e n c e s o f t h e changes are with regards to stakeholders time, resources, capabilities, values, status and satisfaction; 4. H ypothesising the impact of t h e s e c h a n g e s w i t h i n t h e w i d e r context of relational factors such as tense relationships between individuals or groups to whic h stake holders belong; 5. Hy po t he si si ng w h et he r th e s ta ke ho l de r w i ll pe rc ei ve the change as un just (either pro cedurally or d istributively) bas ed upon the nature of changes and t he stakeholder s relational context. The ri sks i dentifi ed usi ng St akeholder Impact Analys is wer e use d to test two hypotheses w ith the overall aim of illustrating that had S takeholder Impact A nalysis been performed duri ng the LASCAD92 project and had the identified risks been mitig ated it would have reduc ed t he ri sk o f pro ject fai lure . [H1] T he causes of the failure identified in other analyses ma p ont o r isks identified by the Stak eholder impact analysis If h ypothesis one is sup ported by the r esults then this co rroborates that Stakeh older Impact analysis cap tures the risks captured by existing approac hes. The hypo thesis was teste d by ide ntifying the causes of the LAS CAD9 2 f ailure a s identified by the Offici al Inquiry and supportin g academi c literat ure and for each cause i dentifying the relevant risks raised by the Stakeholder Im pact Analysis. [H2] R i s k s i d e n t i f i e d i n t h e failed LASCAD92 proje ct wer e appr opriately mitig ated/ parti all y miti gated in the successful LASCAD96 project If hypoth esis two is su pported b y the results the n this corroborates that the kinds of risks identified by Stakeholder Impact Analysis ha ve a significant im pact o n project performance if left unm itigated. Hypo thesis two was tested by i dentifying if t he changes to practices identified by [4] in th e i r c as e - study of the successful turn - around (LASCAD 96) mitig ated or partially mitigated each risk identifi ed b y S takehold er I mpact A nalysis. 5 Results A summary of the resu lts i s pre sente d to highlight the m ost sig nificant risks identified by Stakeho lder Imp act An alysis. Th e results support both [H1] and [H2] . Hy pothesis 1 is supporte d by the fact t hat al l i dentified causes of fai lure i n the literature map to ris ks ide ntified by S takeholde r Im pact A nalysis thus illustrating it is able to comprehensively identify risks (See http:// www.cs.st - andrews.ac.uk/~dsg22/LAS CAD/ tables.pdf ). Hypothesis 2 is supported by the fact that Stakeholder I mpact A nalysis found 24 stakeholder r isks associated wit h the LASCAD 92 and 96 projects. Of these 24 risks, 2 were appropriate ly mitigated or partially m itigated during LASCA S 92. In contrast d ur in g L A S C A D 9 6 , 2 3 o f t he 24 risks we re appro priately m itigated o r partially mitigated ( S e e t a b l e 2 ) . Since LASCAD 96 was considered a success this sugge sts that t he Stak eholder I mpact A nalysis identifies risk factors that if left unmitigated contribute to stakeholder resistance and ultimately pro ject failure. T h e s e f i n d i n g s g i v e g r o u n d s f o r t h e statement that had Stakeholder Impact Analysis been performed during LASCAD 92 it would ha ve reduce d the risk of p roject failure . Table 2 - Mitigated vs un mitigated risks and project o utcome s # Partiall y Miti gat ed Unmiti gated Project Outcome LASCAD92 2 22 Failure LASCAD96 23 1 Success Due to space restric tions only highli ghts of each stakeho lders ’ perspe ctive will be provided starting first with control room staff, then LAS management and finally w ith ambulance crew. It was identified that control room staff m ay oppose the impleme ntation for th e fo llowing reasons: [C5] the system could be perceived to reduce their status as it was designed to routin ise thei r work removing many eleme nts of skill/local knowledge that previously existed; [C7] the project w as a top - dow n initiative and the re w as a history of co nflict a nd n on - cooperation with management; [C8] the project could b e perc eived as distributively unfair as staff received little benefit from the change but would need to retrain and there was a possibility of job cuts. In LASCAD96 the first risk [ C 5 ] w a s p a r t i a l l y m i t i g a t e d b y t h e s o f t w a r e b e i n g designed to support decision -m aking rather than automate it. T his mitigated the risk as rather than routinising work and rem oving elements of skill it supplemented staff skill an d knowledge. The second risk [C7] was partiall y mitigate d by signi fican t changes to LA S manag ement and the b uilding of cooperation by fulfilling staff requests e.g. the provision o f a new more com fortable control room and the hiring of additional staff to reduce the burden of transition on existing staff. Th is mitigated the risk beca use m anagement were less like ly to be perceive d as adve rsaries and partners to cooperate w ith. The third risk [C8] was mit igat ed as staff re ceive d improved workin g condition s and were given contr ol of the roll out by being given a righ t to veto changes that they did not approve of. Th is mitigated the risk b ecause staf f now had control over changes and therefore w ere able to reject changes that w ere perceived to be deliver li ttle net be nefit. It was identified that the LAS man agement m ay oppose the p roject for the following reasons: [M 2 ] they p erceive that they w ould be given inadequate time/resource to mak e the proje ct a succ ess; [M 3] the y do no t hav e the will or ability to de velop skills to m anage the system or its dev elopmen t; [M5] may be reluct ant to report negative info rmation to ex ecutes for fear of losing their status or job; [M 6] due to p oor past relations mana gement ma y v iew control room staf f or am bulance crew as being obstructive when providing genuine negative feedback about the system. The first risk [M 2] w a s m i t i g a t e d i n L A SCAD92 by management being fearful for their jobs ho wever the co nsequen ce of this w as tha t mana gement were reluctant to p rovide negative feedback (bad news) for fear of being seen as obstructive and thus losing their jo b. In LA SCAD 96 the risk w as mitigated by a n ex e c u t i ve l e v el c o mm it me nt t o provide whatever resources and time needed as well as fostering an atmosphere of openness. This mitigated the risk managers as the availability of resources and time reduced the percep tion of inadequate reso urces or tim e being mad e available. The second risk [M3] was mitiga ted in LASCAD92 by top - down pressure to meet ORCON targ ets and risks of job losses. In LASCAD96 thi s risk was mitigated by topdown pressure to m eet targets, the provision of additional resources and re structuring to bolster managem ent and operational staff. This mitigated the risk of lack o f w ill or ability by providin g a suppo rtive a tmosphe re for staff and bringin g in staff with additional skills and experience for o thers to learn from. The third risk [M5] was mitiga ted in LASCAD96 by introd ucing a flexible time - frame thus removing pressure for immediate result s and also recent hiring of additional staff reinforcing the messag e that jobs were not at risk. This mitigat ed the risk because the repor ting of negative information was no longer perceived as adversarial / a problem as there w as plenty of time and there was no atmosphere of job losses. The forth risk [M6 ] w as n ot mitig ated in L ASCAD92 o r 96. It was ide ntified that am bulance crew ma y oppose the pr oject for the following reasons: [A2] the system c ould be p erceived to interfere w ith crew values of ‘rapid response’ if it does no t take into accoun t c rew experience and local know ledge; [A 7] the system could be perc eived as manag ement interfere nce and pr ocedurally unjust due to ongoing issues w ith staff consultation; [A 8] th e system could b e perceived as distributive ly unjust are crew lose a their autonomy from which derive little benefit. The first and second risks [A2][A7] were mitigat ed in LASCAD96 by entering into consultation with staff and involving crew with testing and approval of equipment prior to go - live. T his mitig ated th e risks as crew s co uld inf luence changes to ensure they did not conflict with their valu es o r m odes of operating . Th e th ird risk [A8] was mitig ated by i mprovi ng cr ew wor king condi tions by issui ng t hem wi th p erson al r adios so they cou ld comm unicate with them selves and also upgrading ambulances for comfort. This mitigated the risk becau se the crews could shape the changes to ensure the distributio n of benef it/loss was fair a s they perce ived it. 6 Lessons Learned The LASCAD 92/96 project emphasises the importa nce of stakeh older resista nce and its impact on the success of IT projects. The following list identifies the main lessons learn ed from th is case - stud y: • Resist ance comprises stakeho lders providing feedback on how they perceive a system to impac t their local environme nt and therefore ad dressing their perceived risks is valuable as it facilitates good fit between their local and the system and the changes it brings about. • The majorit y of stakehol der risks can be appropria tely mitigated, or partial ly mitig ated, by seni or management demonstrat ing that they are willing to invest the resource s it takes to get a pr oject done w ell and t h a t t h e y a r e o p e n a n d respond to continu ous consultation/feedb ack from all stakeh olders. • Sources of continuous consultati on/feedback are an important practi ce to mitig ate risk these include: up - front consultation; ong oing drop - in sessions; user acceptance testing where users can delay go - live if un happy. • Software and system design should be mindful of stakeholde r values, satisfaction, and status. One pa rticularly impo rtant area is to avoid fully automating user decision - making and inste ad suppor ting the u ser to make better decisions. This is beneficial as it reduc es the risk of removing satisf ying or status granting aspects work whilst simultaneously enabling the user to incorporate their local kn owledge or expertise. • Stakeholder risks that are mitigated v ia coercion tend to dampe n fee dback loops between stakeholders resulting in poor communication and ultimately a project that is not a go od fit with its environm ent and thu s a failure. 7 Conclusion It is concluded that: i) IT failures diagnosed as errors a t the technical or p roject manageme nt levels are often mistak enly point ing to symptoms of failure rath er than a project’s underlying socio - complexity which is regularly the actual source failure; ii) had Stakeholder impact analysis been performed during th e LASCAD 92 project and the identified risks had been mitigated it would have reduc ed the risk of project failure. Claim i) was corrobora ted b y th e fac t that the LASCA D case stu dy demonstrates that: a) most risks ca n be approp riately m itigated, or partia lly m i t i g a t e d , b y s e n i o r manageme nt demonstra ting that they are willing to invest the resources it takes to get a project done well and that they are open and respond to continuous consultation/feedback from all stakeholders; b) s ources of continuous consulta tion/feedback are an imp ortant p ractice to m i t i g a t e r i s k s t h e s e i n c l u d e : u p- front consultation; o ngoing drop - in s essions; User acceptanc e testing whe re use rs can delay go - live if un happy; c) risks tha t a re mitigated via co ercion tend to dam pen feedback loo ps between stakeholders resulting in poor comm unication and ultimately a project that is not a good fit with its environment and thus a failure. Claim ii) was corroborated by the fact that: d) the risks identified map onto causes of failure identified in o th er analyse s of the failu re; e) all the risks identified in the LASCAD92 project were m itigate d in the successful LASCAD 96 project suggesting their mitiga tion contrib uted to the success of the pro ject. The conclusion is limited by the usual lim itations of cas e - study research. Future w o r k i s p l a n n e d t o c o r r o b o r a t e t h e s e f i n d i n g s w i t h o t h e r c a s e s t u d i e s a n d also perform statistical studies of IT project performance to establish the average performance impact of each kind of Socio - complexity derived risk fa ct or a s i n d ic a t ed in the Stake holder Co nflict Matrix . References 1. Sauer, C.: Deciding the future for IS failures not the choice you might think. In: Currie, W.L., Galliers, B. (eds.): Rethinking Management Information Systems. Oxford Un iversity Press (199 9) pp.279 - 309 2. Wilso n, M., Howcroft, D.: Re - conceptualising failure: social shaping meets IS research. Eur J Inf Syst 11 (2002) 236 - 250 3. Lyytinen, K., Hirschh eim, R.: Information systems failures - a s u r v e y a n d classification of the empirical literature. Oxford Surveys in Information Technology. Oxford Universi ty Pres s, Inc . (1987 ) 257 - 309 4. Fitzgerald, G., Russo, N.L.: The turnaround of the London ambulance service computer - aided despatch system (LASCAD ). Eur. J. Inf. Syst. 14 (2005 ) 244 - 257 5. Colton, K.W.: Computers and Police: Patterns of Success and Failure. Sloan Managem ent Revie w 2 (1972) 75 - 98 6. Lucas, H .C.: W hy information systems fail. Columbia University Press, New York (1975) 7. Boland, R., Hirschheim, R.: Series Foreword . In: Bola nd, R., Hirsch heim, R. (ed s.): Critical Issues in Inf ormation Systems Res earch. Wi ley, Chi chester (1987) 8. Boehm, B.W.: Software Risk Management: Principles and Practices. IEEE Softw. 8 (1991) 32 - 41 9. Chapman, R.J. : The effect iveness of wor king gro up ris k iden tifi cati on and a ssess ment techniques. Intern ational Journal of Project Manag ement 16 (1998) 333 - 343 10. Chapman, R.J.: The controlling influences on effecti ve risk identifi cation and assessment for construction design m anagement. Internatio nal Journal of Project Management 19 (2001) 147 - 160 11. Carr, M.J., Konda, S.L., Monarch, I., Ulrich, F.C., W alker, C.F.: Taxonomy - Based Risk Iden tificat ion. ( June 1993) 12. Moynih an, T.: How Experi ence d Project Managers Assess Risk. IEEE Softw. 14 (1997) 35 - 41 13. Keil, M., Cule, P. E., Lyyt inen, K. , Schmidt , R.C.: A framework for iden tifyin g software project risks. Commun. ACM 41 (1998) 76 - 83 14. Schmidt, R., Lyytinen, K., Keil, M., Cule, P.: Identifying Software Project Risks: An International Delphi Stu dy. J. Manage. Inf. Syst. 17 (2001) 5 - 36 15. Carter, B.: Introduc ing Riskman: The European Project Risk Management Method ology . St atio nery Offi ce Bo oks ( 1996) 16. Kontio, J. : The Riskit Metho d for Software Risk Management. Institut e for Advanced Computer S tudies and Department of Co mputer Scien ce University of M aryland 17. Gotterba rn, D., Roger son, S.: Res ponsible Risk Assessmen t with Softwa re Developmen t: Creatin g the Software Development Impact Statement. Communications of the Associa tion for I nformat ion Systems 15 (2005) 18. Page, D., Williams, P., Boyd, D.: Report of the Inquiry Into The London A mbulance Service. The Comm unications Directorate, South W est Thames Regional Health A uthority, London (1993) 19. Greenwood, D.: A Normati ve Agent Organisati onal M o d e l l i n g a p p r o a c h t o a i d t h e analysis of collaborative business processes spanning multiple enterprises: A modelling approach to aid Service Oriented Information System development in business organisations.: Informatics, Vol. MSc. U niversity of Reading, Re ading (2007) 9 7 20. Greenwood, D.: A Literat ure Survey of Unsupporti ve Stakeholder behaviour and its impact on the developm ent and deployment of Com plex IT systems. Un iversity of St Andrews, St Andrews (2010) 29 21. Beynon - Davies, P.: Information system s `failure': the case of the London A mbulance Service's Compute r Aided Despatch project. Eu r J Inf Syst 4 (1995) 171 - 184 22. Finkelstein, A., Dowell, J.: A comedy of errors: the London Ambulance Service case study. Proceedings of the 8th International W orkshop on Software Specification and Design. IEEE Computer S ociety (1996) 23. Hougham, M.: London Ambula nce Service computer - aided despatch system . International Journal of Project Man agement 14 (1996) 103 - 110 24. Beynon - Davies, P.: Human error and informati on systems failure: the case of the London ambulance service computer - aided despatch system project. Interacting with Computers 11 (1999) 699 - 720