Teaching spreadsheet development using peer audit and self-audit methods for reducing error

Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org TEACHING SPREAD SHEET DEVELOPM ENT USING PEER AUD IT AND SELF-AUDIT METH ODS FOR REDUCING ERR ORS Chadwick D., Sue R., Information Integrity Research Cent re, School of Computing & Mathemat ical Sciences, University of Greenwich, Londo n SE10 9LS, United Kingdom Phone:+44 0208 8331 8509 Fax: 0208 8331 8665 Email:cd 02@gre.ac.uk Abstract Recent research has highlighted the high incidence of errors in spreadsheet models used in industry. In an attempt to reduce the incidence of such errors, a teaching approach has been devised which aids stu dents to r educe their likelihood of making common errors during development. The approach co mprises of spreadsheet checking methods based o n the commonly accepted educational paradigms of peer assessment and self-assessment. However, these paradigms are here based upon practical techniques commonly used by the internal audit function such as peer a udit and control and risk self-assessment. The result o f this symbiosis between educational assessment and professional audit is a method that educates students in a set of str uctured, transferable skills f or spreadsheet error-ch ecking which are useful for increasing error-awaren ess in the classroom and for reducing busi ness risk in the workplace. Keywords: spreadsheets, er rors, audit, risk-managemen t, teaching 1.0 THE PROBLEM OF SPREADSHEET ERRORS The world-wide pro blem of spreadsheet errors and the need to act to prevent, detec t and minimise the effect of errors has been wid ely reported both in industry studies and in more formal acade mic research studies. 1.1 The Indu stry Experience The Business Modelling G roup at KPMG Cons ulting have been only too aware of the problems of spreadsheet errors and freq uently cite the findings of a K PMG survey of financial models ba sed on spreadsheets. The surv ey findings, reported in an article in Int ernal Auditing by Chadwick, D. [1], found that: • 95% of models were foun d to contain major errors (erro rs that could affect decisions based o n the results of the model), • 59% of models were judged to have 'poor' model design, • 92% of those that dealt with tax issues had significant tax errors, • 75% had significant accou nting errors. These findings have been reinforced by a statement in the sa me article by David Finch, H ead of Internal Audit at Superdrug plc, who believed that people shouldn 't be too surprised at the high rate of spreadsheet errors as: "The use of spreadsheets in business is a l ittle like Christmas for child ren. They are too excited to get on with the game to read or think about the 'rules' whi ch are generally boring…. There is often lit tle control over end user developments in spreadsheets with litt le if any standardisation in development proces ses by users in different departments, little risk analysis and a gener al assumption that models, on which important business deci sions are made, are accurate" The UK Custo ms and Excise department collects VAT on behalf of the British gov ernment. Raymond Butler, a computer auditor in the d epartment, has written ex tensively on the frequency of errors found in spreadsheets used by businesses to calculate the amoun t of VAT that they shou ld pay. Butler and his team have done much wor k on risk assessment methods to enable them to deter mine which VAT spreadsheets have a likelihood of error and nee d to be audited ; Butler R. [13]. One of the questions the Custo ms & Excise audit team ask themselves is wh ether the organisation for whom the develo pment is being made have a n adequate policy governing develo pment, testing and use of spreadsheet models an d applications. They also Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org investigate whether there is evidence that this policy is observed and enforced. Butler states that there are many examples of errors fo und in both field audits and ex periments which show that good development pract ice is rarely codified into business pr ocedures and even wh en it is, the rules and restrictions it requires are n ot followed to any sign ificant degree. 1.2 The A cademic Research Exper ience For the past 15 y ears, Professor Raymond Panko at the University of Hawaii has been study ing and writing prolifically upon the phenomenon of spreadsheet error . In addition to compiling bodies o f data from research establishments and bu sinesses around the world he has conducted his own experimen ts with students. It is no t possible to report u pon all of Panko's findings in this paper but a recent paper of his shows a co mparison of findings from several educat ional studies (see fig.1.1 ). Fig.1.1 Spreadsheet Deve lopment Experiments : Panko R. [12] Study Year Sample Subjects Spread sheets % with Errors Cell Error Rate (CER) Brown & Gould 1987 ED 9 27 63% NR Olson & Nilsen (1,2) 1987- 1988 ED 14 14 NA 21% Lerch (1,2) 1988 ED 21 21 NA 9.3% Hassinen (2) on paper 1988 Ugrad 92 355 55% 4.3% Hassinen (2) online 1988 Ugrad 10 48 48% NR Janvrin & Morrison (3) Study 1, alone 1996 Ugrad 78 61 NR 7% to 10% Janvrin & Morrison (3) Study 1, dyads 1996 Ugrad 88 44 NR 8% Janvrin & Morrison (3) Study 2, alone 1996 Ugrad 88 88 NR 8% to 17% Kreie (post test) 1997 73 73 42% 2.5% Teo & Tan (4) 1997 Ugrad 168 168 42% 2.1% Panko & Halverson, alone 1997 Ugrad 42 42 79% 5.6% Panko & Halverson, dyads 1997 Ugrad 46 23 78% 3.8% Panko & Halverson, tetrads 1997 Ugrad 44 11 64% 1.9% Panko & Sprague (4) 1999 Ugrad 102 102 35% 2.2% Panko & Sprague (4,5) 1999 MBA (NE) 2 6 26 35% 2.1% Panko & Sprague (4,6) 1999 MBA ED) 17 17 24% 1.1% Panko & Halverson, monads 2000 Ugrad 35 35 86% 4.6% Panko & Halverson, triads 2000 Ugrad 45 15 27% 1.0% Total Sample 998 1170 51% (7) NR = not reported ED = experienced developer NE = not very experienced with development at w ork Ugrad = undergraduate stu dents (1) Measured errors befo re subject had a chance t o correct them: (2) Onl y measured error rate in for mula cells : (3) Only measured error rate in cells linking s preadsheets : (4) Wall Task designed to be relati vely simple and free of domain kn owledge requirements : (5) MBA students with little or no develo pment experience (6) MBA students with consi derable development experienc e : (7) Weighted average Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org Panko's figures show that, over the given experiment s, 51% of the spreadsheets contained err ors. Of course no mention is made of the type of errors and their severity in terms of spreadsheet integrity ; this aspect is more fully reported in industry based studies (see KPMG survey reported above). However, more importantly from the teaching view, Pan ko goes on to state: "These studies… used a variety of subjects from rank novices to highly experienced spreadsheet developers. All subject groups made errors, and when Pan ko and Sprague [1999]* directl y compared error rates for undergradu ates, MBA students with little or no spreads heet development experience, and MBA stu dents with at least 250 hours of spreadsheet development experience, they found no significant differences in error rates across the groups" *Reference to Panko, R. R., and Sprague, R. H. J. “Hittin g the Wall: Errors in Develo ping and Code-Inspecting a ‘Si mple’ Spreadsheet Mode l.” Decision Support System s , (22) 1999, pp. 337-353 . 2.0 THE CAUSES OF ERRORS DURING TRA INING There is a common pro blem with students learning n ew computing skills. Research conducted by Van Vliet et al [2] into comparing s elf-appraisal and objective tests of learners abilities ha s indicated that novices o f both sexes consistently overrate their own computer literacy skills. This has led the m to think they have correctly solved a pro blem when, in fact, they have not. Panko R.[12], cites one experiment in which student sp readsheet developers were g iven a spreadsheet to build from a written spec ification. The 'developers' were then asked to estimate the l ikelihood that they had made an error during development. The median estimate w as 10%, and the mean was 18 %. In fact, 86% had made an error in their sprea dsheet. When debriefed in class and asked to raise their han ds if they thought they were among the success ful 14%, well over half of all su bjects raised their hands. Research conducted by Chadwick et al. [3] has also sh own that student spreadsheet bui lders frequently make errors that they are quite confident are not there and ar e often amazed wh en the errors are pointed out to them. The same research has catalogued the different types of common errors an d has shown that, amongst student spreadsheet builders, there is a tendency to create formulae that do not accurately model the calculations and p rocessing phenomena of the real-wo rld, and to copy for mulae incorrectly from one part of a spreadsheet model to anot her. An assertion made in Chadw ick et al. [3] is that even wh en training is given it too frequently concentrates on ‘how to do things correctly ’ and often ignores ‘how to av oid doing things incorrectly’. One possible way of ‘avoiding doing thin gs incorrectly’ is to make n ovice spreadsheet builders more aware of co mmon errors and encourage them to apply checking controls during d evelopment commensurate with the a mount of risk associated with producing a possibly incorrect model. This is supported in the profession al audit literature reviewing the teaching of computer aud iting: "In terms of teaching …several ac tivities can help students …understan d that control should be used sparing ly but appropriately; the right amount of control dep ends on the associated risk" Herremans [4] During the exercise repo rted in this paper, students w ere specifically asked to assess the r isks inherent in any model they built. Error-aw areness skills were also extensively developed by p roviding opportunities for students to check for errors : 1. through self-assessment by error- auditing their own spreadsheet models and also tho se deliberately seeded with errors for training purposes, 2. through peer-assessmen t by auditing the models of other students. 3.0 SELF AUDIT Self-assessment perfor med by students has been gaining in popularity in the last few y ears as a means of enabling students to be re flective on the quality o f their own work. Such an approach has been found to be effective in improving intellectua l internalisation and intuition [9]. During the trials reported herein self- assessment techniques w ere based upon the self-audit meth ods in use in industry. Formal lecture sessions were given in which stud ents were encouraged to consider the advantages and disadvan tages of spreadsheet audit in general. The main l ecture themes were designed to improve awareness o f: • the usefulness of auditing spreadsh eet models, Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org • the need to find error s, their causes and effects, • the business risks associated with incorrect spreadsheets, • the common errors by b eing exposed to them through seeded mode ls, • the advantages of using self-aud it check lists during development. 3.1 Improving Awareness of the Usefulness of Self-Auditing Students were made aware that Control Self Assess ment (CSA) techniques were accepted audit methods in industry. They were asked to consider the advantages and disadvantages of performing self-checking as in fig 3.1. Fig 3.1 Control Self Assess ment (CSA) CSA: Advantages CSA Disadvantages Can build CSA into own work schedule May be tendency not to do at all Learn by detecting own errors May be that not all errors get detected Greater awareness so fewer errors later May become blasé and so negligent Etc … 3.2 Improving Awareness of Errors, Their C auses and Effects Students were encouraged to identify the events durin g development which could create, propagate or exacerbate errors. Th ey were asked to tabulate what they thought were possible erro r-events, their causes and possible effects as in fig. 3 .2. Fig. 3.2 Sample Error-Event Table : Error Events, Causes, Effects Error -inducing events identified? Causes? Effect ? Incorrect Formulae 1. Typographical error 2. Wrong arithmetical precedent rules Wrong output data leading to faulty decision-making Mistakes in input data-set 1. Typographical error 2. Wrong data-set 3. Wrong data source Wrong output data leading to faulty decision-making 3.3 Improving Awareness of Business Risks As part of their introduc tion to CSA, students were told that org anisations often extended CSA to consider the inherent risks (Control and Risk Self-Assessment o r CRSA). CRSA, in this context, involved personal awareness of the risks inherent in a spreadsheet pro ject, the possible error situations, alternative arrangements a client may need in event o f failure as well as the dev eloper's own response to fai lure and problems. Students were encouraged to iden tify the possible risks inheren t in an application they had been asked to develop and to identify the minimum ri sk-set. Fig 3.3 Minimum Risk Se t Compiled By A Typ ical Student Risk Self Assessment of Errors In My Spr eadsheet How important to my client is the spreadsh eet I am developing ? Which parts of the spreadsheet m odel are most critical to my clients b usiness? What compensation might th e client demand from me for any error ? Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org 3.4 Improving Awareness Using Specially Seeded Models For Training Improving awareness of e rrors was accomplished by presenting students with s preadsheet models deliberately created with err ors and asking them to id entify the errors and assess the ex tent of commercial risk associated w ith them. An introductory exa mple from the teaching materials is sho wn in 3.4.1 (there were more complicated models!) . 3.4.1 Example of An Error Seeded Model Bungee Breakspeare wants to open his own night-club and has put details of expected income and out-goings in a spreadsheet which he has sent to his bank-manager for a start-up loan.(see fig3.4). Question: Which parts of the spreadsheet carry the most likelihood of an error ? Question : Is there an error in this spreadsheet ? Question: What business risks are associated with an error in this model? Fig. 3.4 Example of Error identificat ion and Risk Assessment quest ion Figures in £ Jan Feb Mar Apr May June Entrance Ticket sales 4,000 3,500 3,000 3,000 4,000 5,000 Bar sales: drink and food 1,500 1,000 1,000 1,000 2,000 2,000 TOTAL INCOME 5,500 4,500 4,000 4,000 6,000 7,000 Wages: bouncers a nd bar staff 1000 1,000 1,200 1,200 1,500 1,500 Electricity charges 0 3,500 0 0 4,000 £0 Rent for premises 0 0 5,500 0 0 5,500 TOTAL OUTGOINGS 1000 4500 6700 1,200 5,500 7,000 MONTHLY PROFIT 4,500 0 -2,700 2,800 500 0 ACCUMULATING PROFI T 4,500 4,500 -2,700 100 600 600 3.5 Practical Self Assessm ent Check List The use of check lists is w ell documented in audit literature. The ir use has also been champ ioned in educational studies with busine ss students learning to b uild spreadsheet models [8]. The checklist used with students in the research rep orted herein was a pre-even t checklist asking students to consider appropriate action prior to building a s preadsheet. Fig. 3.5 The Self-Assess ment checklist stages (RADAR life-cycle, spreadsheet modularisation, logical models, and 2-A's approach are spreadsheet building methods developed and taught at the University of Greenwich - for further information on some of these terms refer to Chadwick et al [ 5] and Rajalingham et al. [6] and [7]. For risk assessment see 3.3 above) 4.0 PEER ASSESSMENT Peer assessment is frequent ly used as an educational t ool because “of its potential for the dev elopment of students’ autonomy, maturity and critical abilities” [10] . The advantages are cited as giving students appreciation of other studen ts’ work and thereby identi fying common mistakes. How ever, there are generally two main criticisms fro m the students’ point of view. It is commonly stated that althoug h the peer assessor is given the opportunity to review another students wor k, they, the peer assessor do not : Before starting your spreadsheet ha ve you considered if and when to perform : 1. the RADAR development stages yes/no? 2. modularisation of the spreadshe et if necessary yes/no ? 3. creation of a logical model of the spreadsheet yes/no ? 4. use of the 2 A’s : which f unctions to test yes/no ? 5. create a User-Guide yes/n o ? 6. create a risk assessment table identifying risks yes/no ? Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org • gain any particular new ski lls by so doing, • have their own proficiency at peer assessment itself a ssessed, • understand why they should perform part of the tu tor's job. These factors have been major drawbacks to the use of peer as sessment generally . However, as a possible solution to this, the appro ach herein described uses methods derived from the peer aud it functions commonly used in industry. Peer audi t is a method pro moted by internal audit departments w hereby application development projects are ch ecked against corporate standards by persons not invo lved in the original development but of equa l skill and business know ledge to the original developers. Such peer auditors are commonly trained in basic aud it techniques and taught how to report their findings. Signif icant savings have been made by companies e mploying this approach as it reduces the need for a costly centra l audit department. It also enables far more freq uent reviews to occur and at much earlier stages in the application development life-cycle. A peer assess ment method used in education and based on professional peer audit was considered to possibly have certain advantages w hich would address so me of the criticisms given by students above. For the assessing students in par ticular, it was considered that they might develop a set of skills co mmensurate with those of practicing au ditors and which would be immediately transferable to a work environment. Proficiency in such sk ills could also be educationally a ssessed. The peer audit skills were taught using lectures an d practical exercises revolv ing around: • improving awareness of the usefulness of peer-audit, • a practical peer audit check lis t, • practical guidance on w riting a peer-audit report, • exercises for peer audit practice. 4.1 Improv ing Awareness of the Usefu lness of Peer-Audit Students were introduced to peer-audit and encour aged to tabulate the advantages and disadvantages. Fig 4.1 Peer Aud it Advantages and Disadvantages As Per Typical Student PA: Advantages PA: Disadvantages Peers have similar spreadsheet skills But may make same mistakes Peers have similar domain knowledge But it may be too l imited Etc… 4.2 Practical Peer Audit Check List and Peer Audit R eport Three types of practical au dit methods were used to dev elop skills in searching for and identifying errors and omissions in other stud ents' work. Peer assessing students were encouraged to: • find errors in other studen ts spreadsheets using a post-event check list, • prepare an audit report on their findings. The post-event check l ist (fig. 4.2) was used by the assessing student to check that the asses sed student had actually done what was requ ired in terms of the standards of model development defined in the requirements specification given to the m. Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org Fig. 4.2 Peer Aud it Check List and Report Contents 4.3 Exercises In U se of the Peer Audit Method The peer audit method w as used for practical assessme nt of each student’s spread sheet. The peer exercise occurred in a univers ity computing laboratory with a working version of the spread sheet. This was in accordance with the work of Boud (in Heywood, 1989) who argued; "peer assessment should be formalis ed in the laboratory. In this way students begi n to take responsibility for their own learning and gain insight into thei r own performance through having t o judge the work of others. [11] Fig. 4.3 Peer Aud it Sheet giving instructions on h ow to perform audit Each student was g iven a written requirements speci fication and given time to build the appropriate spreadsheet model. The mo del was then audited by ano ther student using the peer au dit sheet (sample shown in fig. 4.3) who gav e a mark and wrote an audit report upon his/her findings. The audited s tudent received the mark and the auditing stud ent was themselves graded by the tutor on the quality of their audit report. To avoid collusion an d bias, it was deemed necessar y to select the auditing 'pairs' b y producing a list of names, randomly sorted, an d requiring each to audit the name benea th theirs in the list with a wrap from bottom to top of the list. Th is ensured that:- • students did not audit their p ersonal friends, • no mutual-audit pairs arose i.e. A audits B, B audits A , Peer Audit Check List 1. Check User-guide in e xistence with minimum information 2. Check spreadsheet has been modularised correctly 3. Select and check key f unctions using 2 A’s approach 4. Test several key functi ons with sample input data from the user Peer Audit Report Prepare audit report (one side A 4) includes auditors name, date, iden tifying details of spreadsheet, aud itors findings on above 4 checks. The audit report is to be a WP do cument with same name as spreadshee t file. AUDITOR STUDENT NAME:…………… AUDITED STUDENT NAME:…………… PART A : AUDIT OF SPREADSHEET 1 1 mark if correct, 0 if spreadsheet is wrong Qu 1 : Change the $ to £ exchange rate to 1.69, Change the D.Mks to £ exchange rate to 2.40, Change the Yen to £ exchange rate to 1300 The 3 Mthly Total (in I9) should change to 4773.99 PART B : AUDIT OF SPREADSHEET 2 1 mark if correct, 0 marks if wrong or non-existent Qu 2 : What is the figure for the 3 Mthly Total over all salespeople for April to June? Should be 47425.00 Qu 3 Does User Guide identify spreadsheet builder? Qu 4 Does User Guide show the Date of Creation? Qu 5 Does User Guide specify spreadsheet purpose? Qu 6 Is there a lookup table for commission rates? Qu 7 Check any 3 key functions. Are the functions correct? MARK GIVEN: Now Prepare Your Audit Report on Spreadsheet 2 model . Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org • no mutual-audit triples arose i.e. A audits B, B audits C, C audits A. 5.0 CONCLUSIONS The audit approach w as used on a cohort of second y ear undergraduate informatio n systems students. These fairly novice students (in s erious spreadsheet work) w ere selected for this project for two reasons. They were generally eager to participate in new assessment methods an d also had not yet formed bad habits which resulted in poor spreadsh eet model design. This is in k eeping with the suggestion by Habeshaw et al. [10] and also by Chadwick et al. [1 ] that peer assessment techniq ues should be started early in the programme of study. 5.1 Analysis of Stu dent Feedback On The Exerc ise Anonymous feedback was obtained from 42 participating students who answered fiv e questions and w ere permitted to give commen ts. All reported favourably on most criteria (except report writing ) saying they had not only learned from the e xperience but also enjoyed it. Fig.4.4 Feedback from stud ents Yes No/Not sure 1.SELF AUDIT: was the exercise useful in exposing your mistakes? 50% 50% 2.SEEDED MODEL: were these useful in increasing your ability to find mistakes? 65% 35% 3.PEER AUDIT: was the exercise useful in developing your skills for finding errors in other people's work? 72% 28% 4.PEER REPORT: was the exercise useful in developing your report writing skills? 45% 55% 5.PEER AUDIT: was being audited a learning experience for you? 55% 45% SELF AUDIT: was the exer cise useful in exposing your m istakes? Students commented favour ably on the disciplined approach forced upon them. Althoug h they identified the time overhead of hav ing to think about inherent risks they felt that the insight gained into possible problem areas outweighed the time overhead. SEEDED MODEL: were these useful in increasing you r ability to find mistakes? Students were keen on this method of raising error-aw areness and requested more exerc ises of further complexity. PEER AUDIT: was the ex ercise useful in developing you r skills for finding errors in other p eople's work? Student were enthusiastic that having to audit each other's work had been a ben eficial exercise. Many remarked how surpr ised they were to see the variety of spreadshe et models produced fro m what they had considered to be req uirements that only had 'on e' solution. PEER REPORT: was the exerc ise useful in developing your report writing skills? The seemingly neg ative reply to this question may indicate that the question w as wrongly asked on the follow-up questionnaire. I t may well have been that the rep ort was not useful in 'develop ing' skills already in existence (all the students h ad already received much expo sure to report writing op portunities). PEER AUDIT: was being audited a learning experie nce for you? There is no doubt, fro m the feedback received, that stud ents were motivated to do well by the prospect of having their wo rk examined by another. 5.2 Analysis of The C omplete exercise Crucial to the success of this exercise were: Extracted from Controlling the Subvers ive Spreadsheet – Risks, Audit and Develo pment Methods Proceedings of EuSpRIG 2001 Conference ISBN: 1 86166 17 9 7 Copyright © 2001 Europea n Spreadsheet Risks Interest Group w ww.eusprig.org • early explanation of the rat ionale for the peer and self-asse ssment, • clear and simple instructions for the s tudents to follow, • objective criteria using prescr ibed test data, and • monitoring of the exercise by the tutor through viewing the audit report. The work continues with c urrent cohorts of students and the methods used have been refined. The concept of peer and self audit has prov ed to be so useful with spreadshee t teaching that it is now being extended to the teaching of databases. References [1] Chadwick, D. Stop The Subversive Spreadsheet . Institu te of Internal Auditors. 'Interna l Auditing' magazine May 2000 [2] Van Vliet P., Kletk e M.G and Chakraborty G . The Measurement o f Computer Literacy . Internatio nal Journal of Hu man-Computer Studies Vo l 40 , p835-857 (1994) [3]. Chadwick D, Knight J, Clipsham P. Information Integrity In End-user Systems . Chapman & Hall (Proceedings of the First Ann ual IFIP TC-11 Working Group 11.5 Working Conference on Integrity and Internal Control in Infor mation Systems, Zurich, Swi tzerland 3-4 December 1997) [4] Herremans I.M How To Tu rn Students On To The Id ea of Control. Institute of Internal Auditors. 'Internal Auditin g' magazine August 1998 [5] Chadwick D., Rajalingh am K., Knight B., Edwards D. Quality Control in Sp readsheets: A Visual Approach usi ng Colour Codings To Reduce Er rors In Formulae . British Co mputer Society 2000 (Proceedings of the 8th Inte rnational Conference on So ftware Quality Management, G reenwich, UK 2000) [6] Rajalingha m K, Chadwi ck D. Integrity Control of S preadsheets: Organisatio n & Tools . Kluwer Acade mic Publishers, 1 998, pp 147-168 (Proceedings o f the Second Annual IFIP TC-11 W orking Group 11.5 Working Conference on Integrity and Internal Control in I nformation S ystems, Virginia, USA, 19-20 November (1998) [7] Rajalingham K, Chadw ick D, Knight B, Edwards D.: An Ap proach to Improving the Quality of Spreadsheet Models, British Computer Society, Proceedings of the Seventh Inte rnational Conference on Softw are Quality management, South ampton, UK, (1999) [8] S.E. Kruck, Towards A Th eory Of Spreadsheet Accu racy : An Empirical Study, Conference of the Decision Scien ces Institute, Las Vegas, Nov 1998 [9] S.Habeshaw, G. Gibbs and T. Habeshaw, 53 P roblems With Large Classes – Making Th e Best Of A Bad Job , Techni cal and Educational Services, Bristol, 1992 [10] S.Habeshaw, G. Gibbs an d T. Habeshaw, 53 Interestin g Ways To Assess Your Stud ents , Technical and Education al Services, Bristol, 1993 [11] J. Heywood, Assessment in Higher Education , Wiley, Bath 1989 [12] Panko R. Spread sheet Errors: What We Know. Wh at We Think We Can Do , Proceeding s of the First Symposiu m of the European Spreadsheet Risks Interest Grou p, Greenwich, UK, 2000 [13] Butler R., Risk Asses sment For Spreadsheet Develo pments : Choosing Which Models to Audit Proceedings of the First Sy mposium of the European Sp readsheet Risks Interest Group , Greenwich, UK, 2000