Key attributes of a modern statistical computing tool

Key attributes of a mo dern statistical computing to ol Amelia McNamar a ∗ Stati stical and Data Sciences, Sm ith Coll eg e June 5 , 20 18 Abstract In the 1990s, statisticia ns b egan thinking in a principled wa y ab out ho w com- putation co uld b etter sup p ort the learnin g and doing of statistics. Since then, the pace of softw are dev elopmen t has acce lerated, adv ancemen ts in computing and data science ha v e mo ved the goalp osts, a nd it is time to r eassess. Soft w are con tinues to b e dev elop ed to help d o and learn statistics, but there is little critical ev aluation of the resulting to ols, and no acc epted fr amew ork with whic h to critique them. This pap er present s a set of attributes necessary for a mo d ern statistica l computing to ol. The framew ork was designed to b e b roadly applicable to b oth novic e and exp ert us er s , with a p articular fo cus on m aking m ore sup p ortive statistical compu ting environ- men ts. A mo dern statistical computing to ol should b e accessible, pro vide easy en try , privilege data as a fir st-order ob ject, supp ort exp loratory and confirmatory analy- sis, allo w for flexible plot creation, s upp ort randomization, b e interac tiv e, include inherent do cumen tation, su pp ort narr ativ e, pub lishing, and repro d ucibilit y , and b e flexible to extensions. Ideally , all these attributes could b e in corp orated in to one to ol, supp orting u sers at all level s, bu t a more reasonable goal is for to ols designed for no vices and professionals to ‘reac h across the gap,’ taking insp iration from eac h others’ strengths. Keywor ds: Soft w are design, Softw are ev aluation, Exploratory data analysis, Data visual- ization, Randomization, Bo o tstrap, Repro ducibility ∗ The author is grateful to N ick Hor to n and Rob Go uld, who read drafts of this paper and ga ve many insightful comments. Tha nks als o to Alan Kay and Mark Hansen, who read early versions of t his w or k in m y do ctor al diss e r tation. 1 INTR ODUCTION T o ols shap e the wa y w e see the w orld, and statistical computing to ols are no exception. Affordances for building g raphics, represen ting data, a nd mo difying analysis a ll impact ho w users conceiv e of their statistical products. As our world b ecomes increasingly da t a-driv en, it is imp ortant to critically examine the to ols we are using and lo ok tow ard t he future of computational p ossibilities. The use of the term ‘to ol’ to mean computer soft w are or a programming language hark ens to a time when computers do more than just amplify h uman a bilities: they also augmen t them (Pe a, 1985). In the same w a y ph ysical to ols allo w us to do more than w e could on o ur o wn, computers can allow hu mans to ‘see’ and ‘think with’ data in higher dimensions and with more clarity than they o therwise could. Statistical computing to ols ha v e historically b een delineated into too ls for learning and to ols for doing s tatistics (Baglin, 2013; McNamara, 2015). In terestingly , while statisticians ha v e thought and written a b out pr inciples underlying the to ols for learning statistics, al- most no critical work has b een done to ev alua t e professional to ols for doing statistics. In the educational contex t, R olf Biehler’s 1997 pap er Softw are fo r Learning and for Doing Statistics outlined principles for a statistical computing to ol that w ould supp ort no vices in learning statistics and da t a a nalysis (Biehler, 199 7 ). It provided a framew ork for the assessme n t of statistical education softw are, and motiv a t ed the dev elopmen t of new to ols. The motiv ation and criticism of professional to ols is m uc h less ric h, and tends t o fo cus on language prop erties (Ihak a and Gen tleman , 1996; Mora nda t et a l., 20 1 2). T o day , the lines b et w een educational and professional to ols are starting to blur, a nd w e b elieve this lo w ers the barrier to en try for statistical computing. This pap er presen ts a list of attributes capturing features needed f or to ols for b o t h no vices and professionals. The attr ibutes aim to b e as broad a s p ossible, but t hey are fo cused sp ecifically o n the dev elopment of more supp ortiv e en vironmen ts. Ho p efully , this list can b e used t o frame critical discussions of s tatistical computing tools of all t yp es. The target audience of this pap er are soft w are deve lop ers who are considering the dev elopmen t and improv emen t of statistical computing to ols. Practicing educators may also find the pap er in teresting, as it fo rms a scaffolding for ev aluating existing to o ls and decidin g whic h 2 to use in a particular course. And, statisticians could do to think mor e critically about the to ols they use and help create. Although the attributes ha v e b een designed to b e applicable to a broad range of users, it is useful to fo cus our discussion on “a user” rather than “ t he user” (Agre, 1995). F or purp oses of discussion, w e consider our t a rget user to b e a journa list lo oking to bring more computation into their w ork. The practice of data journalism has b een accelerating, but jo urna lism sc ho ols ha v e b een slo w to mo dify the curriculum to teac h computational skills (Berret and Phillips , 2016), As a result, man y journalists ha v e limited experience with programming and statistics, but w an t to tell da t a -driv en stories. They need to mo v e from nov ices to pro ducers v ery quick ly . A ke y goa l of journalism is comm unication, and as news publications hav e em braced the in teractiv e w eb, j o urnalists are at on the forefront of publishing mo dern, data - ric h rep or t s. Considering a data jo urna list as our target user means prio r itizing to ols t ha t ar e easy to learn but also p ow erful and flexible. W e could hav e fo cused just a s e asily on a n um b er o f other s p ecific us ers. F or example, a graduate studen t in a scien tific or so cial sc ien tific field wh o ne eds to use statistical me tho d- ology to complete their w ork. W e ma y imagine those graduate studen ts are already us ing a statistical computing to ol, but man y use Excel (W eiss, 20 17), and if they need t o get up to sp eed with another pac k age they ar e often self-taught (Lo wndes et al., 20 17). Once again, these users need to ols that are easy to use, a nd b ecome ‘invis ible,’ allow ing them to get their w ork done. Because of the increasing imp ortance of repro ducibilit y in science, they also need to ols tha t allow them to do cumen t their w ork. F or someone who uses a statistical computing pack age ev ery day , it may b e hard to imagine b eing new to it , or conceptualize ho w the to ol could b e impro v ed. Ho w ev er, in order to broaden the use and understanding of statistics, w e need to mak e it easier to do statistics, and t o do statistics w ell. Man y of the ideas presen ted here are not new. In particular, these attributes attempt to distill principles and c haracteristics prop osed b y Rolf Bieher, Alexander Rep enning, and John T uke y (Biehler, 1997; Rep enning et a l., 2 010; T uk ey, 1965), while also considering the recen t dev elopmen ts in data and computing. John T uk ey w a s considering the “tec hnical to ols of statistics” in 1965, a nd describing a vision f or the future of statistical programming to ols, including ‘More of the essen t ia l erector-set ch aracter of data-ana lysis tec hniques, 3 in whic h a kit of pieces are av aila ble f or assem bly into any of a m ultitude of analytical sc hemes” (T ukey, 1 965). Thirt y y ears later, Rolf Biehler defined three primary problems, the “complexit y of to ol problem” (existing to ols w ere to o hard to for no vices to learn), the “closed micro w orld problem“ (learning t o ols w ere designed f or one pa r t icular t yp e of problem or da ta set and couldn’t b e extended) and the “v ariet y problem” (b ecause of the closed microw orld problem, it w as necessary to use man y t o ols to do ev erything an instruc- tor w an ted t o cov er) (Biehler, 1997). Most recen tly , Alexander Rep enning, Da vid W ebb, and Andri Ioannidou outlined the six requiremen ts for a “computational thinking to ol,” including having a low thr eshold, a high ceiling, and b eing equitable ( R ep enning et a l., 2010). A surv ey of statistical computing t o ols (McNamara , 2016) helps ground these ideas in the existing computational landscap e. Again, since statistical computing to ols ha v e of t en b een delineated in to professional and educational to ols, w e tak e represen tativ e examples from each ‘category’ when w e refer t o existing soft w are. When to ols for learning statistics are men tioned in this pap er, the most common examples will b e Tink erPlots and F athom, t w o interactiv e to ols for statistics education (K onold a nd Miller, 2005; Finzer, 2002). These to ols are graphical, drag- a nd-drop interfaces that mak e analysis highly visual. Most refer- ences to professional to ols will b e to the prog r amming la ng ua ge R (R Core T eam, 201 6) or to SAS, Stata, a nd SPSS. Considering the v arious p ositive qualities of current to ols for doing and teaching statis- tics alongside Biehler’s g oals (Biehler, 1 997) and com bining them with ideas from Rep enning et al. (2010) and T uk ey (1965), w e dev elop ed a list o f 10 attributes fo r a mo dern statistical com- puting to o l. These are summarized in T able 1. While these attributes attempt to b e exhaustiv e, they are also designed as a springb oard for discussion. Because there has b een relativ ely little recen t critique give n to statistical computing to ols, this list is an attempt to star t the con v ersation. Eac h requiremen t will b e discussed in more detail in its resp ectiv e section. 4 1. Accessibilit y 2. Easy en try fo r novice users 3. D a ta as a first-order p ersisten t ob ject 4. Supp o rt for a cycle of exploratory and confirmato r y analysis 5. Flexible plot creation 6. Supp o rt for randomization throughout 7. Interactiv it y at ev ery lev el 8. Inheren t do cumen tatio n 9. Simple suppo rt for narrative, publishing, and repro ducibilit y 10. F lexibility to build extensions T able 1: Summary of attributes 5 1 Accessi bilit y T o ols should alwa ys b e accessible, particularly to new users. As a baseline requiremen t, soft w are should b e affor dable (o r free), work with a v ariet y of op erating systems, and b e easy to install (Dunham and Henessy, 2008; Rep enning et al., 2 0 10). In this context, most to ols f o r teac hing a r e accessible, b ecause they are designed to work across platforms and are priced inexp ensiv ely . Ho w eve r, professional to ols tend to b e exp ensiv e and ina ccessible for non-professional o r o ccasional users. They are not accessible for small new spap ers, nonprofits, individuals, o r K- 12 sc ho ol systems. Users in these contex ts mus t consider if they hav e the funding for a to ol, if it will w o r k on the computers they ha v e access to, and if they ha v e the user privileges to install softw are. System administrators can b e few and far b et w een in newsro oms and underfunded sc ho ol systems . One wa y to ensure accessibilit y is to create a w eb- ba sed to ol—cloud services allo w users to access soft w are from an y machine with in ternet access. Bey ond the accessibilit y of a to ol to the masses, it is also imp ortan t to consider the needs of p eople with disabilities. F o r a to ol to b e required in public sc ho ols, it must b e compatible with a ccessibilit y features on mo dern computers (Office of the Chief Informatio n Officer, 2001). Some pro gress has been made o n programming la ng uages a ccessible for blind users (Stefik et al., 2011; Go dfrey, 2013), but giv en that man y educational to o ls a re vi- sual, it is not clear if an y of them are accessible to blind users. Of course, there are other disabilities that can impact a p erson’s ability to use a to ol. Considering “ univ ersal design” (a principle of designing things to b e usable by all p eople) (Connell et al., 199 7 ) should b e an a sp ect of the dev elopmen t of any new statistical computing to ol. 2 Easy En try F or No vic e Users T o ols to b e used by no vices—and really , all to ols—should mak e it easy to get started. This attribute comes directly fr o m Repp enning et al’s w ork on to ols for computational thinking (Rep enning et al., 20 10). It should b e clear what the to ol do es, ho w to use it, and what the most salien t comp o nen ts are. The to ol should pro vide immediate gratification, rather tha n a p erio d of frustrat ion ev en tually leading to success. 6 Easy entry means users should b e able to jump directly in to ‘doing’ data analysis without ha ving to think ab o ut min utiae. No vices should to b e able to b egin exploratory data analysis within the first 10- 1 5 minutes of using a to ol. Biehler a rgued, “In secondary education, but a lso in in tro ductory statistics in higher education, using a command la nguage system is problematical. W e are convinc ed that a host system with a graphical user interface o ffers a more adequate basis” ( Biehler , 1997). By Biehler’s estimation, a system providing easy en try fo r nov ices will lik ely ha v e a visual comp onen t, either initially or througho ut. Indeed, visual to ols lik e Tink erPlots and F athom allo w no vices to create link ed plots and m ultiv ariate dat a visualizations within the first minute of b eginning the soft w are. Curriculum dev elopmen t using t he programming languag e R has b egun to put first emphasis on exploratory data analysis, rather than data structures (Pruim et al., 20 1 4), so these goals can also b e achiev ed in a scripting context. Giv en the success of the blo c ks-based language Scratc h in computer science education (Resnic k et al., 2009), it seems p ossible a graphical system w ould b e b etter for no vices. Ho w eve r, there are many other w a ys easy en tr y could b e achie v ed, suc h as the use of la nguage lev els (Hsia et al., 2 005), or a ccessible IDEs (K¨ olling, 2010). 3 Data as a Firs t-Order P ersistent Ob ject A dat a a nalysis to ol must necessarily deal with dat a . A to ol cannot b e considered to b e designed for statistical computing if it do es not make data its primary ob j ect of in terest. The wa y data are for ma t t ed and represen ted within the system is also of crucial imp ortance. In this con text, formatting and represen tat io n refer sp ecifically to ho w the data app ear to the user, no t ho w they are stored within the computer’s memory system. Mo dern data analysis to ols should mak e it easy to access common data t ypes (flat files, hierarc hical data fo rmats, APIs, etc.) and ‘see’ t he f ull data (whether in a f o rmat allo wing for v alue-reading or a higher-lev el view). Da t a should b e a p ersisten t ob ject, with a repro ducible w orkflo w of wrangling to take raw data to clean. 7 3.1 Viewing data Man y to ols (including spreadsheets) mak e a view o f the data the primary f o cus. In con v er- sations with data journalists, they often men tion scrolling through a spreadsheet, ‘reading’ the data v alues as their first line of inquiry . Scien tists also lik e to visually lo ok through their dat a when they b egin. While there are few recen t studies to supp ort this, an early exp eriment on Lotus 1-2- 3 suggested users sp end around 42% of their time viewing in- dividual cells (Bro wn and Gould, 1986). In con trast, pro gramming languages lik e R and Python hav e tra ditionally not shown data to users when it is read in, instead requiring the use o f function calls to view the first few row s of data. This can b e a stic king p o in t for users tra nsitioning from spreadsheet programs, so In tegrated Dev elopmen t En vironmen ts pro viding a data preview ha v e b ecome p opular (R Studio T eam, 2014). Whether prov ided b y default or requested b y the user, most current t o ols provide data suc h that users can immediately read eac h individual v alue. Ho w ev er, there are other w a ys to ‘see’ an en tire dat a set. F or example, Victor Po w ell’s CSV F ing erprin t creates a colored image as a high- lev el ov erview of t he data (P o w ell , 20 1 4). Colors indicate data types (to see whether it is mostly numeric , catego r ical, in teger, etc) a nd missing data (Po w ell, 201 4). This simple visualization giv es a lot of insigh t, and suggests that there may b e other visual metaphors to represen t r aw data that could b e equally helpful. The more a user can glean from a n initial glance at their dat a , the easier it is f or t hem to b egin to dig in to it. 3.2 Rectangular v ersus hierarc hical data Analysts are typically accustomed to thinking of data in a t idy rectangular format, com- p osed o f rows and columns or observ ations and v ariables. Rectangular data can b e consid- ered ‘tidy’ if ev ery row represen ts one case (e.g., a p erson, g ene expression, o r exp eriment), and ev ery column represen ts a v ariable (i.e., something measured or recorded ab o ut the case) (Wic kham , 2014a). Tidy data are o ften visualized as a spreadsheet, and spreadsheets are the most common w a y p eople around the w orld in teract with data (Bryan, 2 016). In t erestingly , no vices who ha v e no t encountere d data b efore often do not use rectan- gular fo r ma t s to r epresen t their data, but rather default to a list- ba sed or hierarc hical format (Lehrer and Sc hauble, 200 7; Finzer, 2013, 2 014). So, although rectangular data 8 has b ecome prev alen t, it may not b e the most natura l format. There are hierarchical and list-based da t a fo rmats like JSON and XML which are used commonly on the w eb. These t ypes of data a re imp or tan t for data science (Nolan and T emple Lang, 2014). Ho w ev er, they may require the dev elopmen t o f new visual metaphors b ecause the tidy rectangle will no longer suffice. How can you see a clear ov erview of an en tire hierarc hical dataset? One observ a tion may stretc h down the screen, and Po w ell’s colo red o v erview certainly do es not directly translate. 4 Supp ort for a Cycle o f Explorator y and Confirma- tory Analysis Statistical computing to ols should promote explorator y analysis, and its twin, confirma- tory analysis. The complemen tary explorato ry and confirmato ry cycles were suggested b y John T uk ey in his 1977 b o ok, and hav e b een re-emphasized b y curren t educators (T uk ey , 1977; W eisb erg, 2005; Biehler et al., 2013). The use of t he term ‘cycle’ indicates how it - erativ e the data a nalysis pro cess is. Each step can lead back to prior steps. The cycle can include generating statistical questions, collecting data, ana lyzing data, and in terpret- ing results (Carver et a l., 20 1 6). Ha dley Wick ham lists imp ort , tidy , transform, visualize, mo del, comm unicate (Wic kham , 20 1 4b). In a p edago gical setting, educators often talk ab out t he PPDA C cycle: Problem, Plan, Data , Analysis, Conclusions, t ypically attr ibuted to Wild and Pfannkuc h (1999). If users find something in teresting in a cycle of exploratory a na lysis, they need t o fol- lo w with confirmatory analysis. The idea of ED A is to explore data deeply b y computing descriptiv e statistics and making man y graphs—of one v aria ble or sev eral– to gain an understanding of the underlying structure. Altho ug h EDA can app ear sub jectiv e, it some- times comprises the b est and riche st metho d for analysis, part icularly for finding patterns in data and p erforming info rmal inference (Mak ar and Rubin, 201 4; Rubin et al., 2 0 06). Exploratory data analysis can also b e used in the con text of statistical mo deling (Gelman, 2004). The difference b etw een exploratory and confirmatory analysis (or inf o rmal and formal 9 inference) is lik e the difference b etw een sk etc hing or taking notes and the act of creating the final pain ting or writing an essa y . One is more creative a nd expansiv e, and the other tries to pin down the particular information to b e highlig h ted in the final pro duct. A system supp orting exploration and confirmation should provide a w o rkflo w connecting these tw o t ypes o f a ctivities. Users need ‘scratc h pap er’—a place to play without the r esults b eing set in stone. While data analysis needs to lea v e a clear trail o f what w as done so someone else can repro duce it, a scratch pap er env ironmen t migh t allow a user to p erfor m a ctions not ‘allo w ed’ in the final pro duct, like mo ving data p oin ts. Biehler called this capa bility ‘draft results’ (Biehler, 19 9 7). Man y curren t systems for teaching statistics pr ovide rapid exploration and prototypin g (allo wing users to manipulate data o r pla y with g r aphic represen tations), but typically do not supp ort the more f o rmal final analysis. In con trast, professional to o ls tend to mak e it difficult to play with data (in R , creating m ultiple g r aphs takes effort, as do es mo difying parameter v alues), and they ma y not supp ort cyclical exploration or rapid plot g eneration. Again, this is limiting, as a sense of pla y and disco v ery is importa n t to da t a analysis. Data scien tists rep eatedly cycle bac k thro ugh questioning, exploration, and confirmation or inference, so analysis is neve r a linear pro cess f rom b eginning to end. A statistical computation to o l should supp or t this cyclical pro cess. 5 Flexible Plot Cr e ation T o fully supp ort data ana lysis (b oth exploratory and confirmatory), a to ol needs to em- phasize plot ting. Computational to o ls make it p ossible t o visually explore lar g e data sets in w ays that would b e difficult or imp o ssible using j ust p encil a nd pap er. Visualization greats John T uke y and Jacques Bertin b oth dev elop ed visualization metho ds for summarizing and visualizing patterns in data b efo r e computer graphics (T uk ey, 1977; Bertin, 1983 ). These stat ic plots are still useful no w that computers can generate them, but a statistical computing to ol should give h umans abilities b eyond what they could ac hiev e with p encil and pap er. An exemplary metho d is the G rand T our, whic h t a k es high-dimensional data and pro duces pro jections in to a v ariet y o f 2- and 3-D spaces, w alking a user through many views of their data to expose clusters and t r ends ( Buj a a nd Asimo v , 1986). A simpler 10 example that can also pro vide insigh t is the generalized pairs plot, whic h displays all 2 - v ariable relationships in the data (Emerson et al., 201 3). These plots allow humans to lo ok for patterns in higher dimensions than they could o rdinarily conceptualize. Pro viding easy plotting functionalit y of many v ariables should b e a goal of ev ery to ol, whether for learning or for do ing statistics. T o o ls, particularly those for no vices, m ust c ho ose whether to provid e a few simple plotting functions o r the a bilit y to fully customize graphics. While it can seem simpler to prov ide a small set of standard data visualizations, creating visualizations fro m primitive s b o t h pro vides more flexibility for the user and rein- forces the mapping b et ween a bstract data and visual aesthetics on the screen (W eisb erg, 2005; Wilkinson , 2005; Wick ham , 2009). Ideally , a statistical programming to ol would mak e it simple to b egin plotting (to facilitate EDA) and to pro duce standard graphics, while also allo wing users to create no v el plot t ypes. 6 Supp ort for Randomizatio n Througho ut Computers hav e made it p ossible to use randomization and b o ot stra p metho ds where appro ximating fo rm ulas w ould once hav e b een the only recourse. These metho ds are not only more flexible than traditiona l statistical tests, but can also b e more in tuitiv e fo r novic es to understand (Pfannkuc h et al., 2 0 14; Tin tle et a l., 2012). Randomization and sim ulation can help make inference from data, even if those data are from small sample sizes or no n- random collection metho ds (Efron and Tibshirani, 1986; Lunneb org, 199 9; Ernst , 2004). Randomization a nd the b o otstrap can also b e used t o v a lida te mo dels ( Ma jumder et al., 2013; Buja et al., 2009; Gelman , 2 004), pro vide a visual represen tation of uncertain t y in a plot (Hullman et a l., 2015), or p erfo rm graphical inference, a metho d of assigning signifi- cance to plo t s b y using a series of randomized plots to pro vide a “n ull” visualizations to com- pare true visualizations against (Wic kham et al., 2 010; Ma j umder et al., 2013; Buja et a l., 2009). These metho ds ha v e b een ga ining p opularity in statistics researc h and tric kling dow n to the educational con text as w ell. Sev eral p opular introductory stat istics textb o oks fo cus on randomization and simulation metho ds (Diez et al., 2 0 14; Tin tle et al., 20 14; Lo ck et al., 2012), and o ther resources help get instructors up to speed (Hesterberg , 2015). These 11 materials av oid the issue that man y in tro ductory statistics courses fall into, where the course can b egin to feel lik e a g rab-bag o f metho ds. Instead, they sho w randomization as a unifying metho d to answ er many statistical questions using o ne framew o rk. The application of randomization a nd the b o otstrap is a place where to ols for teac hing statistics shine. P opular applet collections pro vide simple ra ndo mizatio n and b o otstrap functionalit y (Chance and R ossman, 2006 ; Lo ck Morgan et al., 2014). Tink erPlots and F athom also prov ide in tuitiv e visual in terfaces for this (Finzer, 2002; Konold and Miller , 2005). How ev er, professional to o ls ha v e lagg ed behind. R provides the most complete functionalit y , but it is no t a lwa ys simple to use. Because of their in tuitiv e nat ur e and generalizability , randomization and b o otstrap metho ds can b e helpful for novic es and exp erts alik e. They can b e used in a v ar iety of con texts, including g raphical inference metho ds bridging the gap b etw een exploratory a nd confirmatory analysis. 7 In te ractivit y at Ev ery Le vel In t eractiv e systems enable users to b e mor e engaged and pla yful with data. Rather than t yping commands, users should b e able to in teract with their data. And the more direct the manipulation, the b etter. This means v aluing pinc h-zo om ov er a drop down men u with an o ption for zo o m, clic k-and-drag selection ov er a form allo wing the user to enter filtering v alues, and link ed plots a nd analysis ov er a set o f disconnected pro ducts. Here, pro ducts encompasses any thing that comes out of the analysis, including plots, mo del output, and summary statistics. In t eractivit y is b ecoming standard on the w eb. Users of G o ogle maps know they can pan and zo om a map, and Apple has strong opinions on whic h direction is more ‘na t ural’ to scroll. O n smartphones w e launc h angry birds, drop pins on our lo cation, and swip e left to r eject a date. Data analysis platforms need to fo llo w suit. F or nov ices, w e w an t to “T each ab out, and with, interactiv e graphics” (R idg ew ay, 2015) so they b ecome a dept at seeing da t a in this w ay . As Biehler suggests, w e w a n t to encourage direct manipulation rather than mo difying a script ( Biehler, 1 997). T o day , educational to ols pro vide this type o f direct ma nipulat io n, 12 but professional pro gramming to o ls of t en do not. Ho w eve r, eve n textual programs can shorten the time b et w een making a change in the co de and seeing t he results. Computer futurist Bret Victor has made shortening this lo op one of his driving design principles, to pro vide users with the abilit y to see the direct results of their actions without waiting for something to compile (Victor, 2012). The dev elopmen t o f d3.express sho ws pro mise in bringing this paradigm to the visualization libra r y d3 (Bosto ck, 2017). In the con text of statistical prog ramming, D eb orah Nolan and D uncan T emple Lang mak e the distinction b etw een dynamic do cumen ts (those that are compiled and t hen auto- matically include the results of em b edded co de), and in teractiv e do cuments (those that let a reader in teract with comp onen ts lik e graphics) (Nola n and T emple La ng , 2 007). Giv en the go als of in teractivit y at ev ery lev el, and the imp o rtance of publishing, a mo dern sta- tistical programming t o ol should provide ‘dynamic-in teractiv e’ graphics, where users can in teract with a n y comp onen t of the do cumen t and hav e the results up date in real time. In t eractivit y can tak e place at three lev els. The fir st is in the con text of dev eloping an analysis. Ideally , users should b e able to build their analysis in teractiv ely . Men us and wizards are a t yp e of ‘in teraction,’ but are not direct in teraction and don’t add any intuition ab out t he pro cess. Instead, a to ol should aim to allow fo r the most direct manipulation p ossible. The second leve l is within the analysis session, where all results should themselv es b e in teractiv e. The to ol should supp ort g raphs as an inte rface to the data (Biehler, 1997). Beha viors lik e brushing and linking should do dynamic subsetting ( Wilkinson, 2005; F ew, 2010). All g r aphs should b e zo omable, supp o rt brushing and linking, and allow f o r simple to oltips t o identify data p oints. It should b e easy to change the data cleaning metho ds and see how that c hange is reflected in the analysis afterward, and par a meters should b e easily manipulable. The system should a lso make it p ossible to see m ultiple co ordinated views o f ev erything in the user’s en vironmen t. The imp ortance of a co ordinated view is supp orted b y researc hers who suggest a llo wing f o r m ultiple views of the same data ma y help p eople gain a mor e in tuitive understanding (Shah and Ho effner, 2002; Bakk er, 2002). Finally , the finished data pro duct should b e interactiv e. This means that the audience of a piece of data analysis—ev en if they do not know m uc h ab out statistics—could play 13 with the parameters and con vince themselv es the data w ere not do ctored. As may b e exp ected, standalone educational to ols do a b etter job of pro viding interac- tivit y than professional to ols. Tink erPlots and F athom a re highly interactiv e, allowing users to drag-and- drop v ari- ables on to their plo t s and supp o rting brushing and linking b etw een plots. Highlighting cases in t he data table highlights them in ev ery plot. These to ols mak e it easy to in ter- activ ely dev elop analysis and play with it, but do not supp ort sharing in teractiv e results with someone who do es not hav e the softw are. On the o ther hand, in teraction has historically b een mor e c hallenging in professional to ols. The history of statistical computing traces bac k to the pre-gr a phics era of computers, so most systems rely on static co de. This paradigm means users are not incen tivized to return to the b eginning of their analysis to see ho w a co de mo dification w o uld tric kle do wn. If a pro grammer wan ts to adjust a parameter v alue in their co de, they must mo dify the co de and re-run it, making the comparison b et w een states in their head. Comparing t w o states in this w a y may b e p ossible, but comparing more than tw o is difficult. This is a cognitiv e burden w e no longer need to put on users (Victor, 2 012). If results were immediately accessible, it would ma ke it p ossible to mak e hu ndreds o f comparisons in just a f ew seconds. In recen t ye ars, some of these p ossibilities ha v e b egun to emerge. ‘Noteb o ok’ f unctional- it y in sev eral en vironmen ts allows users to execute co de c h unks directly within t heir source file ( Perez and Grang er, 2015; RStudio T eam, 2016). F or exp erienced progra mmers, the pro duction of interactiv e do cumen ts that resp ond to user input is p ossible (Chang et al., 2015; Bosto ck, 2013; Saty anaray an et al., 2016). While these pac k ages allow expert users to create dynamic graphics, they are to o complicated for a b eginner. As a result, most curren t published work with interactiv e abilities is the result of a b esp ok e pro cess. Because few to ols exist to fa cilitate the dev elopment of fully in teractiv e data pro ducts, p eople who w an t to g enerate suc h pro ducts must hard-co de them fo r a particular application. Tw o exemplary pieces of journalism include a simulation-based lo ok at h urricane impacts in Houston by ProPublica, whic h allow s readers to manipulate parameters o f the sim ulation (Satija et al., 2016), and the IEEE pro gramming language 14 ratings (Cass et al., 2014) whic h provides access to the weigh t parameters used for each data source in the rating alg o rithm. The p o w er a nd usefulness of a truly interactiv e dat a ana lysis platfo r m is easy to imag- ine. If all par a meters w ere adjustable, it would b e easier to get an in tuitiv e sense of the parameter space, and therefore the fragility of a particular piece o f a n analysis. 8 Inherent D o cumen t ation Systems should provide inherent do cumen ta tion, so computing to ols “highlight the logic of what is going o n” (K a plan, 2007). Most prog r amming langua ge do cumen tatio n is hard for novic es to comprehend, so w e first w a nt help that is helpful. How ev er, the idea o f a inheren t do cumen tation go es one step further, to help that is integrated into the pro cess of using a to ol. Instead of ha ving to go to a second place to learn what a feature is or what a f unction do es, ob jects should pro vide do cumen tatio n as a unified part o f themselv es. Ideally , ev ery comp o nen t of a system should visually sho w the user what it is going to do, v ersus just telling them. How ev er, ev en in textual languages inheren t do cumen tation can b e achie v ed by bringing the syn tax of the languag e more in line with human langua g e. F unction names tha t describ e what they do are more v aluable than those that preserv e k eystrok es. Supp o rtiv e f eatures like tab completion can mak e do cumen tation o f parameters more inheren t to the analysis pro cess. F or example, if a to o l is going to p erform k-means clustering, the basic lev el of do c- umen tat io n should b e the words “k-means.” Ideally , the user should see a visual repre- sen tatio n o f the algorithm, and as it is applied to the data, in terim steps should b e visu- alized (M ¨ uhlbac her et al., 20 14). Of course, using a computer is not the same a s moving through the real w orld, so inte rface designers m ust think carefully ab out visual metaphors that mak e the most sense. Sometimes, this means mimick ing t he real world (as in the desk- top metaphor, with fo lder icons and a trash can) and sometimes dev eloping a new visual language (as may need to happ en for visualization o f mo dels, database op erat io ns, and the lik e). In teractiv e con trols of a system should giv e some idea o f what they a r e going t o do, either b y their design or by the presen tation of ‘scen t ed widgets’, embedded visualizations pro viding hints t o users ab out what elemen ts are capa ble o f ( Pousman et al., 20 07). 15 9 Supp ort for n arrativ e, publis hing, and repro du cibil- it y One imp orta nt comp onen t o f data science is the comm unicatio n o f results. W e hav e alr eady considered the imp ort a nce o f flexible plot creation, whic h is a form of visual comm unication. In addition to plots, almost all data analytic pr o ducts require some form o f narrativ e to accompany the w ork and con textualize it for readers. The pro ducts o f a statistical computing system should b e as easy to understand as the pr o cess of creating them, and they should b e simple to share with others. In tegrated narra t ive and button-clic k publishing will pr ovide a ffordances that suppo rt repro ducibilit y . Repro ducible, in teractiv e w orkflo ws ma y help to build confidence in r esults b ecause they can b e easily v erified ev en b y non- exp erts. 9.1 Narrativ e Historically , analysis w o r kflo ws ha v e tended tow ard a paradigm o f doing ana lysis in one do cumen t a nd narra tiv e in another. Programmers traditionally separate the do cumen tation of their co de from the co de itself (co de commen ts no twithstanding). Data analysts often create their data analysis co de first, then go back to create a narrative surrounding the analysis. Data journalists refer to the pro cess of p erforming analysis in Excel and writing ab out the results in W ord as ke eping a ‘da t a dia r y .’ In contrast, a statistical programming to ol should hav e afforda nces to encourage nar- rativ e alo ng side or mixed in with the co de to facilitate the in tegration of storytelling and statistical pro ducts. Donald Knuth calls this ‘literate progra mming’ b ecause it is easier for h umans to read and understand (Knuth, 19 8 4). Curren tly , the most successful to ols allo w users to write f ormatted text and delimited co de, then pro cess the do cumen t to create a final pro duct with text, co de, and co de out- put (P erez and G ranger, 2015; Xie, 201 4). Ev en those to ols leav e something to b e desired. They feel constrained, and do not lend themselv es to t he t yp e of expressiv e w o r k that c ha racterizes data science. Delimiting co de c h unks is a f airly ligh t w eight pro cess, but it do es require some additional syn tax. And including inciden tal num b ers into narrative sen- 16 tences can b e tricky . A b etter solution would allow f o r explicit linking b etw een co de c h unks (or, automatic detection of reactiv e connections), a nd the ability to drop an y piece of an analysis in to the t ext. 9.2 Publishing Ideally , data a nalysis results and related pr o ducts could b e published with ease. Jour na l- ists could create a data- driv en w ebsite, citizen scien tists could share insights in the data they help ed create with their friends and family , and p eople w o rking tog ether across an organization (o r across the glob e) could sta y up-to-da t e on their collab orators’ contribu- tions. In all these scenarios, the publishing format should allow for exploration (discussed in more depth in Section 7). In f act, t he ideal case w ould b e a finished pro duct allow ing for full access to a ll the computation in the analysis. In this w a y , users could con tin ue to explore the data, mo dify t he a nalysis, and see the effects of their c hanges on the analysis and visualizations. As the exp ected user base f or analysis publication is wide (encompassing b oth no vices and exp erts) the language the analysis is written in should b e the same as the language it is published in. Curren tly , it is o ften necessary t o translate fr o m one fo rmat to ano ther to share analysis. F or example, a data journa list using RMarkdo wn to do cumen t t heir analysis will need to format it after the fact using their newspaper’s conte n t mana gemen t system. T o ach iev e the goa l of na t ive publishing, it is lik ely new link age pip elines will need to b e dev elop ed in or der to streamline these transitions. In dat a journalism, simple publishing abilities for fully interactiv e results of a data analysis could emp ow er journa lists to pro duce ric her articles. Suc h articles could b e ac- companied b y the repro ducible co de that pro duced them, a llo wing readers to a udit the story . Similarly , as repro ducibilit y b ecomes more v alued in the a cademic communit y , data pro ducts a re more o ften accompanied with fully repro ducible co de. If the co de w ere inter- activ e, it w ould widen the p o ten tial audience of the academic w o r k. 17 9.3 Repro duc ibilit y Repro ducibilit y supp orts the aims of science, and should therefore b e in tegrated with the w o rk of data science ( Buckhe it and Donoho, 1995; Sandv e et al., 201 3; Ince et al., 20 12; De Leeuw, 2009). T eaching no vices to use to ols that supp ort repro ducibilit y can help ensure it b ecomes an in tegral part o f their statistical and data w orkflo w ( Carver et al., 2016). There are many definitions of repro ducibilit y . Here, w e tak e a somewhat narrow view. A repro ducible ana lysis is one that can b e re-r un (p oten tially years later, or b y a differen t p erson) with the same data to pro duce exactly the same result. A slight extension to this is an analysis that can b e re-run with a mo dified vers ion o f t he original data to get analogous results (K andel et al., 20 11; Sandve et al., 20 13; Broman, 20 1 5). F or example, t he initial analysis w as do ne o n 2016 data but needed to b e run again on 2017 data, or the initial analysis used corrupted data that should b e replaced b y a corrected v ersion. It ma y sound simple to ac hiev e this goal. How ev er, in practice there are man y factors that make it c hallenging. Sof tw are v ersions can c hange, pack age dep endencies can get brok en, and—most disruptiv e to the pro cess—authors often do not manage to do cumen t their entire pro cess. They ma y ha v e do ne data cleaning o ut side the main softw are pack age (e.g., the bulk of the analysis was done in R but t he author did data cleaning in Excel b efore the analysis), or run a nalysis steps without adding them to the co de do cument. They may pro vide out-of- date co de, or co de with bugs that need to b e addressed b efore it will run. These pro blems can b e at least part ia lly addressed with to oling. In t egrated narrative a nd simple publishing will necessarily encourage repro ducibilit y . If analysis deve lop ers ar e writing nar rativ e as they write co de, the r esults will b e easier t o in terpret and more like ly t o b e housed in the same place. If it is easy to publish this t ype of do cumen t, readers will ha v e access to a ric her v ersion of the analysis tha n is t ypically shared. Therefore, the pro ducts of statistical computing to ols should contin ue to b ecome more repro ducible. Ho w ev er, there is more work to b e done b efore any statistical computing to ol can b e said to fully supp ort the entire sp ectrum of repro ducibilit y . A f undamental f eat ur e supp orting repro ducibility is the ability to sa v e the data analysis 18 pro cess. Some teac hing to ols ( e.g., applets) do no t allow state to b e sa v ed in an y wa y . In other systems, lik e F athom and Excel, analysis is not repro ducible b ecause it w as pro duced in teractiv ely . Ev en in 1997, Ro lf Biehler was aw are o f this drawbac k to in teractiv e systems; “It ma y b e considered a w eakness of systems like Data D esk that the link age structure is not explicitly do cumen ted as it is the case with explicit pro gramming or if w e ha d written the list of commands in an editor. An impro v emen t w ould b e if a list of commands or another represen tation of the link age structure would b e generated aut o matically” (Biehler , 19 97). Most in teractiv e to ols do allow the user to sav e the environme n t that pro duced the pro duct, but do not do cumen t the steps t ak en within the en vironmen t. An indep enden t researc her could use the sav ed do cumen t to explore the analysis, but may not b e able to discov er the steps to pro duce the final pro duct. These types of to ols also mak e it imp ossible to re-run the analysis on sligh tly differen t data. Again, professional to ols allo wing for the in tegration of narrative a nd co de are b eginning to supp ort some of these goa ls. Using R and R Markdo wn, for example, users can now author en tir e analyses within a single do cument, fulfilling Broman’s ‘ev erything with a script’ and ‘turn scripts into repro ducible rep orts’ (Bro man, 2015; Xie, 20 14). Some of these to o ls are simple enough to b e integrated in in tro ductory college statistics courses (Baumer et al., 2014). Ho w eve r, ev en exp erts trying to implemen t repro ducible w orkflo ws hav e found it difficult to fully do cumen t their pro cess (FitzJohn et al., 2014; Garijo et al., 201 3). F or no vices, full repro ducibilit y is ev en more ch allenging (Garij o et al., 20 1 3). F uture systems should therefore b e designed in order to supp ort repro ducibilit y more fully . This ma y entail sa ving a v ersion of the computer’s state, trac king all ‘scratc h w ork’ alongside co de put into a ‘final draft,’ auto mat ically recognizing dep endencies on files, pac k ag es, and custom functions, and pro viding a visual represen tation of those dep endencies to t he user. This vision w ould mo v e close to Nolan and T emple Lang’s vision of dynamic, in teractiv e do cumen t s (Nolan and T emple La ng, 2 007). 10 Flexibility T o Buil d Extensio n s Of course, a statistical computing t o ol m ust hav e stat istical metho ds built in to it. While these attributes hav e outlined elemen ts tha t approach metho ds (suc h a s graphics and ran- 19 domization) they shy a w ay from sp ecifying an y pa rticular mo dels or tec hniques. This is b ecause statistics is alw a ys c hanging, so one of the most imp ortant attributes of a statistical computing to ol is the ability to extend it. The flexibilit y to build extensions is necessary in order to prev en t a to ol from b ecom- ing obsolete. Users m ust b e able to create new comp onen t s of the system as metho ds are dev elop ed, computers improv e, or scien tific disco v eries are made. T o b e a computational thinking to ol, building extensions is a required feature suc h that the system has a “high ceil- ing,” prev enting users from ‘a ging out’ or ‘exp eriencing out’ of a system (Rep enning et al., 2010). In a statistical computing to ol, it should b e p o ssible to dev elop new visualization t ypes and dat a pro cesses from other mo dular pieces. Professional to ols can b e lo ok ed to for inspiration, b ecause they tend make it easier to create new comp onen ts of t he system using old ones. R eve n ha s a cen tralized rep ository where other users can easily find and imp o rt others’ w ork (R Core T eam, 2015). Curren tly , the to ols easiest fo r no vices to use fail to pro vide a high ceiling, although Biehler argued that “adaptability (including extensibilit y) is a cen tr a l requiremen t for data analysis systems to cop e with the v ariety of needs and users” (Biehler , 1 997). An y system hoping to stay the test of time m ust pro vide the flexibility to build exten- sions. 11 CONCLUSION This list of 10 attributes aims to encompass the most imp ortan t qualities f o r a mo dern statistical computing to ol. W e hav e fo cused on an idealized data jo urnalist as our tar get user, but hop efully the attributes ar e more broadly relev an t, encompassing some o f the needs of science and so cial-science graduate studen ts, novices at a v ariet y of other ag es, and seasoned statistics professionals. Of course, there are other features that o ne might desire for their to ols. The list fo cuses on things that could b e built in to a system b y a n engineer, whic h ov erlo oks t he imp o rtance of a welc oming and supp ortive comm unit y of users. It also has not touc hed on the la ng uage attributes commonly cited by computer scien tists, suc h as sp eed and completeness, and it assumes to ols w o uld b e stable and free of errors. Do es the ideal to ol need to supp o r t 20 Ba y esian statistics? Should it include an algebra solv er? While some of these questions can b e encompasse d in to the “flexibilit y to build extensions,” there a r e certainly op en questions. More than any thing, this list of attributes w as designed to start a critical con v ersation ab out the design of statistical computing to o ls. Considering the existing to ols for statistical computing, McN amara (201 6) suggests that no ne o f them fulfill all the attributes outlined ab ov e. Most to ols can b e described as either a to o l for learning statistics or a to ol for doing statistics . Those fo r learning statistics tend to b e b etter at accessibilit y , easy en t ry , exploratory data a nalysis, flexible plot creatio n, randomization, and in teractivit y . F or example, Tink erPlots and F athom are highly in teractiv e a nd intuitiv e, but mak e it difficult to share results. Spreadsheets lik e Excel are highly accessible to a broad audience, but obscure the computational pr o cesses taking place. In con trast, pro fessional to ols lik e R privilege data as a first-order ob ject, supp ort r epro ducibilit y , and ha v e the flexibilit y to build extensions, but are harder to g et started using a nd the data- analytic pro ducts they create are usually not in teractiv e. F or more details, see McNamara (201 6). No existing to ols curren tly satisfy all the attributes, whic h suggests the need for new or impro v ed soft w a re. It w ould b e ideal to conceiv e of a single to ol that could supp ort users at all lev els. F o r example, a blo c ks programming lang uage with streamlined domain- sp ecific language could step novices into more complex analysis. How ev er, there are few examples of similar to ols in other domains so it seems unlik ely suc h a system will emerge, and indeed, pro jects whic h try to b e all things for a ll p eople often fail. If we ac kno wledge that users will lik ely hav e to mo v e fro m one type of to o l to ano ther, soft w are dev elop ers should b e lo oking for w a ys to ‘bridge the gap’ b etw een the tw o ty p es of to ols (McNamara, 2 015). In other w ords, in to ols with tra ditionally difficult learning curv es, designers should consider how to lo w er the barrier to en try , while in to ols where users tend to ‘exp erience o ut’, designers should build (either tec hnically o r p edagogically) an o nra mp tow ard the next to ol. R has historically b een difficult t o get started using, but curricula and pack ages ha v e b een dev elop ed to low er the barrier to entry (Baumer et a l., 2014; Pruim et al., 2014). Researc hers hav e also b egun studying instruction metho ds that b est supp ort learning of b oth statistics and statistical computing (Baglin, 20 13) These 21 efforts hav e not solv ed t he problem of easy en try , but are easing the transition. More work needs to b e done, but other to ols could tak e inspiratio n from these initial effor t s. As new to ols are dev elop ed and existing o nes are refined, statistical practitioners need to remain activ ely enga g ed in their dev elopmen t and critique to ensure they can supp ort learning as w ell as doing statistics. Ho p efully , this pap er can a ct as a guide as w e b egin t o engage more f ully with this conv ersation. References Agre, P . E. (1995). Th e So c i a l and Inter actional Dimensions of Human - C omputer In- terfac es , c hapter Conceptions of the User in Computer Systems Design. Cam bridge Univ ersity Press. Baglin, J. (2013 ). Applying a theoretical mo del f or explaining the deve lopmen t of tec h- nological skills in statistics education. T e chnolo g y Inno vations in Statistics Educ ation , 7(2). Bakk er, A. (2002) . Route-ty p e and landscap e-type softw are for learning statistical data analysis. In Pr o c e e dings o f the 6th Internationa l Confer enc e on T e aching Statistics . Baumer, B., C ¸ etink a y a Rundel, M., Bray , A., Loi, L., and Horton, N. J. (20 1 4). R Mark- do wn: In tegrating a repro ducible analysis to ol in to in tro ductory statistics. T e chnolo gy Innovations in Statistics Educ ation , 8(1). Berret, C. and Phillips, C. (2016). T eac hing da t a and computat io nal journalism. T ec hnical rep ort, Columbia Journalism Sc ho ol and Stanfo r d Univ ersit y . Bertin, J. (1983). Semiolo gy of Gr aphics . Univers it y of Wisconsin Press. Biehler, R. ( 1 997). Softw are for learning and for doing statistics. Internation al Statistic al R eview , 65(2):16 7–189. Biehler, R., Ben-Zvi, D., Bakk er, A., and Mak a r , K. (2013). Thir d Internationa l Handb o ok of Mathema tics Educ ation , chapter T ec hnology fo r Enhancing Statistical Reasoning at the Sc ho ol Lev el. Springer Science + Business Media. 22 Bosto c k, M. (2013). D3.js: Data-driven do cumen ts. http:/ /d3js.org/ . Bosto c k, M. (2017 ) . d3.express. https://medi um.com/@mbostock/a- better- way- to- code- 2b1d2876a3a 0 . Broman, K. (2015). Initial steps tow ard repro ducible researc h. http://kbro man.org/st eps2rr/ . Bro wn, P . S. a nd Gould, J. D. (1986). An exp erimen t al study of p eople creating spread- sheets. ACM T r ansactions on Offic e Information Systems , 5(3):258 –272. Bry a n, J. (2016). Spreadsheets. useR! Conference, https://git hub.com/je nnybc/2016- 06_spreadsheets . Buc kheit, J. B. and Do noho, D. L. (1995). W av elab and repro ducible researc h. Wavelets and Statistics . Buja, A. a nd Asimo v, D. (1986). Grand tour methods: An outline. In Allen, D. M., editor, Pr o c e e dings of the Sevente enth Symp o s i um o n The Interfac e , pages 63–67. Elsevier Science Publishers B. V. (North- Holland). Buja, A., Co ok, D ., Hofmann, H., La wrence, M., Lee, E.-K., Sw a yne, D . F., and Wick ham, H. (2009). Statistical inference for exploratory data analysis and model diagnostics. Philosophic al T r ansa ctions of the R oyal So ciety A , 367 :4361–438 3. Carv er, R., Ev erson, M., Gabrosek, J., Horton, N. J., Lo c k, R. H., Mo ck o, M., Rossman, A., Ro w ell, G . H., V elleman, P ., Witmer, J. A., and W o o d, B. (201 6). Guideline s for Assessment and Instruction in Statistics Educ ation: Co l le ge R ep ort 2016 . American Statistical Asso ciation. Cass, S., D iak op oulos, N., and R omero, J. J. (2014 ) . In teractiv e: The top prog r a mming languages: IEEE Sp ectrum’s 201 4 ra ting. I EEE Sp e ctrum . Chance, B. and Rossman, A. (2006) . Using sim- ulation to teac h and learn statistics. ICOTS-7, http://www. ime.usp.br / ~ abe/ICOTS7/ Proceeding s/PDFs/InvitedPapers/7E1_CHAN .pdf . 23 Chang, W., Chen g, J., Allaire, J. J., Xie, Y., and McPherson, J. (2015 ). Shiny: Web applic ation fr amework for R . R pac k ag e v ersion 0.12.0. Connell, B. R., Jones, M., Mace, R., Muelle r, J., Mullic k, A., Ostroff, E., Sanford, J., Steinfeld, E., Story , M., and V anderheiden, G . (1997 ). The principles of univ ersal design. T ec hnical rep ort, The Cen ter for Univ ersal Design. De Leeu w, J. (2009 ) . Statistical soft w a re - ov erview. T ec hnical rep ort, Departmen t of Statistics, Univ ersit y o f Califor nia, Lo s Angeles. Diez, D . M., Barr, C. D., and C ¸ etink a y a Rundel, M. (20 14). I ntr o ductory Statistics with R andomization and Simulation . Op enIn tro. Dunham, P . and Henessy , S. (2 008). Equity and use of educational tec hnology in mathemat- ics. In Heid, M. K. and Blume, G. W., editors, R ese ar ch on T e chnolo gy an d the T e a c hing and L e arning of Mathematics , v olume 1. National Council o f T eac hers o f Mathematics. Efron, B. and Tibshirani, R . (1986). Bo o t stra p metho ds fo r standard errors, confidence in terv als, and other measures of statistical accuracy . Statistic al Sc i e n c e , 1(1 ) :5 4–77. Emerson, J. W., Green, W. A., Sc hlo erk e, B., Crowley , J., Co ok, D., Ho fmann, H., and Wic kham, H. (2013). The generalized pairs plot. Journal of C omputational an d Gr aphic al Statistics . Ernst, M. D . (2004). P erm utation metho ds: A basis for exact inf erence. Statistic al Sc i e nc e , 19(4):676– 685. F ew, S. (20 1 0). Co ordinated highligh ting in conte xt: Bringing m ultidimensional connec- tions to light. T ec hnical rep ort, P erceptual Edge. Finzer, W. (2002). F athom: Dynamic data softw are (v ersion 2.1). Computer softwar e. Emeryvil le, CA: Key Curriculum Pr ess . Finzer, W. ( 2 013). The data science education dilemma. T e c h nolo gy Innovation s in Statis- tics Educ ation , 7(2). 24 Finzer, W. (2 014). Hierarc hical da t a visualization as a to ol for dev eloping stu- den t understanding of v ariatio n of data generated in sim ulatio ns. ICOTS9, http://iase - web.org/icots/9/proceedings/pdfs/ICOTS9_9B1_F INZER.pdf . FitzJohn, R ., P ennell, M., Zanne, A., and Cornw ell, W. (2014). Repro ducible researc h is still a c hallenge. T ec hnical rep ort, rO p enSci. Garijo, D ., Kinnings, S., Xie, L., Xie, L ., Zhang , Y., Bourne, P . E., and Gil, Y. (2013). Quan tifying repro ducibility in computational biology: The case of the tub erculosis dru- gome. PLOS O NE , 8(11):e80278 . Gelman, A. (2004). Explorator y data a nalysis for complex mo dels. Journal of Computa- tional and Gr aphic al Statistics , 13(4):75 5–779. Go dfrey , A. J. R. (20 13). Sta tistical softw are f r om a blind p erson’s p ersp ectiv e. The R Journal , 5(1):73 – 79. Hesterberg, T. (201 5). What teache rs should kno w ab out the b o o t strap: Resampling in the undergraduate statistics curriculum. The Ame ric an Statistician , 69(4):371–3 8 6. Hsia, J. I., Simpson, E., Smith, D ., and Cart wrigh t, R. (2005). T aming Ja v a for the classro o m. SIGCSE’05, https://www .cs.rice.e du/ ~ javaplt/drj ava/papers /drjava- language- l evels.pdf . Hullman, J., R esnic k, P ., and Adar, E. (2 0 15). Hyp o thetical outcome plots outp erform error bars a nd violin plots for inferences ab o ut reliability of v aria ble ordering. PLOS ONE , 10(11):e014 2444. Ihak a, R. and Gen tleman, R. (199 6 ). R : A language for data analysis and g raphics. Journal of Computational and Gr a phic al Statistics , 5(3):299 –314. Ince, D . C., Hatton, L., and Graha m-Cumming, J. (201 2). The case for op en computer programs. Natur e , 482:485 –488. Kandel, S. et al. (2011). Researc h directions in data wrangling: Visualizations and trans- formations for usable a nd credible dat a. In formation Visualization , 1 0(4):271–2 88. 25 Kaplan, D. (2007). Computing and in tro ductory statistics. T e chnolo gy Innovations in Statistics Educ ation , 1(1). Kn uth, D . E. ( 1984). Literate programming. The Computer Journal , 27(2):97–1 11. K¨ olling, M. (2010 ). The Gr eenfo ot prog ramming en vironmen t. A CM T r ansactions on Computing Educ ation , 10(4):1 4:1–14:21. Konold, C. and Miller, C. D. (2 0 05). Tink erPlots: Dynamic data exploration. C omputer softwar e Emeryvil le, CA: Key Curriculum Pr ess . Lehrer, R. a nd Sc hauble, L. (2007). Con trasting emerging conceptions of distribution in con texts o f error and nat ura l v ar iation. In Lov ett, M. C. and Shah, P ., editors, T hinking with Data . La wrence Erlbaum Asso ciates. Lo c k, P . F., Lo c k, R. H., Lo ck , D. F., Lo ck Mor g an, K ., and Lo c k, E. F. (2 012). Statistics: Unlo cking the Power of Data . Wiley . Lo c k Morgan, K ., Lo ck , R. H., Lo c k, P . F ., Lo c k, E. F., and Lo c k, D. F. (2014). StatKey: Online to ols for b o otstrap in terv als and ra ndo mizatio n tests. ICOTS-9, http://www2 .stat.duke .edu/ ~ kfl5/Lock20 14.pdf . Lo wndes, J. S., Best, B. D., Scarb o rough, C., Afflerbach, J. C., F r a zier, M. R., O’Hara, C. C., Jiang, N., and Halp ern, B. S. (2017). Our pat h to b etter science in less time using op en data science to ols. Natur e Ec olo gy & Evolution , 1 :0160. Lunneb org, C. E. (19 99). Data Analysis by R esamp ling . Cengage Learning, Boston, MA. Ma jumder, M., Hofmann, H., and Co ok, D. (2013). V a lidation of visual statistical inference, applied to linear mo dels. Journal o f the Americ an Statistic a l Asso ciation , 108 (503):942– 956. Mak ar, K . and Rubin, A. (2014 ) . Informal statistical inference revisited. In Mak ar, K., de Sousa, B., and Go uld, R., editors, I COTS-9 . McNamara, A. (2015). Bridging the Gap Be twe en T o ols for L e arning and for Do ing Statis- tics . PhD thesis, Univ ersit y of California, Los Angeles. 26 McNamara, A. (2016) . On t he stat e of computing in statistics education. submitted. Morandat, F., Hill, B., Osv ald, L., and Vitek, J. (2 0 12). Ev aluating the design of the R language: O b jects and functions fo r data analysis. In ECOO P’1 2 Pr o c e e dings of the 26th Eur op e an c o n fer enc e on Obje ct-Oriente d Pr o gr ammin g . M ¨ u hlbac her, T., Piringer, H., Gratzl, S., Sedlmair, M., and Streit, M. (2014). Op ening the blac k b ox: Strategies f or increased user in v olveme n t in existing algorithm imple- men ta t io ns. IEEE T r ansactions on Visualization and Computer Gr aphics , 20(12):16 43 – 1652. Nolan, D. and T emple Lang, D. (20 07). Dynamic, interactiv e do cumen ts for teac hing statistical practice. International Statistic al R evie w , 7 5 (3):295–3 2 1. Nolan, D. and T emple Lang, D. (2014 ) . XML an d Web te ch n olo gies for D ata Scienc es with R . Springer-V erlag. Office of the Chief Information Officer (2 0 01). Requiremen ts fo r accessible electronic a nd information tec hnology design. T ec hnical rep ort, D epartmen t o f Education. P ea, R. D . (1985). Bey ond amplification: Using the computer to reorganize mental func- tioning. Educ ational Psycholo gist , 20(4):1 67–182. P erez, F. and G ranger, B. E. (201 5). Pro ject Jup yter: Computational narrativ es as the engine of collab or ativ e data science. T ec hnical rep ort, Pro ject Jup yter. Pfannkuc h, M., Wild, C., and Regan, M. (2014 ). Using T o ol s for L e arning Mathematics a n d Statistics , c hapter Studen ts’ difficulties in practicing computer-supp orted data analysis: Some h ypo t hetical generalizations fr om results of t w o exploratory studies. Springer. P o usman, Z ., Stask o, J. T., and Mateas, M. (20 07). Casual information visualization: Depictions of data in ev eryda y life. IEEE T r ansactions on Visualization and C omputer Gr aphics , 13(6):11 4 5–1152. P ow ell, V. (201 4). CSV fingerprints. http://setosa.io /blog/2014/08/03/csv- fi ngerprints/ . Pruim, R., Horton, N. J., and Kaplan, D . ( 2 014). Start te achin g w ith R . Pro j ect MOSAIC. 27 R Core T eam (2 0 15). Comprehensiv e R ar c hive netw ork. http://cran.r- project.org/ . R Core T eam (2 016). R: A language and envir onm ent for s tatistic al c omputing . R F oun- dation for Statistical Computing, Vienna, Austria. Rep enning, A., W ebb, D., and Ioannidou, A. (2010) . Scalable game design and the dev el- opmen t of a che c klist f o r getting computationa l thinking into public sc ho ols. SIG CSE’10, https://dl. acm.org/ci tation.cfm?id=1734357 . Resnic k, M., Maloney , J., Monroy-Hernandez, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silv er, J., Silverm an, B., and Kafai, Y. (2009). Scratch: Programming for all. Communic a tions of the ACM , 52(1 1):60–67. Ridgew ay , J. (2015). Implications of the data rev olution for statistics education. Interna- tional Statistic al R eview , pages 528 –549. RStudio T eam (2 014). RStudio: In t egrated dev elopmen t for R. http://www. rstudio.co m/products/rstudio/ . RStudio T eam (2016 ) . R noteb o oks. http://rmarkdown.rs tudio.com/r_notebooks.html . Rubin, A., Hammerman, J. K., and Konold, C. (2006 ) . Exploring informal inference with in teractiv e visualization softw are. In R ese ar ch p ap ers fr o m ICOTS 7 . Sandv e, G. K ., Nekrutenk o, A., T a ylor, J., and Hovig, E. (2013). T en simple rules for repro ducible computational researc h. PL oS C o mputational B iolo gy , 9(10):e1003 285. Satija, N., Collier, K., Shaw, A., and Larson, J. ( 2016). Hell and high w ater. Th e T e x a s T ribune . Sat y anar ay an, A., Russell, R., Hoffsw ell, J., and Heer, J. (2 0 16). Reactiv e vega: A stream- ing dat a flow architec ture for declarativ e in teractiv e visualization. IEEE T r ansactions on Visualization and Computer Gr a phics , 22(1):659 –668. Shah, P . and Ho effner, J. (2002). Review of gr a ph comprehension researc h: Implications for instruction. Educ ational Psycholo gy R evi e w , 14 (1):47–69. 28 Stefik, A., Hundhausen, C., and Smith, D. (201 1 ). On the design of an educational infrastructure for the blind a nd visually impaired in computer science. SIGCSE’11, https://dl. acm.org/ci tation.cfm?id=1953323 . Tin tle, N., Chance, B., Cobb, G ., Rossman, A., Roy , S., Sw anson, T., and V anderSto ep, J. (2014). I ntr o duction to Statistic al Inv e stigations . Wiley . Tin tle, N., T opliff, K ., V a ndersto ep, J., Holmes, V.- L., and Sw a nson, T. (2012) . Reten- tion o f statistical concepts in a preliminary randomization- ba sed introductor y statistics curriculum. Statistics Educ ation R ese ar ch Journal , 11(1):2 1 . T uke y , J. W. (19 65). The tec hnical to ols of statistics. The Americ an Statistician , 19(2):23– 28. T uke y , J. W. (197 7 ). Explor atory Data Analysis . Addison-W esley Publishing Compan y . Victor, B. (2012) . In v en ting on principle. http://worrydrea m.com/#!/InventingOnPrinciple . W eisb erg, S. (2 005). Lost o pp ortunities: Wh y w e need a v ariet y of statistical languages. Journal of Statistic al Softwar e , 13(1):1–1 2. W eiss, C. J. (2017). Perspectiv es: T eac hing c hemists to co de. Chem ic al & Eng i n e ering News , 95(35) :3 0–31. Wic kham, H. (2009). ggplot2: Ele gant gr a p hics for data anal ysis . Springer New Y ork. Wic kham, H. (2014a ). Tidy data. Journal of Statistic al Softwar e , 59(10 ) :1 –23. Wic kham, H. (2014b). Wh y dplyr? useR! Conference, https://www .youtube.c om/watch?v=dWjSYqI7Vog . Wic kham, H., Co ok, D ., Hofmann, H., and Buja, A. (2010). Graphical inference for infovis. IEEE T r an s a ctions on Visualization and Computer Gr aphics , 16(6). Wild, C. J. a nd Pfannkuc h, M. (19 99). Statistical thinking in empirical enquiry . In terna- tional Statistic al R eview , 67(3). 29 Wilkinson, L. (2005 ). The Gr ammar of Gr aphics . Stat istics and computing. Springer Science + Business Media. Xie, Y. (2014). Dynamic Do cuments with R an d knitr . Chapman & Hall/CR C The R Series. 30