Reproducible (operations) research

Calhoun: The NPS Institutional Archive

Faculty and Researcher Publications Faculty and Researcher Publications Collection

2011-10

Reproducible (operations) research

Nestler, Scott

Operations Research Society of America

OR/MS Today, October 2011, pp.22-28

http://hdl.handle.net/10945/47788

Reproducible

(Operations)

Research

By

Scott Nestler

A primer on

reproducible research

and why the O.R.

community should

care about it.

22

"Let's be clear: the work of science has nothing whatever to

do with consensus, which is the business of politics. What is

relevant is reproducible results. The greatest scientists in

history are great precisely because they broke with the

consensus. "

- Michael Crichton

uried in the middle of physican-author Michael Crichton's quote

about the place (or lack thereof) of consensus in science

,

is an alliteration - reproducible results - that he claims (and I suspect most would agree) is relevant to science [1]. The traditional meaning of reproducible results addresses the verification of a scientific experiment by other researchers using an independent experi-ment. However, computational science poses new challenges to the scientific tradition

[2]. Science often proceeds by iterative refinements where the works themselves (Le., the explicit computations) are seldom published, and it is difficult for others to refine or improve them [3]. As expressed by the Yale law School Roundtable on Data and Code Sharing, "Generating verifiable knowledge has long been scientific discovery's central goal, yet today it's impossible to verify most of the computational results that scientists present at conferences and in papers" [4].

ORIMSTODAY October 2011

c

repn cone duce expe anyo reeOJ same coml

resea

ofre~ tiona thep reprc resea regar medi nal p topic Evid A had t have

Engle

Som! nals, artic! quesl meth this il ject i~ pies f Vitan Time Trust

R,

mati( olof;} that t Most ventil sonal equiv [10]"

I

NFC

INFC tical j said a publi, ical ir other dates

R

R, (

soluti

~

,

; 1 II 5 October 10 II

Using the definition of Baggerly and Berry,

reproducible research (RR) generally means that

conclusions from a single experiment can be repro-duced based on the measurements from that single experiment [5]. Note that it does not mean that

anyone conducting the experiment again (but

recording different measurements) would get the

same results. A more precise definition for use in the computational sciences, might be: Reproducible

research

refers to the idea that the ultimate product of research is tile paper along with the full

computa-tional environment used to produce the results in the paper such as the code, data, etc. necessary for reproduction of the results and building upon the

research [6]. While the majority of the discussions

regarding RR have been in some intersection of the

Author

1

Database ... Analytic code --+- Presentation code --+- Processing code

•

Reader

medical, bioinformaticslcomputational biology,sig-

I

figure

1:

The

research

pipeline as a model for reproducible

research [16[.

nal processing and statistical communities, this is a

topic which operations researchers should care about as well.

Evidence of a Problem?

A NUMBER OF SCIENTIFIC JOURNALS have recently

had to publish retractions or statements of concern [7]. These

have included highly regarded journals, such as: LOl1cet, New Ellgland JOllrnal of Medicille, and Allllois of Tnternol Medicine. Some awareness of this issue extends beyond scholarly jour-nals, due to reporting in the popular media. For example, an

article in The New Yorker last year attempted to answer to the question, "Is there something wrong with the scientific

method?" While reproducibility (or replicability, as termed in this instance) was not the primary focus of the article, the sub-ject is addressed 18]. The author provides a number of

exam-ples from medicine (e.g., anti-psychotic drugs, cardiac stents, Vitamin E), psychology and ecology. An article in the New York Times declared: "Reporters Find Science Journals Harder to Trust, but Not Easy to Verify [9]:'

Robert Gentleman, a well-known statistician and

bioinfor-matician. points out that in much modern research, "

method-ology is often so complicated and computationally expensive

that the standard ... journal paper is no longer adequate ....

Most statistics papers, as published, no longer satisfy the con-ventional scientific criterion of reproducibility: could a

rea-sonably competent and adequately equipped reader obtain equivalent results if the experiment or analysis were repeated?

[ 101" Reading through the various journals published by INFORMS, e.g., Operations Research, Manager/'lell1 Science, INFORMS JOllmal all Computing, etc. or the American Statis-tical Association's Tecllllometrics, it appears the same could be

said about many of the articles appearing in our professional

publications. While many articles in these journals are

theoret-ical in nature, they often include results from simulations or

other computational experiments that could be prime candi -dates for RR. if the authors were: (I) aware of the advantages of

RR, (2) familiar with helpful techniques and technological

solutions, and (3) committed to ('working reproducibly."

History of and More Details About RR

THE FOUNDATIONS OF REPRODUCIBILITY can be found in Aristotle's Dictum about there being no scientific knowledge in the individual [I I]. Jon Claerbout, a geophysics professor at Stanford University, observes, '(From Euclid's

rea-soning and Galileo's experiments, it took hundreds of years for the theoretical and experimental branches of science to

devel-op the standards for publication and peer review that are in use today. Computational science. rightly regarded as the third branch, can walk the same road much faster" [121. Most

sources credit Claerbout as being the first to champion RR in

the computational sciences [2, 13]. In 1995, Buckheit and

Donoho summarized Claerbout's ideas as follows: "An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which gen-erated the figures" [14J. In recent work, David Donoho

sug-gests that the word "knowledge" be substituted for

"scholarship" [l3].

Gentlemen and Temple Lang take these suggestions a step

further. Besides the figures in a published document, they advocate that authors should provide «explicit inputs and code that can be used to replicate the results;' i.e., tables, figures, etc.

based on computation and data analysis [15]. Figure I shows the "research pipeline" modellhat Peng and Eckel, who work

in bioinformatics, use to describe RR 116]. Some of the key

steps are: (I) the pipeline begins with measured data that is

transformed by processing code into analytic data; (2) the

ana-lytic code turns the analytic data into computational results; (3)

these are then summarized by the presentation code into fig-ures and tables in the text. Note that authors and readers use the pipeline in different directions. Authors start with data,

generate analysis and produce the paper itself; readers start at

Ihe other end with lhe text of the paper, and if they are

inter-ested, examine the analysis by acquiring the data and code that

the authors have provided.

REPRODUCIBLE

O

.

R

.

"

Motivations to Work Reproducibly

DONOHO RECENTLY PROVIDED this list of reasons

for working reproducibly [13

I.

While his target audience is primarily biostatisticians, many of his suggestions apply

to O.R. analysts with little or no modification. 1. Improved work products and habits. Because we

know that our scripts will be available to others, we will improve them to a higher level of quality than we would if they were only for our own consumption. But, researchers later returning to their earlier works could also benefit. (I know this is true in my case and

suspect it holds for others.)

2. Improved teamwork. When working as part of a team,

our colleagues can see what we are doing in a more transparent manner and may be more likely to propose improvements. Additionally, confidence in results produced by other team members will be higher.

3. Greater impact. If we make it easier for other researchers to use the methods we have developed, it should lead to more acknowledgement (by way of increased citations) from other researchers using our

computationally reproducible work.

4. Greater continuity and cumulative impact. For ongoing, longer-term projects, working reproducibly can ease the integration of other researchers and students into the team and better preserve the efforts

of team members after they depart the project.

Donoho also points out that taxpayers should want publicly funded research efforts to result in computational reproduci bil-ity. Besides providing good stewardship of work product pur -chased with public funds, working reproducibly also: (I) ensures that access to the work continues after the project is

over; and (2) increases the availability of publicly-funded sp

on-sored research to other researchers and the general public. He states in summary, "I believe anyone who understands the

process and the benefits (of RR) will eventually be moved to practice it" [131.

Funding agencies and grant reviewers can also influence reproducibility through a variety of means, to include: (I) requiring some projects to fully implement reproducibility in their workflow and publications; (2) funding the creation of tools that better support reproducibility in their field; and (3)

others as outlined by the Yale Law School Roundtable on Data

and Code Sharing [4

I

.

Common Objections to RR

RESEARCHERS GIVE a number of reasons for not

"working reproducibly." Some of these are simply resi s-tance to change or "knee-jerk" objections, while others

are more considered objections and deserve more

thoughtful responses. Here are some of the protests that Donoho and his colleagues have encountered when encouraging others to practice RR, along with some po s-sible responses [ 17

I:

1. It takes extra work. This is indeed true, especially when starting to do RR, and breaking your old, informal, non-reproducible habits.

2. Nobody else does it. If you actually work reproducibly,

and make your code and data available, it will: (a) get

noticed, (b) get used, and (c) become a reliable tool.

3. My work is too complicated. This is unlikely, but even

if true, why should anyone believe what you write and publish? Don't you exercise your computations on test data to verify them? Let others see what you have done. Reproducibility may even be morc important in this case.

4. It undermines the creation of intellectual capital. Because tools are given away before they ripen, the researcher cannot develop a toolkit over a career. Maybe, but is the purpose of your publication scholarship, personal aggrandizement and/or financial gain? Repeating prior advice, working reproducibly

can improve teamwork and get your work noticed.

5. Legal issues. There are indeed concerns with

copyrights, patents and licenses. Victoria Stodden has several articles on this issue that might be of interest to Interested Parties and

Shared Responsibilities

SCIENTISTS and researchers

themselves are not the only ones with important roles in RR; significant responsibility also rests with others. Editors of scholarly journals playa key

~

Researchers give a num ber those with concerns about

. this aspect of RR, such as the

of reasons for not "working Reproducible Research

reproducibly." Some are simply Standard (RRS), discussed resistance to change or "knee- shortly [18,191·

jerk" objections; others are more • Possible Solutions and

considered objections. . . Ways to Ease the Pain

A LITERATE PROGRAM, as defined by Donald Knuth in 1992, is a document that contains both code and text segments [201. The text provides an expla -nation of what the code actually does when it is executed. Lit -erate programs support two types of transformations, for different audiences. Weaving a literate program creates the doc-ument for a human reader, while tang[hlg the same me hides

the text and allows the code to be compiled or evaluated by a

role in establishing reproducibility standards in their fields. They can do this in a number of ways, including: (I) imple -menting policies for the provision of stable URLs for open data and code associated with published papers; (2) requir: ing the replication of computational results prior to publ i-cation; and (3) requiring appropriate code and data

citations. ORIMSTODAY October 20 II comput on usin com me Leisch, ' Rand tl As sl in a Lal mat; Uta extracte weavers ing: odf and Mi StatWea How. this appl lab-base which h; supplies have ere reprodUi a datab., propose> For use it an add-;, A COl called B code inti evidentl of the so website

I

Onel which is used for software replicatiJ One 51, IN A working and his c 1. Post. and p alrea( 2. Theo and e differ 3. One-, sh ort-Some The first could fin. of the

set

journals I tional res

ltluence ude: (I) ibility in ation of and (3) on Data for not y resis-: others ~ more sts that I when ne pos-y Icibly, ) get lOl. even e and n test lI1tin

:

1.

Ie lIlcial ,Iy d. 1 has rest to ·ut s the ed nd In !AM,as :ontains 1 expI a-ted. Lit-)I1S, for :he doc-Ie hides ted by a ctober 20 II

computer. Within the statistical community, the focus has been on using literate programming or close variants. The most

common implementation is .Sweave [21J, from Friedrich Leisch, which combines the statistical programming language

R and the typesetting markup language laTeX.

As shown in Figure 2, "Weaving" a Sweave document results in a laTeX me that can be processed into a PDF or other for-mat; "tangling" the same document yields code that has been

extracted for use within R. I am aware of a number of other weavers, for use with various programs and languages, includ

-ing: odfWeave (for R with OpenOffice documents), R2wd (R

and Microsoft Word), SASweave (SAS with L.,TeX), and StatWeave (R, SAS, Stata, and Maple) [6].

However, one need not be a statistician using

R

to benefit from this approach. Donoho and his colleagues provide extensive Mat -lab-based tools, including the Wavelab and Sparselab packages, which have been developed over a IS-year period [ 17[. leVeque supplies a Python interfuce to Fortran code [23]. Peng & Eckel

have created a "elCher" package for R, which enables modular reproducibility by storing results of intennediate computations in a database [16]. Instead of laTeX, anod,er biostatistics researcher proposes using &~ensible Markup Language (XML) with R [24[. For use in computational biology, Mesirov provides GenePattem,

an add-in to Microsoft Office [2].

A commercial effort called "Inference," from a company called Blue Reference, attempted to integrate Rand Matlab

code into Microsoft Office documents. Even though this effort

evidently ceased development in 2009, demonstration copies

of the software are still available at no cost from the company website [25].

One proposal to assist with RR legal issues is Stodden's RRS,

which is similar to the way the GNU Public License (GPL) is used for open-source software. RRS, however, goes beyond

software to include all other data and procedures necessary for

replicating a computational experiment [18].

One Size Does Not Fit All

IN ADDITION TO GIVING EXAMPLES of successes

working reproducibly with Matlab-based toolboxes, Donoho and his colleagues identity three areas where RR has failed [171. 1. Postdocs. Postdocs are typically in a rush to publish

and prefer to work in a manner with which they are already comfortable. Do you blame them?

2. Theorists. A theoretical paper with very few diagrams

and calculations may not benefit from or look any

different as a result of application of RR.

3. One-off Projects. Some projects are either so small or

short-lived that the additional work just isn't worth it. Some of these exceptions might apply within O.R. as well. The first and third are likely directly transferrable. While one

could find instances of theoretical papers fitting the description

of the second item on this list, the majority of papers in O.R.

journals do include multiple figures and numerous comp

uta-tional results that could benefit from RR.

R CMD

Sweave

\

pdflatex

.

lex

.pdf

1

J

(

.

Rnw )

I

. I

_.R

₎

R CMD

Stangle'

I

Figure 2: Results of weaving and tangling a Sweave document (221.

One Journal's Efforts

OVER THE PAST COUPLE YEARS, Biostatistics, a peer-reviewed journal published by Oxford University Press, has

encouraged authors to employ RR. The editors, Peter Diggle and Scott Zeger, recently provided the following explanation of why they chose this route 126]:

"Ollr aim was not to police the technical correct7less of published work bllt rather to recognize that a nontrivial statistical analysis involves ma1/y deci-sions by the mlalyst that are ope" to debate. This is especially true ill more complex mwiyses, for exam-ple, if! a Bayesian analysis lIS;llg Markov chain Mo1lte Carlo that involves choices about the prior, the samplel~ the burn-ill, and the cOflverget'lce crite

-ria. All these may affect the inference draw1l from the data, arId the reader would be lVeli served by giv-ing the ability to find Ollt."

The editors further point out that, by making their work

reproducible, researchers can render state-of-the-art methods more accessible to others and help preclude the abuse of methods

in situations where the assumptions on which they rest are not likely to be satisfied. Similarly, in 0.1<. we often make modeling asslunptions and decisions. While we may state these in words,

translating them unambiguously into an implementable form for

modeling purposes by other researchers can be difficult.

Searching for Evidence of RR In O.R.

My EFFORTS TO DISCOVER RR in O.R. began with querying proponents in other disciplines, including several researchers previously mentioned. When that failed, my

endeavor moved to the Internet. A search of the INFORMS

website for the term "reproducible" yielded eight results. Of

these, three were germane. First, the Finance department

edi-tor's statement for Management Science says, "Authors of

empirical and quantitative papers should provide or make

available enough information and data so that the results are reproducible" [271. The second result was in an article about Office of Management and Budget Circular A-4 in a Decision Analysis Society Newsletter from 2004 [28]. The last was in

the title and abstract for a talk at ICS 2009, the 11th

INFORMS Computing Society Conference, held in

Charleston,

sc.

The presentation, "GAMSWorid and the Growing Demand for Reproducible Computational

REPRODUCIBLE

O.R.

26

ments;' attempted to draw the attention of the mathematical programming community away from a myopic focus on

performance testing and benchmarking. The GAMSWorid website (http://www.gamsworld.org) offers "well-focused,

tested and maintained components (e.g .• model libraries,

tools for generating, collecting and analyzing results) to use as building blocks in making reproducible experiments" [29 [.

A similar search for the word "replicable" produced II more

results; none related to RR. Searches for related terms might yield other instances I did not discover.

Some Examples From O.R.

FOR A RELATIVELY RECENT EXAM PLE of work that is not easily reproducible, look at a paper published in Naval Research Logistics in 2009 on which I was a co-author [30]. This

The previously described tools for producing RR in statistics

will be of use to some o.R. analysts, but they clearly are not

suf-ficient for all purposes. In particular, the various methods of combining R and laTeX are of limited use to those in

optimiza-tion. However, ubiquitous mathematical modeling languages

(e.g., GAMS) and high-quality commercial optimization pack-ages (e.g., IBM's CPLEX, free for academic research) are avail-able and can make models portable, supporting RR. The GAMSWorid site previously introduced appears to be worth consideration for those working in math progrmlffiing or opti-mization. Also, COmputational Infrastructure for Operations Research (COIN-OR) is an initiative for open-source software

in o.R.; the items in their mission statement indicate that their

projects support RR, without explicitly mentioning the term. article studies the Shewhart chart of Q statistics proposed for the So What Can We Do About It?

detection of process mean shifts in start-up processes and short AS MENTIONED EARLIER, the responsibilities for RR are runs. Exact expressions for the rllil-Iength distribution are shared. If we, as individual researchers, work through the chal-derived and evaluated using an efficient computational proce- lenges involved with making research reproducible, there are

dure that Gill be considerably faster than using direct simulation. ways to overcome these obstacles. A few examples are presented

While we provided pseudo-code for both Ollr procedure and the here; there are probably more, perhaps more applicable to some

direct simulation methods, and included the somewhat standard areas of

o.R.

but not known to me. Others may yet need to

be

phrase, " ... are available upon request from the allthors;' I SliSpect developed. If YOll are already using LaTeX and either R or

Mat-it would be difficult for other researchers (and perhaps even our- lab for statistical analysis and simulation, the move to using

selves) to replicate the results ~ The Internet is potential[y a Sweave or a similar option is

based on solely on what was pro- . . . . painless. If not, consider an

inter-videdinthepaperitself. significant a id to those

WhO

~

im step; lise scriptable code, A much more highly refer- desire to make their results rather than spreadsheet or other enced paper(6JS CItes accordmg reproducib[e. GUI-based analysis, which are to Google Scholar) published in inherently less reproducible.

Inte1faces in 1995, "Global Supply Chain Management at Digital If you aren't yet convinced, but want to know more about Equipment Corporation;' was volunteered by one of the article's RR, visit Reproducible Research Planet, http://ww w.rrplan-authors (Gerald Brown) as a poor instance of RR [31

J.

He et.org. There is an active Google Group on the topic available explains, "There was no way Digital Equipment would release at http://groups.google.com/grouplreproducible-research. their strategic plan, and we did not think to produce a pilot:' Additionally, the website http://www.reproducibleresearch.org Brown was also part of a team that wrote an earlier paper"Oesign contains links to a number of useful resources, including many

and Implementation of Large Scale Primal Transshipment Algo- of the papers cited in this article. If you are working in

opti-rithms" in 1977 that was much closer to being considered RR. mization, visit the GAMSWorid site, its companion Google

This work included statements such as, "a set of standard test Groups site, http://groups.google.com/group/gamsworld, and

problems [from NETGEN, 43] that have also been solved by the COIN-OR repository, http://www.coin-or.org.

other contemporary codes." ... "The FORTRAN program For those serving on editorial boards of scientific

jour-GNET/Depth [6], 1975, is distributed to researchers for a nomi- nals, consider whether the articles that appear in your

nal handling charge on an exclusive use basis. For further infor- publication, and your readers, could benefit from authors mation write ... " [32]. He reports that they shipped hundreds of who worked in a reproducible manner. When was the last these, worldwide. So, it appears that this approach can work. time your editorial statement was reviewed and updated? The Internet is potentially a significant aid to those who Perhaps it is time for a revision that encourages or even

desire to make their results reproducible by ma.lting available promises to reward RR. The same applies for organiza-the underlying data and algorithm code. An example of this tions who sponsor scientific publications. If INFORMS.

can be seen at http://faculty.nps.edu/awashburn/, where the with 12 scholarly journals, were to follow the lead of "Downloads" link includes additional hyperiinks to numerous Biost(llistics, imagine how far reproducibility in O.R. zipped applications and Excel workbooks that accompany could progress in a relatively short time.

many of Alan Washburn's recent publications [33]. While this doesn't fully meet the definition of RR provided earlier, it is

evi-dence of steps toward reproducibility taken by one operations

researcher and his colleagues.

That's a Wrap!

BAGGERLY AND BERRY CAUTION US, "RR is n eces-sary, but not sufficient, for good science. It needn't contain

ORIMSTOOAY October 20 II

KNITR

diffiCl

for m

l

Try it

[

trial

0

See

fo

to

no~

REPRODUCIB

L

E

O.R.

28

the motivation for what was done, and the motivation may

be data·dependent .... perhaps the data we used were

'cleaned' before we got them. These potentially fatal biases

will not be known by someone checking reproducibility, and they may not be known to the primary analyst" [51. David Banks at Duke University and a former editor of the jOllrnal

of

the

American Statistical

Association

i

s

skeptical of

the RR

movement to date. He has expressed, "My own sense is that very few applied papers are perfectly reproducible." Howev·

er, he also suggests that a reproducibility standard is a noble aspiration 1341.

The question that Sergey Fome! and Jon Claerbout (the "father" of the RR movement in the past two decades) suggest we ask ourselves before we publish our next paper is, "Have I done enough to allow the readers of my paper to verify and reproduce my computational experiments?" [121. If your answer is "no;' or even "not quite;' consider the ideas present -ed here and how they might help you in an effort to make your

A C K N O W L E D G M E N T S The author thanks Professor Emeritus Gordon Bradley, Distinguished Professor Gerald Brown and Distinguished Professor Emeritus Alan Washburn, his colleagues at the Naval Postgraduate School, for revi ew-ing drafts of this article and providing constructive feedback that led to

significant improvements.

work an exemplar ofRR in O.R., or ... Reproducible Operations Research (ROR).IORMS

Scott Nestler is a lieutenant c%nel in the U.S. Army and an

assistant professor in the Operations Research Department

at the Naval Postgraduate School. Previously, he was on the faculty at the U.S. Military Academy at West Point, served as

the Chief of Strategic Assessments, Multi-National Force Iraq

(MNF-I) at the U.S. Embassy in Baghdad, Iraq, and has worked

as an O.R. Analyst on the Army Staff in the Pentagon. Nestler has a Ph.D. in Management Science from the Robert H. Smith School of Business, University of Maryland·College Park; he is also an Accredited Professional Statistician™.

R E f E R E N C E S

1. Crichton. M .. speech at California Institute of Technology, Pasadena, Calif., Jan. 17. 2003.

2. Mesirov, J, -Accessible Reproducible Research.-Science Magazine,

Vol. 327, Jan. 22, 2010.

3. Gentleman, R., "Reproducible Research: A Bioinformatics Case Study,· Statistical Applications in Genetics and Molecular Biology, Vol. 4, No.1, pp. 1-23, 2005.

4. Yale Law School Roundta'ble on Data and Code Sharing.

-Reproducible Research: Addressing the Need for Data and Code

Sharing in Computational Science," Computing in Science and Engineering, Vol 5. No.5, September/October 2010.

5. Baggerly, K., and O. Berry, "Reproducible Research: AMSTAT NEWS,

January 2011.

6. Reproducible Research Planet!, http://www.rrplanet.com/

reproducible-research/reproducible-research.html.

7. Laine, C. and others. "Reproducible Research: Moving Toward Research the Public Can ReaUy Trust,· Annals of Internal Medicine, pp.

450-454, March 20. 2007.

8. Lehrer. 1.. "The Truth Wears Off,· The New Yorker; Dec 13, 2010.

9. Bosman, J.. "Reporters Find Science Journals Harder to Trust, But Not Easy to Verify," New York Times, Feb 13, 2006.

10. Green. P., "Diversities of Gifts, But the Same Spirit," The Statistician, pp. 423·438, 2003.

11. Turner, W., "History of Philosophy: Ginn and Company. Ch. 11,

1903.

12. Fomel. S" and J. Claerbout, "Reproducible Research: Computing in

Science & Engineering, pp. 5-7, January/February 2009.

13. Donoho, D., "An Invitation to Reproducible Computational Research." Biostatistics, Vol. 11, No.3, pp. 385-388, 2010.

14. Buckheit. J., and D. Donoho, "Wavelab and Reproducible Research:

"Wavelets and Statistics." Springer-Verlag, 1995.

15. Gentleman. R .. and D. Temple Lang, -Statistical Analyses and Reproducible Research: Bioconductor Project Working Papers, Paper 2. 2004.

16. Peng, R" and S. Eckel. "Distributed Reproducible Research Using Cached Computations,· Computing in Science & Engineering, pp.

28-34. January/February 2009.

17. Donoho, D., and others, "Reproducible Research in Computational Harmonic Analysis: Computing in Science & Engineering, pp. 8·18. January/February 2009.

18. Stodden, V" -The Legal Framework for Reproducible Scientific Research: Licensing and Copyright" Computing in Science &

Engineering, pp. 35·40, January/February 2009.

19. Stodden. V .. "Enabling Reproducible Research: Licensing for Scientific Innovation," International Journal of Communications Law & Policy. pp. 1-25, winter 2009.

20. Knuth, D., "literate Programming: Center for the Study of language and Information. Stanford, Calif., 1992.

21. Shotwell, M. BioStatMatt, http://biostatmatt.com/archives/1402. 2011.

22. Leisch. F .• "Sweave: Dynamic Generation of Statistical Reports Using

Literate Data Analysis," Compustat 2002 - Proceedings in Computational Statistics, pp. 575-580, 2002.

23. LeVeque, R., "Python Tools for Reproducible Research on Hyperbolic

Problems,-Computing in Science & Engineering, pp. 19·27. January/February 2009.

24. Temple lang, D .. "Embedding S in Other languages and

Environments," Proceedings of the 2nd International Workshop on Distributed Statistical Computing, March 15·17, 2002.

25. Blue Reference, "Inference: A Solution Platform for Business Professionals,· http://inference.us. 2009.

26. Diggle, P., and S. Zeger. "Editorial,· Biostatistics, Vol. 11, No.3, p. 375,2010.

27. Editorial Statements. Management Science, INFORMS.

http://www.informs.org!content/download/15885/182755/file/Depart

ment Editorial Statements.pdf, p. 2.

28. Morgan. K .. "Opportunity for DA: Decision Analysis Newsletter, Vol.

23, No.1, pp. 6-7, March 2004.

29. Dirkse, S .. and others, "GAMSWorld and the Growing Demand for Reproducible Computational Experiments.-Conference Program,

ICS2009, 2009.

30. Zantek, P., and S. Nestler, "Properties of Q-Statistic Monitoring

Schemes for Start·Up Processes and Short Runs.-Naval Research

Logistics, April 2009.

31. Arntzen, B., and others, "Global Supply Chain Management at Digital Equipment Corporation: Interfaces, Vol. 25. No.1. pp. 69·93,

January/February 1995.

32. Bradley, G., and others, "Oesign and Implementation of large Scale

Primal Transshipment Algorithms: Management Science, Vol. 24, No.!.

pp. 1·34, September 1977.

33. Washburn, A., Naval Postgraduate School.

http://faculty.nps.edu/awashburn/.

34. Banks, D., "Reproducible Research: A Range of Response: Statistics, Politics and Policy, Vol. 2, No.1. Art. 4. 2011.

OR/MSTODAY October 20 I I