Analysis and Findings

Here you report the results of your analyses. Before discussing how to write this section, I want to emphasize the critical importance of documenting every- thing you do for every analysis you conduct for the paper.

Document Everything

For science to be science, research must be transparent, repeatable, and verifi- able. The best way to ensure this is to produce a file full of comments that executes all of the analyses mentioned in the paper. This should also include any cleaning or transforming of the data. In other words, you should be able to hand someone else your raw data and your computer code and they should be able to run that code against that data and reproduce every analysis you conducted for your paper. This includes the main tables and figures you produce, but it should also in- clude every alternative model specification and diagnostic test you ran during the analysis or described in the paper whether it be in the body of the paper, a foot- note, or an appendix. This computer file should also include information about the

software that was used, any additional packages that were installed, and any other information required to perform the analyses. Each analysis or test performed by the code should be preceded by a comment in the code that describes exactly what the next section of code does.

Researchers working in R are used to doing this in the form of what is often called a script file. Such files generally have the extension .R at the end of the file. Those who use Stata for their statistical analysis can accomplish the same thing by creating what Stata calls a .do file. SAS and SPSS also allow users to write out commands in a text file that can be submitted to the software for execution. While many people point and click through the menus of Stata or SPSS, both programs print the actual syntax of commands they execute in their output. Thus, users of the pulldown menus can still copy and paste the syntax into a file so they have an accurate record of what they have done.

There is no substitute for these computer code files. Authors can try to be as precise as they can be when they write about what they did in their analyses, but any description in words alone is necessarily less precise than providing the actual code itself. More journals are requiring authors to submit the data and code nec- essary to reproduce the findings presented in their papers. Besides, being careful about documenting what you’ve done will end up saving you time and improving the quality of your work. You’ll make fewer mistakes, and when editors or re- viewers ask you to revise something for your manuscript before it can be accepted for publication, revising the analysis will be trivial if you have documented your code well.

Finally, every data set also needs a complete codebook documenting the data and metadata. Metadata is that information that provides meaning to the actual data. Metadata includes the names of variables, labels for the values of variables, information about the sample, when the data was collected, etc. Without this information, knowing that two variables are statistically significantly related to each other is meaningless.

Descriptives and Diagnostics

As for writing the analysis section of the paper, I think it is important to start with a table of descriptive statistics for all the individual variables that are used in

the analysis. For continuous variables, you might present the mean, standard devi- ation, minimum value, and maximum value. For categorical variables, you might present frequency distributions instead. The point is that before you consider how any set of variables might be related to each other, you should understand the structure of each variable individually. This is also when you would report any oddities about any individual variable that becomes evident from this description. One mistake authors make when presenting descriptive statistics is to fail to take into account any transformations that are made or any observations that are later dropped from the analysis for some reason. If your table of descriptive statis- tics reports that 2000 people in your survey reported voting in the last election, but your analysis table only shows a sample size of 1500, then the voters described in the first table are not the same ones being used for the analysis in the later ta- ble. The most common reason for this discrepancy is the problem of missing data. Of course, dealing with missing data itself is an important topic that merits ex- plicit attention in your paper. Note: case wise deletion in response to missing data is by far the most common practice and almost certainly your worst alternative. Strongly consider multiple imputation.

At some point you need to perform a complete set of diagnostic tests regarding your analysis. There are standard things to consider, including multicollinearity, heteroscedasticity, clustering in the data, influential cases or groups of cases, miss- ing data, selection bias, etc. You want to know how robust your findings are to different model specifications or choices in measurement.

The results of these diagnostic efforts should be reported in the body of the paper, in footnotes, or in an appendix to the paper. Major concerns that require critical choices should be described in the body of the paper. Results should be presented in a precise manner. Footnotes that say the author looked for evidence of a particular problem and found nothing are insufficient. The actual evidence needs to be presented. Footnotes that say the author ran the analysis a different way and got essentially the same results are not sufficient. The alternative analysis should be presented, probably in an appendix. Remember also that all the code for conducting your diagnostics should be included with computer code that produces the main results.

It is your responsibility to evaluate your findings. You want to be the one to find problems in your paper before a reviewer, editor, or future reader finds them. More importantly, your job is to challenge your own findings. It is not your job

to defend a particular result or theoretical perspective from criticism. Rather, you should be the leading critic of your own work in the sense that you should seek to uncover any limitations, weaknesses, or contradictions yourself.

Far too often scholars get personally invested in a research program to the point where they become defensive when others critique their work or offer a dif- ferent perspective. Science advances when we build on each other’s work and we recognize the improvements to our own work made by others. Progress stalls when scholarly camps form and dig in their heels defending their perspective rather than learning from the perspective of others. Too many scholars feel they need to prove that they are right when science demands we open ourselves to the possibility that we are wrong.

How Many Models to Report?

There are many issues to consider when presenting the main results of the analysis. Some authors prefer to present a series of analyses beginning with a baseline model. They then estimate additional models with additional variables, eventually finishing with the model they believe correctly captures the process they are studying. I can see circumstances where this might be useful. For exam- ple, a first model might represent conventional wisdom in a subfield and the point of the paper is to demonstrate how conventional wisdom changes under a different model specification.

However, I generally recommend against presenting a series of models when only the final model is the one the author believes is correctly specified. Review- ing results from misspecified models can be distracting for the reader and can be a waste of space in the article. In most cases, the theory should lead to the specification of a correct model and that should be the focus of the analysis.

Once you are satisfied with your diagnostics, you can interpret the results of your model.2 Start with the key findings that are most relevant for evaluating your theory. Discuss secondary findings and control variables later. It is important to

2_{Many authors would interpret the results of their main analysis and then go through a series}

of diagnostics and robustness checks. That’s perfectly fine. Where those checks appear in the paper is much less important than making sure they are done and that they do appear in the paper somewhere.

lead with the most important results. You are not writing a murder mystery or a joke – there is no need to hold the punchline until the end.

Statistical Significance and Substantive Importance

It is worth writing about the statistical significance of a finding and the di- rection of that finding – positive or negative. It is also critically important to tell the reader whether the finding is or is not consistent with what you hypothesized. However, these bits of information are not enough. You must provide a substantive interpretation of your findings. Present findings in terms of meaningful quantities of interest that will allow you and the reader to reach richer conclusions about the substantive importance of your findings. Passing a threshold of statistical sig- nificance does not ensure that the findings are substantively important. This is a particular danger when dealing with large data sets. At the same time, failure to cross a traditional threshold of statistical significance does not necessarily mean that a finding is of no substantive importance. This can often occur in analyses that include multiplicative interaction terms.

For example, imagine that you ran a model predicting the percentage of the vote received by a candidate as a function of how much money they spent on their campaign. Instead of just reporting a coefficient and its statistical significance, you should report the results in terms such as an increase in spending of $100,000 translates into an increase in vote share of about two percentage points, controlling for other factors in the model. Expressing results in terms of the units in which the variables are measured – in this case, dollars and percentage points, respectively – makes the findings much easier for a broad range of readers to understand. It also allows you to talk about the magnitude of the finding. In this example, you could comment on whether a $100,000 spending increase is large or small, and you could also do the same for a two percentage point increase.

If you are actually doing this analysis, you might be worried about a small number of very high spending candidates unduly influencing your results. You might also reasonably think that there is a diminishing marginal utility for each additional dollar that is spent. As a result, you might follow what most researchers do when analyzing the impact of campaign spending, which is to transform the spending variable by taking its natural log. This only highlights even moreso how important it is to provide meaningful interpretation of your results. Suppose you

find that an increase in the natural log of spending is associated with an increase in the vote share received by the candidate. Most people do not think intuitively about what an increase in the natural log of something really means. It is the author’s job to translate this result back into terms the reader can understand.

Deciding what is a reasonable increase for your independent variable of in- terest can be difficult. Some might choose to interpret a one standard deviation increase. Others might choose a two standard deviation increase. Still others might choose the interquartile range. Still others will evaluate a change from two standard deviations below the mean up to two standard deviations above the mean. Still others will go from the minimum observed value to the maximum observed value. If you are plotting a figure, these last two approaches are reasonable, but if you are just trying to understand the average magnitude of the effect, these two approaches represent rather extreme shifts in the independent variable.

Sometimes it can be useful to illustrate findings using specific cases. Scholars studying the 50 US states, for example, might describe the impact of an indepen- dent variable on the outcome of interest in terms of comparing one state that has a lower than average value of that independent variable to another state that has a higher than average value of that independent variable. Instead of saying when the independent variable increases from one standard deviation below the mean to one standard deviation above the mean, the outcome changes by some amount, the author might instead describe it in terms of comparing a state like New York to a state like Nebraska.

Special Methods Words

The discussion of statistical significance in the previous section leads to a broader point about writing empirical papers. There are a number of words that have special meaning in the context of statistical methodology. Words like signif- icant, likelihood, variance, outlier, correlation, and many others are used in every- day language, but they have a specific meaning in statistics. I strongly recommend that you limit the use of words like these to their narrow statistical meaning. If a member of Congress is quite different from other members of Congress in a non- statistical way, do not call him an outlier, and do not say that they are significantly different. This will help avoid confusion for your readers.

Reporting Uncertainty

Your results should provide a clear representation of the statistical uncertainty associated with them. Here is where figures can be very helpful. A table with coefficient estimates with indicators of whether or not they are statistically sig- nificant at some level is not sufficient. A table with coefficient estimates and the associated T-scores used to determine statistical significance is not much better. A table with coefficient estimates and their associated standard errors is a bit better. At least a reader can use this information in most circumstances to construct a range of plausible values with the coefficient estimates. In the typical regression model, for example, the reader can mentally construct a 95% confidence interval by adding two times the standard error to the coefficient estimate, and subtract- ing two times the standard error from the coefficient estimate. Of course, you could make it easier for the reader by just providing the coefficient estimate and a corresponding confidence interval.

Conveying statistical uncertainty should carry over to your substantive inter- pretation as well. Returning to our previous example, if an increase in spending of $100,000 leads to an increase in a candidate’s vote share of two percentage points on average, you should also report the upper and lower bound implied by the confidence interval. Suppose in this case the 95% confidence interval ranges from a projected increase in vote share as low as one percentage point or as high as three percentage points, that information should be conveyed to the reader. All statistical estimates come with uncertainty. Thus, all substantive interpretation of statistical results should clearly convey that uncertainty.

Tables or Figures?

Researchers differ on their preference for presenting results in tables or in fig- ures. Currently it is trendy to present everything in figures, but I think there are still some results better shown in tables. For example, some scholars now show the results of regression models as figures with a point plotted to represent the value of a coefficient, a line drawn through the point to represent a confidence interval, and often a vertical line at zero to show which confidence intervals do and do not include zero. Such plots are helpful because they illustrate the confi- dence interval. However, such plots also encourage comparison of the sizes of the

coefficients across variables. Such comparisons are often misleading because the measurement scales of the variables in question differ, and some may be arbitrary. Some scholars will standardize coefficient estimates before producing such plots, but as Gary King clearly demonstrated, comparing standardized coefficients fre- quently obscures more than it reveals.3 My point is simply that you should think carefully about how to present your results in the clearest manner possible. That might include tables or figures.

Regardless of whether you use tables or figures, each table and figure should be able to stand on its own. In other words, each table and figure should have an informative title, explanatory notes, appropriate labels, etc. as needed so that someone who ignored the article and looked only at each table and figure could still understand each table and figure. This requires careful attention to detail. Variable names should be spelled out in meaningful words. For example, write out logged per capita income rather than LN PC INC or as X27 or however the variable might be named in your data set. Even when researchers properly label independent variables in a table, they often forget to define the dependent variable. Tables and figures should include explanatory notes that describe the source of the data. If a figure is based on statistical results, the note should reference those results: e.g. this figure is based on model 2 from table 3.

The explanatory note should also define what is presented in the table. For example, you might say that table entries are coefficients from a logit model, and include 95% confidence intervals constructed via a standard bootstrap. Whatever it takes to convey to the reader what the entries of the table mean must be included. By the time you are ready to share your manuscript with someone other than your advisor or best friends in graduate school, you should format the tables and figures as you see the best ones presented in the leading journals in your field. You can cut and paste raw output and figures from your statistical software into your rough drafts of papers, but by the time you are showing your work to others, it should be formatted to look professional.

3_{King, Gary. 1986. ”How Not to Lie With Statistics: Avoiding Common Mistakes in Quanti-}

Be Clear and Complete