Making statistical methods in management research more useful: some suggestions from a case study

I present a critique of the methods used in a typical paper. This leads to three broad conclusions about the conventional use of statistical methods. First, results are often reported in an unnecessarily obscure manner. Second, the null hypothesis testing paradigm is deeply flawed: estimating the size of effects and citing confidence intervals or levels is usually better. Third, there are several issues, independent of the particular statistical concepts employed, which limit the value of any statistical approach: e.g. difficulties of generalizing to different contexts, and the weakness of some research in terms of the size of the effects found. The first two of these are easily remedied: I illustrate some of the possibilities by re-analyzing the data from the case study article. The third means that in some contexts a statistical approach may not be worthwhile. My case study is a management paper, but similar problems arise in other social sciences. Keywords: Confidence, Hypothesis testing, Null hypothesis significance tests, Philosophy of statistics, Statistical methods, User-friendliness.

💡 Research Summary

The paper offers a systematic critique of the statistical practices commonly found in management research, using a single case study to illustrate broader methodological shortcomings. It begins by noting that while statistical techniques are ubiquitous across the social sciences, their practical contribution to managerial decision‑making is often limited. Three interrelated problems are identified.

First, the reporting of results is frequently opaque. Most articles present only p‑values and a binary statement of statistical significance, omitting any measure of the magnitude of the observed effect. This leaves readers unable to assess whether a statistically significant finding is also substantively important. The author recommends that researchers always accompany p‑values with effect‑size metrics (such as Cohen’s d, η², or odds ratios) and their corresponding confidence intervals. By doing so, the audience can gauge both the direction and the practical relevance of the effect.

Second, the null‑hypothesis significance testing (NHST) paradigm itself is fundamentally flawed for the purposes of most management studies. The binary decision to “reject” or “fail to reject” a null hypothesis encourages a dichotomous view of reality that does not reflect the nuanced nature of organizational phenomena. Instead, the paper argues for an estimation‑oriented approach: researchers should focus on quantifying the size of the effect and the uncertainty surrounding that estimate. To demonstrate the feasibility of this shift, the author re‑analyzes the data from the case study, replacing the original NHST results with point estimates and 95 % confidence intervals. The re‑analysis reveals that the same data can be interpreted in a richer, more informative way without sacrificing statistical rigor.

Third, the paper points out that statistical methods may be of limited value when the underlying effects are trivially small or when the sample size is insufficient to detect meaningful differences. In such situations, achieving statistical significance often requires inflating the sample, which can be costly and may still yield results that lack practical relevance. Moreover, the author emphasizes the difficulty of generalizing findings across different organizational contexts, cultures, or time periods. These contextual constraints are independent of the specific statistical technique employed and must be addressed during study design.

The author proposes concrete remedies. Reporting standards should be revised to require effect sizes and confidence intervals alongside p‑values. Researchers should adopt an estimation mindset, treating NHST as a supplementary tool rather than the primary inferential framework. Finally, before committing to a statistical analysis, scholars should evaluate whether the expected effect size justifies the effort and resources required, and they should explicitly discuss the limits of external validity.

In conclusion, the paper asserts that these adjustments—more transparent reporting, a shift from dichotomous hypothesis testing to estimation, and careful consideration of contextual generalizability—will enhance the usefulness of statistical findings in management research. While the case study focuses on a management article, the identified problems and suggested solutions are applicable across the social sciences, promising a broader improvement in research quality and relevance.