Statistics<\/em><\/p>\nFirst, emphasize to students that statistical inference doesn\u2019t prove anything. It can only lend a level of confidence in the results of an analysis. Analysis needs to walk hand in hand with a well-structured research plan, sound data, and good judgment in interpretation. That combination is what makes a study convincing.<\/p>\n
Statistical confidence, in turn, needs to be clarified. Rejecting (or failing to reject) a hypothesis at the 90% level means that if we were to repeat the experiment 100 times with 100 independent random samples, we would expect the same result approximately 90 times (Phillips 2016). Yet in a fast-changing business environment, we\u2019ll never be able to replicate an experiment many times under identical conditions. Discuss: Where does that leave us as management researchers?<\/p>\n
This frequentist view of statistics thus seems fanciful, reasonable in principle but not in practice. Yet it\u2019s perfectly plausible compared to the alternative! Students may be exposed to the Bayesian view, but should not be encouraged to rely on it. Bayesianism \u2013 the most prominent alternative to the frequentist view \u2013 treats probability as a measure of belief or certainty about an event. <\/p>\n
Belief? Certainty? It\u2019s absurd to think that simple equations could say anything meaningful about our inner psychology. Moreover, Bayesianism highlights the question of judgment. New graduate students can easily master the mathematics needed for either statistical paradigm. Their judgment, on the other hand, will mature as their studies progress and as their careers progress. They should not be encouraged to use subjective methods early on.<\/p>\n
Even simple factor analysis requires judgment. (How many factors?) A structural equation model presents many needs for judgment in reducing variable to factors. The point here is that students want to see their research published. Even in double-blind review situations, where the age of the researcher is not evident, the consequences of the researcher\u2019s judgments are evident. It\u2019s risky to submit a judgment-based paper early in one\u2019s career, less risky for a more mature researcher.<\/p>\n
Now what about \u201cindependent random samples\u201d? Many times I\u2019ve asked a first-year graduate class what \u2018random sample\u2019 means, without getting an accurate answer. (Same for \u2018independent.\u2019) Colleagues, please ensure the correct definition is burned into students\u2019 brains, along with the idea that a complicated analysis means nothing, and is not publishable, if the sample is not random: The mathematics of statistical inference depend utterly on the randomness of samples. Then tell them, \u201cDon\u2019t even think of turning in an empirical paper that doesn\u2019t state how you obtained a truly random sample \u2013 or doesn\u2019t convincingly explain how your sample departs from randomness only negligibly.\u201d<\/p>\n
A separate discussion is needed about randomness when the student draws the sample from a huge online (secondary) data set. And still another discussion on how to present and interpret simple analyses of nonrandom samples.<\/p>\n
Students also tend to forget that samples are for making inferences about a population \u2013 and their reports must identify and delimit the population that\u2019s being studied!<\/p>\n
Disciplinary tradition<\/em><\/p>\nI asked a Paper Development Workshop author, \u201cWhy did you use this technique?\u201d<\/p>\n
He replied, \u201cThis is the customary statistical technique in my subfield of management.\u201d <\/p>\n
Well! He did not say it was the right technique for the research question and for the data set \u2013 he said it was the customary technique. Meaning, it was what he was taught, what he habitually turns to, and what he believes editors of journals in his field are looking for and comfortable with. And often, one confidently imagines, a round peg forced into a square hole.<\/p>\n
As Thomas Kuhn (2012) would say, science is a social endeavor. Scientific results are functions of the researcher[1] and the researcher\u2019s community, as well as of the object of study. Yet innovative science means breaking free of consensus science\u2019s mold. <\/p>\n
This can be accomplished by teaching a variety of statistical techniques[2], teaching how to match them with the research question and with available data formats, and awarding promotion credit for publications in good journals that are outside the academic department\u2019s official preferred set. Meaning, journals whose editors have more diverse but still rigorous views concerning methodologies.<\/p>\n
Artificial intelligence<\/em><\/p>\n\u201cEvaluating the output of [AI] models requires expertise and good judgment\u2026. AI is likely to widen workforce divides\u201d[3]. That is, in our context, the divide between researchers who exercise the required judgment and those who do not.<\/p>\n
LLMs return the most frequently expressed views, rather than the maverick or leading-edge ideas. Again, a consensus trap. \u201cResearching is thinking: noticing\u2026 gaps in the conventional wisdom\u2026The risk of outsourcing all your research to a super-genius assistant is that you reduce the number of opportunities to have your best ideas\u201d [4].<\/p>\n
In other words, relying too much on AI is just like relying too much on statistical inference: Excessive dependency on either AI or statistics detracts from an innovative and well-rounded research project (Phillips 2024).<\/p>\n
3. Summary<\/h3>\n
This essay has emphasized judgment as an essential element of research, but has not yet defined it. I consider it a combination of: Integrating one\u2019s life experiences, including work, travel, family and so on; recognizing biases and fallacies; reading critically and widely (not just academic journals); conversing with diverse kinds of people, including academic colleagues and practicing managers, while respectfully entertaining contrary opinions; caring sincerely for the research topic; and consulting one\u2019s own heart and gut as well as head.<\/p>\n
The essay may appear to contradict itself, urging judgment as essential to an overall study, yet urging young researchers not to inject it too much into their statistical analyses. One definition of a gentleman is a man who knows how to play the saxophone but reliably refrains from doing so. In the same way, maturing researchers cultivate judgment but apply it in their analyses only judiciously at first, then more liberally as their research experience accumulates.<\/p>\n
Of course graduate courses in research skills must teach statistical analysis. They should also include discussion of all the items mentioned in the present article, as well as how to craft a research question and a sampling plan; how to tell good data from questionable data; how to interpret results; how to write well; and how to cautiously assess results\u2019 generalizability.\nIn this way, the next generation of graduates will be not only good analysts, but good researchers.<\/p>\n
Footnotes<\/h3>\n
1 We may put a fine point on Kuhn\u2019s assertion by noting that Reverend Bayes was a Presbyterian minister, perhaps predisposed to favoring belief over evidence. A Wikipedia article, https:\/\/en.wikipedia.org\/wiki\/Thomas_Bayes<\/a>, suggests Bayes\u2019 interest in probability arose in the course of an argument with David Hume concerning whether to believe in miracles. Though the belief element persists to this day in Bayesian research circles, Bayes\u2019 Theorem does have a useful, belief-free mathematical interpretation. It can be taught to beginning statistics students without reference to psychology or theology.\n2 Not that the researcher will use them all. However, when it\u2019s feasible to use multiple methodologies \u2013 \u201ctriangulation\u201d \u2013 stronger papers can result.\n3 The Economist<\/em> Feb. 15, 2025, p.57.\n4 The Economist<\/em> Feb. 15, 2025, p.63.<\/p>\nReferences<\/h3>\n\n- Kuhn, Thomas S. (2012 edition) The Structure of Scientific Revolutions<\/em>, University of Chicago Press.<\/li>\n
- Phillips, Fred (2016) “Deciphering p values: Defining significance.” Science<\/em> 05 Aug: Vol. 353, Issue 6299, p.551.<\/li>\n
- Phillips, Fred (2024) \u201cFrom My Perspective: Artificial Intelligence and Research Productivity.\u201d Technological Forecasting & Social Change<\/em>, 212 (2025) 123937.<\/li>\n<\/ul>","protected":false},"excerpt":{"rendered":"
First received 27 February 2025. Published online 25 June 2025. DOI: 10.62478\/ITBA9624 IPNS: k2k4r8o0qjwvqpb8nz5brp55oiot3k6xai9zu0rn9srrrrfijmfat284 [PDF Download] Fred Philips College of Engineering & Applied Sciences, and Alan Alda Center for Communicating Science, SUNY-Stony Brook, and TANDO Institute, USA (fred.phillips@stonybrook.edu) ABSTRACT An experienced journal editor remarks on an apparent imbalance in preparing young management researchers to do […]<\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[64,65,63],"class_list":["post-937","post","type-post","status-publish","format-standard","hentry","category-commentary","tag-research","tag-science","tag-statistics-education"],"_links":{"self":[{"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/posts\/937","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/comments?post=937"}],"version-history":[{"count":5,"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/posts\/937\/revisions"}],"predecessor-version":[{"id":951,"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/posts\/937\/revisions\/951"}],"wp:attachment":[{"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/media?parent=937"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/categories?post=937"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scienceasset.com\/journal\/rosa\/wp-json\/wp\/v2\/tags?post=937"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}