printNew Data, New Statistics:
from Reproducibility Crisis toward New Requirements to Data Analysis and Presentation in Social Sciences
Deviatko I.F.
Dr. Sci. (Soc.), Full Professor, National Research University Higher School of Economics; Chief Researcher, Institute of Sociology of the Federal Center of Theoretical and Applied Sociology of the Russian Academy of Sciences, Moscow, Russia deviatko@gmail.com
Deviatko I.F. New Data, New Statistics: from Reproducibility Crisis toward New Requirements to Data Analysis and Presentation in Social Sciences . Sotsiologicheskie issledovaniya [Sociological Studies]. 2018. No 12. P. 30-38
The article analyzes main causes and consequences of the interdisciplinary crisis of the reproducibility and reliability of the results of scientific research that has unfolded in the social sciences in parallel with the «data revolution». This crisis is expressed not only in the growing concern of scientists about the reliability of research results and the possibilities to establish the practices securing the transparency of empirical data and the statistical software used for their analysis, but also in disputes on limitations of the routine approach to significance testing and feasibility of alternatives based on Bayesian approach. Some aspects of the relationship between theory and data-driven methods of searching for patterns in empirical data are briefly discussed in the context of describing a new approach to multimodel analysis aiming at evaluation of model robustness and model uncertainty.
Девятко И.Ф. От «виртуальной лаборатории» до «социального телескопа»: метафоры тематических и методологических инноваций в онлайн-исследованиях // Онлайн-исследования в России: тенденции и перспективы / Под общ. ред. А.В. Шашкина, И.Ф. Девятко, С.Г. Давыдова. М.: ООО «Онлайн маркет интеллидженс», 2016. Гл. 1. С. 19–33. [Deviatko I.F. (2016) From «Virtual Lab» to «Social Telescope»: Metaphors of Theoretical and Methodological Innovations in Online Research. In: Online-research in Russia: Trends and Prospects. Ed. by A.V. Shashkin, I.F. Deviatko, S.G. Davydov. Moscow: Online Market Intelldgence: 19–33. (In Russ.)]
Девятко И.Ф. TETRAD-методология: завершение процедурной эпистемы? // Вестник Академии наук СССР. 1991. № 2. С. 79–94. [Deviatko I.F. (1991) TETRAD-methodology: A Completion of Procedural Episteme? Vestnik Akademii nauk SSSR [Herald of the Russian Academy of Sciences]. No. 2: 79–94. (In Russ.)]
Китчин Р. Большие данные, новые эпистемологии и смена парадигм / Пер. с англ. // Социология: методология, методы, математическое моделирование (4М). 2017. № 44. C. 111–152. [Kitchin R. (2017) Big Data, New Epistemologies and Paradigm Shifts. Sotsiologiya: metodologiya, metody, matematicheskoe modelirovanie (4M) [Sociology: Methodology, Methods, Mathematical Modeling (4M)]. No. 44: 111–152(In Russ.)]
Шродт Ф.А. Семь смертных грехов современного количественного анализа в политической науке / Пер. с англ. // Социология: методология, методы, математическое моделирование (4M). 2016. № 43. С. 154–210. [Schrodt Ph.A. (2016) Seven Deadly Sins of Contemporary Quantitative Political Analysis. Sotsiologiya: metodologiya, metody, matematicheskoe modelirovanie (4M) [Sociology: Methodology, Methods, Mathematical Modeling (4M)]. No. 43: 154–210 (In Russ.)]
Appelbaum M., Kline R., Nezu A., Cooper H., Mayo-Wilson E., Rao S.M. (2018) Reporting Standards for Quantitative Research in Psychology: The APA Publications and Communications Board Task Force Report. American Psychologist. Vol. 73 (1): 3–25. DOI: 10.1037/amp0000191.
de Groot A.D. (1956/2014) The Meaning of «Significance» for Different Types of Research. [Transl. and annot. by E.-J.Wagenmakers et al.] Acta Psychologica. Vol. 148 (May): 188–194.
Doucouliagos, C., Stanley T.D. (2013) Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity. Journal of Economic Surveys. No. 27(2): 316–39.
Gigerenzer G. (2004) Mindless Statistics. The Journal of Socio-Economics. Vol. 33(5): 587–606.
Glymour C., Scheines R., Spirtes P. and Kelly K. (1987) Discovering Causal Structure. San Diego, CA: Academic Press.
Graham J.H., Özener B. (2016) Fluctuating Asymmetry of Human Populations: A Review. Symmetry. Vol. 8(154): 1–36.
Haller H., Kraus S. (2002). Misinterpretations of Significance: A Problem Students Share with Their Teachers? Methods of Psychological Research. No. 7(1): 1–20.
Ioannidis J.P.A. (2005) Why Most Published Research Findings Are False. PLoS Med. No. 2(8): e124. DOI: 10.1371/journal.pmed.0020124.
Kerr N.L. (1998) HARKing: Hypothesizing after the Results Are Known. Personality and Social Psychology Review. Vol. 2(3): 196–217.
Lehrer J. (2010) The Truth Wears off: Is There Something Wrong with the Scientific Method? The New Yorker. December, 13.
Munafò M.R., Nosek B.A., Bishop D.V.M., Button K.S., Chambers C.D., Percie Du Sert N., Simonsohn U., Wagenmakers E.J., Ware J.J., Ioannidis J.P.A. (2017) A Manifesto for Reproducible Science. Nature Human Behaviour. Vol. 1(0021). DOI:10.1038/s41562-016-0021.
Nelson L., Simmons J.P., Simonsohn U. (2012) Let’s Publish Fewer Papers. Psychological Inquiry. Vol. 23(3): 291–293.
Nuijten M.B., Hartgerink C.H., van Assen M.A., Epskamp S., Wicherts J.M. The Prevalence of Statistical Reporting Errors in Psychology (1985–2013). Behavior Research Methods. Vol. 48: 1205–1226.
Oakes M. Statistical Inference: A Commentary for the Social and Behavioral Sciences. New York: Wiley, 1986.
Open Science Collaboration (Nosek B.A. et al.) (2015) Estimating the Reproducibility of Psychological Science. Science. 2015. No. 349(6251): aac4716. DOI: 10.1126/science.aac4716.
Rosenthal R. (1979) The «File Drawer Problem» and Tolerance for Null Results. Psychological Bulletin. Vol. 86(3): 838–641.
Rouder J.N., Speckman P.L., Sun D., Morey R.D., Iverson G. (2009) Bayesian t Tests for Accepting and Rejecting the Null Hypothesis. Psychonomic Bulletin & Review. No. 16(2): 225–237. DOI:10.3758/ PBR.16.2.225.
Selvin H.C. (1957) A Critique of Tests of Significance in Survey Research. American Sociological Review. Vol. 22(5): 519–527.
Stapel D.A., Lindenberg S. (2011) Coping with Chaos: How Disordered Contexts Promote Stereotyping and Discrimination. Science. No. 332(6026): 251–253.
Wagenmakers E-J., Verhagen J., Ly A., Bakker M., Lee M.D., Matzke D., Rouder J.N., Morey R.D. (2015). A Power Fallacy. Behavior Research Methods. No. 47(4): 913–917. DOI: 10.3758/s13428-014-0517-4.
Yong E. (2012) Replication Studies: Bad Copy. Nature. No. 485(7398): 298–300.
Young C. (2018) Model Uncertainty and the Crisis in Science. Socius: Sociological Research for a Dynamic World. Vol. 4: 1–7. DOI: 10.1177/2378023117737206.