SUPPORTING EVIDENCE

The curricular materials that we have provided in the science evidence evaluation unit are grounded in the scientific research that has established the kinds of reasoning skills that are challenging for students (and often even well-educated adults) as well as successful methods for teaching these skills  Below, we briefly describe some of the studies that have inspired and provided empirical support for these materials.

Correlation & Causal Reasoning

• Seifert, C., Harrington, M., Michal, A., & Shah, P. (2022).  Causal theory error in college students’ understanding of science studies. Cognitive Research: Principles and Implications, 74, 1-22. https://doi.org/10.1186/s41235-021-00347-5
  • One of the most frequent errors in interpreting scientific findings is interpreting a causal relationship from correlational data. So, Seifert et al. developed a brief intervention to help students avoid making causal theory errors.. In the first week, advanced psychology students completed a pretest, during which they read a short science news article and then rated the article’s quality and convincingness. The following week, students received the intervention which included a summary of a correlational finding accompanied by 10 questions explaining that a causal relationship was erroneously concluded, and had students draw causal diagrams of alternative causal models that could explain the correlation. During the final week, students received a post test which was the same structure as the pretest, but with a different article. Students’ scores improved significantly after receiving the intervention. Thus, the causal diagram intervention was effective in improving student’s causal theory reasoning.

• Adams, R. C., Sumner, P., Vivian-Griffiths, S., Barrington, A., Williams, A., Boivin, J., Chambers, C. D., & Bott, L. (2017). How readers understand causal and correlational expressions used in news headlines. Journal of Experimental Psychology: Applied, 23, 1–14. https://doi.org/10.1037/xap0000100
  • The authors measured peoples’ causal interpretations of news headlines. Participants ranked the extent to which one variable caused the other. They found that people distinguished 3 main categories of causal statements: direct causal (‘A causes B’); ‘can cause’ statements (‘A can cause B’); and moderate cause statements (‘A might cause B’; ‘A is associated with B’). Importantly, participants did not reliably distinguish between ‘A might cause B’ and ‘A is associated with B’ statements, suggesting they did not reliably distinguish between correlation (‘associated’ statements) and causal claims (‘might cause’ statements)..

• Coleman, A. B., Lam, D. P., & Soowal, L. N. (2015). Correlation, necessity, and sufficiency: common errors in the scientific reasoning of undergraduate students for interpreting experiments. Biochemistry and Molecular Biology Education, 43(5), 305-315.
  • Assessing a study’s ability to determine correlation, necessity or sufficiency is a key skill for readers of scientific literature. Coleman et al. (2015) tested senior undergraduate students on this skill, first with a pretest, then a posttest after receiving a direct instructional intervention. The instructional module included both a lecture and lab component that explicitly taught sufficiency and necessity in experimentation. Additionally, to control for any learning that may have occurred simply from taking the assessment twice, they had one class of undergraduates take the pretest and a different section of the same class take the post test after receiving a control intervention. The control intervention covered the same biological topics as the instructional intervention but without the emphasis on necessity and sufficiency. Coleman et al. (2015) found that across each group performance on the assessments overall improved from pretest to posttest. Likewise each intervention group had increased performance on the section of the assessment testing for their ability to distinguish correlation versus causation despite this topic not being explicitly covered by either module. Neither group showed significant improvement on utilizing sufficiency and necessity when interpreting simple experiments, though initial scores for this section were already high relative to other sections. But, the group that received the instructional intervention saw improvement on assessment scores in the section that required them to use necessity and sufficiency when interpreting a complex experiment. 

• Corter, J., Mason, D., Tversky, B., & Nickerson, J. (2011). Identifying causal pathways with and without diagrams. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 33, No. 33)
  • Corter et al (2011) tested participants’ ability to identify indirect causal relationships, and how this ability is influenced by the presence or absence of a causal diagram. There were three groups in their first study; two groups saw text describing causal assumptions either listed horizontally or vertically. The third group saw the same causal assumptions displayed in a diagram. All groups were instructed to list all the causal relationships (direct or indirect) between two of the variables. The results revealed that the diagram group outperformed both text conditions, with the vertical text group performing the worst. Additionally, the errors made by each group were due to omitting causal pathways. As the path length increased, the more likely it was to be omitted, but the diagram group omitted fewer of the longer paths than the text groups. In an attempt to replicate and expand on the results of the first study, Corter et al. (2011) conducted a very similar experiment with different participants. For the second study there were 4 groups: vertical text, horizontal text, diagram reading left to right and diagram reading right to left. The results of the second study were very similar to the first study with the two diagram groups outperforming both text groups. Additionally, the diagram right to left group outperformed the left to right diagram group, and spent more time on the task than each of the other groups. Ultimately, Corter et al. (2011) conclude that causal diagrams can be more effective at helping people identify indirect causal relationships than text, especially if these diagrams are displayed in the counterintuitive direction (right to left for Westerners). 

• Easterday, M. W., Aleven, V., & Scheines, R. (2007). Tis better to construct than to receive? The effects of diagram tools on causal reasoning. Frontiers in artificial intelligence and applications, 158, 93.
  • Easterday et al. (2007) tested first year undergraduates on their causal reasoning abilities. In their study, all students received a pretest that consisted of a brief text describing a policy argument and a series of multiple choice questions relating to the causal mechanisms presented by the text. Then, students were randomly assigned to one of three groups. Each group received training on causal reasoning, but for two of the groups the training included causal diagrams while the text group contained no visuals. Post-training, students were tested again twice more. The performance tests varied across the three groups with the text group receiving only text about a policy argument, while the diagram group received the text plus a correct causal diagram, and the tool group received the text plus access to a computer program where they could create their own causal diagram. All students had to answer multiple choice questions about the causal mechanisms in the argument. Finally, students were tested a third time (learning test), with all groups receiving only policy text and multiple choice questions. The results showed that on the pretest all groups scored at chance level. On the performance test, students who received the correct causal diagram outperformed both the tool and text groups. For the learning test, the tool group outperformed the diagram and text groups. Easterday et al. (2007) conclude that text plus causal diagrams are beneficial for helping students understand causal mechanisms, but giving them the tools to develop their own causal diagrams based on the text is most effective at improving their causal reasoning skills.

• Easterday, M. W., Aleven, V., Scheines, R., & Carver, S. M. (2009). Constructing causal diagrams to learn deliberation. International Journal of Artificial Intelligence in Education, 19(4), 425-445.
  • Easterday et al. (2009) had three goals: determine whether causal diagrams as opposed to text increase deliberation performance, whether constructing causal diagrams increases learning, and identify the student struggles with constructing and interpreting causal diagrams. In their experiment, students were divided into three groups: text only, text with diagrams provided and text with a computer tool to construct their own diagram. Every group completed a pretest containing text only, some training sessions, a posttest that corresponded to their assigned group, and a transfer test with only text. All groups performed at chance level on the pretest. For the posttest, the diagram group outperformed both the text and tool groups. However, on the transfer test, the tool group outperformed both the text and diagram group.  In a second study, Easterday et al. (2009) compared the performance of novices to experts on their understanding of causal mechanisms. The procedure was similar to the first study only this time participants were asked to explain their thoughts out loud. The research team then coded these recordings based on the types of errors made by the participants. Both the expert and novice in the text only group performed poorly relative to the other groups. In the tool session, both the expert and novice made errors in their diagrams, but the errors made by the novice translated to major interpretation problems when answering questions relative to the minor errors made by the expert. Finally, the diagram expert and novices all made two major errors: relying on background knowledge rather than the diagram, and speculation errors by adding predictions that were not included in the diagram. The novices in this group also made errors in reversing causation, false uncertainty, chaining, and impasse. So, diagrams are useful for deliberation, practice constructing diagrams is beneficial for future deliberation, and the errors made vary by group.

• McNair, S., & Feeney, A. (2015). Whose statistical reasoning is facilitated by a causal structure intervention?. Psychonomic Bulletin & Review, 22(1), 258-264.
  • Prior research has shown that interventions explaining a causal model relating base rates to the rest of the data can be successful in increasing Bayesian reasoning skills. To expand on this literature, McNair and Feeney (2015) sought to identify which groups of people benefit most from the causal model interventions. Specifically, they looked at how numeracy related to Baysian reasoning prior to intervention and whether numeracy mediates the effect of causal model intervention on Bayesian reasoning. Numeracy is the ability to perform basic math using frequencies and percentages. Participants completed a measure of numeracy then were divided into two groups, each group answered a probability question that required the use of base rates. But one group received a causal model in the question while the other group did not.  McNair et al. did not find that the inclusion of a causal model did not lead to increased use of Baysian reasoning overall. However, they did find that participants with high numeracy scores used Bayesian reasoning more than low numerates regardless of whether the causal model was given or not. Additionally, the greatest predictor of Bayesian reasoning usage was both numeracy and question type such that the group that utilized Bayesian reasoning most often were the participants who were high numerates and received the question with the causal model. This effect held even when controlling for other related variables such as general cognitive ability.  So, the causal model intervention is effective for high numerates, but not for low numerates.

• Zheng, M., Marsh, J. K., Nickerson, J. V., & Kleinberg, S. (2020). How causal information affects decisions. Cognitive Research: Principles and Implications, 5(1), 1-24.
  • Zheng et al. (2020) conducted a series of studies to determine how causal information’s effect on decision making is regulated by prior knowledge. In each of the studies, the participants were presented with a scenario and had to make a decision. One of the groups received a causal diagram about the underlying mechanisms influencing the scenario while another group received no additional information. In the first study, the scenario was about weight management and the results showed that the participants who received the causal information performed worse than individuals with no additional information. Since most people have some experience with weight management, the research team developed a second study to determine whether this poor performance of the causal information group was due to their prior knowledge. In the second study, the scenario discussed type II diabetes. The results showed that individuals who had no personal experience with type II diabetes performed better when they had causal information than when they did not. In contrast, individuals who had experience with type II diabetes performed worse when they had causal information. The following two studies confirmed that people are generally able to use causal diagrams about novel situations, but one’s level of familiarity with the information can result in poor performance and less confidence when provided with causal information.

Effect Size & Correlation/Causation

• Rodriguez, F., Ng, A., & Shah, P. (2016).  Do college students notice errors in evidence when critically evaluating research findings? Journal of Excellence in College Teaching, 27, 63-68.
  • Rodriguez et al. (2016) designed a study to assess student’s ability to recognize common errors in interpreting research results. Participants read a series of eight articles. Some articles contained effect size errors (over interpreting results), others contained misinterpretations of correlational evidence (interpreting correlational results as causational) while others correctly interpreted effect size and correlational evidence. Students were asked to rate the quality of the articles and to explain their reasoning for this rating. The results showed that students ranked articles containing effect size errors as lower quality than the article without these errors. However, there was no difference in rating of articles containing correlation/causation errors and articles without these errors. Additionally, when not explicitly told to think critically about the study, students provided more personal reasons for their rankings as opposed to scientific reasonings. But, when explicitly asked to think critically about the article, students provided more scientific evaluations than personal statements. Students provided more scientific evaluations when the articles contained effect size errors than when they did not, but there was no difference in scientific evaluations between articles containing correlational evidence errors and articles without correlational errors. The scientific evaluations for flawed articles did have more depth than non-flawed articles, but overall the evaluations were very surface level.

Control of Variables Strategies

• Lorch Jr, R. F., Lorch, E. P., Calderhead, W. J., Dunlap, E. E., Hodell, E. C., & Freer, B. D. (2010). Learning the control of variables strategy in higher and lower achieving classrooms: Contributions of explicit instruction and experimentation. Journal of Educational Psychology, 102(1), 90.
  • Prior research has shown effective interventions for teaching control of variable strategies (CVS) during experimentation when taught in one-on-one settings. Lorch et al. (2010) tested the effectiveness of CVS interventions in a classroom setting. Fourth grade classes from high and low performing schools were selected to participate. The classes were assigned to one of three categories: instruction, manipulation or both. The instruction classes received only verbal instructions about CVS. The manipulation classes used only discovery learning based methods in which students explored experimentation in small groups with no feedback. The both groups received both explicit instruction and time for discovery learning. The day prior to the intervention, students completed a pretest. They received two posttests, one the day after the intervention and the other in the next semester. All three groups in both high and low performing schools showed some improvement between the pretest and both posttests. However, the classes who received direct instruction and discovery learning opportunities showed the most improvement. Additionally, the classes in higher performing schools outperformed the classes in lower performing schools overall. So, interventions that combine both direct instruction and discovery learning are effective for teaching CVS in classroom settings. 

• Schwichow, M., Croker, S., Zimmerman, C., Höffler, T., & Härtig, H. (2016). Teaching the control-of-variables strategy: A meta-analysis. Developmental Review, 39, 37-63
  • Schwichow et al. (2016) conducted a review of 72 studies regarding interventions for teaching control-of-variables strategy (CVS). Schwichow and his team wanted to identify effective instructional strategies, assessment methods, and individual differences among students that promote learning of CVS. Their meta-analysis revealed an average effect size of g = 0.61 (95% CI = 0.53–0.69) across all 72 studies. Which indicates that CVS can be effectively taught to students. Additionally, Schwichow et al. (2016) identified study features that had a significant impact on increasing effect size. These features include, a demonstration of a good experiment, inducing cognitive conflict and a hands-on assessment. 

• Sao Pedro, M., Gobert, J., Heffernan, N., & Beck, J. (2009). Comparing pedagogical approaches for teaching the control of variables strategy. In NA Taatgen & H. vanRijn (Eds.), Proceedings of the 31st Annual Meeting of the Cognitive Science Society (pp. 1294-1299).
  • Controlling for variables is a necessary feature to conduct a quality experiment. So, Pedro et al. (2009) wanted to assess middle school students on their understanding and implementation of control of variables strategies (CVS). First all students received a pretest, which included three questions explicitly about CVS strategies. Two weeks later, the intervention took place. The students were divided into three different groups: direct intervention with reification, direct intervention without reification and discovery learning. Each of the groups were working with a virtual ramp experiment that had four changeable features. For direct instruction groups, the students read about CVS and answered questions about confoundedness in the initial setup, then the software included questions requiring students to interact with the ramp and assess confoundedness again. The questions were multiple choice only for the without reification group and the with refication group included multiple choice and open ended questions. Students read again about why their setup was or was not confounded and were explicitly told which variables were or were not controlled for. The discovery group also saw the initial setup and could interact with the ramp, but they did not receive instruction about CVS and received no feedback on their work. The day following the interventions, all students were given a post-test about CVS. Students were instructed to create an unconfounded experiment with the ramp and they received no feedback. Then students were asked the same CVS questions that were on the pretest as well as an additional open ended question. Pedro et al. (2009) found that each intervention group program performed equally on CVS post test. But both of the direct learning groups outperformed the discovery learning group when instructed to develop an unconfounded study by adapting a confounded study.

Anecdotes

• Rodriguez, F., Rhodes, R., Miller, K., & Shah, P. (2016). Examining the influence of anecdotal stories and the interplay of individual differences on reasoning. Thinking and Reasoning, 22, 274-296.
  • Through two experiments, Rodriguez et al. (2016) sought to determine how the presence of anecdotal stories in scientific articles impact individuals’ scientific reasoning. In the first experiment participants read eight fictional news stories about a flawed study. Depending on the condition, the article either contained an anecdote or not. In the second experiment, Rodriguez et al. (2016) controlled for word count, so the articles either contained an anecdote or a short paragraph providing context. The participants ranked the study on quality and convincingness and also provided reasons for their ranking. Additionally, the participants completed measures of individual differences including prior beliefs, need for cognition, scientific training and more. The results of both experiments revealed that the presence of an anecdote decreased the number of scientific responses participants provided in their explanations. When controlling for word count, the researchers found no difference between the ratings of quality for both conditions. This result contrasted their finding in experiment one, which indicates that increasing article length may increase perceived quality. 

• Michal, A. L., Zhong, Y., & Shah, P. (2021) When and why do people act on flawed science? Effects of anecdotes and prior beliefs on evidence-based decision-making. Cognitive Research: Principles and Implications, 6, 1-23.
  • Michal et al (2021) originally set out to explore how the presence of anecdotes in scientific articles impact reader’s ability to identify flawed studies. Results from the first experiment revealed no significant effect for anecdotes impacting participants’ use of scientific reasoning skills. But more surprisingly, participants stated that they were more likely to implement the intervention supported in the study they rated as more flawed compared to the study they rated as less flawed. In a follow up experiment to assess this surprising result, participants completed a pretest answering questions about their prior beliefs about different classroom interventions. Then, they read four scientific articles describing a flawed study that supported one of the four interventions and ranked the article on evidence strength, persuasiveness, and how likely they were to implement the interventions. Participants could expand on their ratings in open ended questions. Results of the second experiment showed that participants were more likely to implement the interventions that they gave a high plausibility score to in the pretest. This result held even when participants were able to recognize that the evidence supporting each of the interventions was flawed. So, prior belief has a significant effect on the decision to implement interventions and that this effect outweighs people’s scientific assessment of the evidence.

Neuroscience

• Rhodes, R., Rodriguez, F., & Shah, P. (2014).  Explaining the alluring influence of neuroscience in scientific reasoning. Journal of Experimental Psychology: Learning, Memory, & Cognition, 40, 1432-1440.
  • Rhodes et al. (2014) designed two experiments to determine how the presence of irrelevant neuroscience information influences scientific reasoning. In both experiments, participants completed a prior beliefs measure, read a short article about a flawed fictional study, rated the fictional study on quality and convincingness, and took assessments of thinking disposition. The article about the fictional study varied by experiment and condition. In the first experiment, the neuroscience condition article included a short paragraph that included neuroscience information that did not explain any potential effect of the study. The control group read the same article but without the neuroscience paragraph. For the second experiment, Rhodes et al. (2014) controlled for word count in the article. So the neuroscience group read the same article as the first experiment, but the control group article replaced the neuroscience paragraph with a paragraph that was about the topic but did not provide any relevant information about the study or its effects. The results from both experiments revealed that having congruent prior beliefs was associated with higher article quality ratings. They found no difference in article ratings by individual thinking dispositions. The majority of participants in both experiments and both conditions failed to identify the methodological flaw in the study. They also found that the presence of neuroscience information led people to report higher perceived understanding of the mechanisms of the effect despite the paragraph not explaining any of the potential mechanisms. Additionally, the neuroscience groups reported higher scientist quality than the control groups, regardless of prior beliefs and individual differences. 

Everyday Scientific Reasoning

• Shah, P., Michal, A., Ibrahim, A., Rhodes, R., & Rodriguez, F. (2017). What makes everyday scientific reasoning so challenging?  B. Ross (Ed). The Psychology of Learning and Motivation,66, 251-299.
  • Shah et al. conducted a literature review (2017) to assess the current research on science evaluation. They focused on four major categories within scientific reasoning: the common errors made by the public, contextual factors, individual differences and possible interventions. Their search revealed people generally fail to recognize threats to validity within a study, including failure to control for confounds, drawing causal claims from correlational studies, and number absolutism. Individual’s ability to identify these errors and think critically are inhibited by contextual features of the article, such as the presence of an anecdote, prior beliefs, and appeal to authority. One’s ability to overcome these contextual constraints vary by person depending on their cognitive flexibility, numeracy, faith in intuition and more. In an attempt to improve people’s generally poor scientific reasoning skills, several intervention methods have been proposed. Some of these methods involve changing the framing of studies so people are less biased towards their prior beliefs, or including visualizations displaying possible causal mechanisms as alternative explanations can lead to less causal claims being made from correlational data. Most importantly, Shah et al. (2017) found that explicit instruction, that includes examples, about scientific reasoning skills and errors can lead to decreased reliance on heuristics and increased use of critical thinking when encountering a scientific article. 

• Campbell, B., & Lubben, F. (2000). Learning science through contexts: Helping pupils make sense of everyday situations. International Journal of Science Education, 22(3), 239-252.
  • To understand how students incorporate science into everyday contexts, Campbell and Lubben (2000) assessed students’ understanding of experiments, ability to apply science to daily life, recognition of social and economic implementation of science, and their perceived source of knowledge. Secondary school students in a contextual science course completed an assessment that contained three different sections. In one section, students were given a summary of a science-based action and had to describe why that action was taken. Another section asked students to design an experiment to solve a scientific question. The final section described a science-based situation that one may encounter in their normal daily life and students had to propose a solution.  Additionally, for each section students were asked to list where they acquired the knowledge to answer that question (such as a science class, or at home). The researchers assigned point values to student answers based on their use of scientific concepts. The scores revealed that the majority of students had poor understanding of experimental design, how to apply science to everyday life situations,  and the social and economic implications of science. Additionally, for the questions about experimental design, the majority of students cited their science class as the source of their knowledge. However, for the questions about explaining a science-based action and using science to solve daily life problems, less than 40% of students cited their science class. These results concerned Cambell and Lubben because these students were in a contextual science class that teaches scientific concepts through scenarios relevant to the students. So, a context based class is not sufficient for enhancing students’ knowledge of science and ability to incorporate science into daily life. 

• Golumbic, Y. N., Dalyot, K., Barel-Ben David, Y., & Keller, M. (2022). Establishing an everyday scientific reasoning scale to learn how non-scientists reason with science. Public Understanding of Science, 09636625221098539.
  • Golumbic et al. (2022) developed the Everyday Scientific Reasoning Scale which can be used to measure how non scientists reason when they encounter science information in daily contexts. This scale underwent validity tests by experts, received feedback from non-scientists, and was compared to other published scales that measure similar skills. The results of the pilot test of the scale on the general public revealed that higher education is associated with higher scores regardless of whether the higher education was in a STEM field. The pilot test also revealed that science education did not have a major impact on scores, possibly because science classes focus on learning facts as opposed to learning how to apply scientific skills to daily situations. 

• Klaczynski, P. A. (2000). Motivated Scientific Reasoning Biases, Epistemological Beliefs, and Theory Polarization: A Two-Process Approach to Adolescent Cognition. Child Development, 71(5), 1347–1366. https://doi.org/10.1111/1467-8624.00232
  • This paper describes how both early (7-8th grade) and middle adolescents (10-11th grade) critically evalute flawed evidence that is either theory (belief)-congruent or theory (belief)-incongruent. The study first collected data on participants’ social class (lower to upper) or religion, as well as various epistemic beliefs (e.g., Need for Cognition, Need For Closure, Heart over Head) to establish whether participants were more knowledge-driven or belief-driven. The main findings were that middle adolescents showed higher competencies in scientific reasoning overall compared to early adolescents, suggesting that analytical reasoning improves with age. However, both early and middle adolescents showed biased reasoning (e.g. reasoning that preserved their prior beliefs) as well as greater polarization after viewing unfavorable (belief-incongruent) evidence. Both groups showed more complex reasoning about belief-incongruent than belief-congruent evidence, suggesting that evidence that conflicts with prior beliefs is especially effective for triggering analytic thinking. Finally, students who were more knowledge-driven (as measured by epistemological scales) showed more analytic reasoning overall than those who were more belief-driven.

Teaching Statistical Reasoning

• Garfield, J. (2002). The challenge of developing statistical reasoning. Journal of statistics education, 10(3).
  • This review by Garfield (2002) summarizes the state of research about statistical reasoning skills. Overall, studies of statistical reasoning have shown that students, the general public and sometimes even researchers fail to grasp statistical concepts. Moreover, even if one can explain the concepts, they may not be able to apply those concepts to a relevant situation outside of the classroom. Garfield (2002) compiles a list of systematic statistical errors identified by various studies, including misunderstanding averages, reasoning poorly about sample sizes, and more. So, despite statistics courses expressing the goal of teaching students statistical reasoning skills, these classes fall short. But, in addition to recognizing these problems, the literature also suggests different interventions and to improve statistical reasoning such as including text and verbal examples, or utilizing software that allows students to alter different factors such as sample size and compare the results. Additionally, Garfield (2002) draws attention to the importance of effective assessments, tests with only right and wrong answers do not allow teachers to determine the statistical reasoning skills of their students. Instead, utilizing open ended or applied questions, could better demonstrate a student’s understanding of statistical concepts and their statistical reasoning skills. 

• Lehman, D. R., & Nisbett, R. E. (1990). A longitudinal study of the effects of undergraduate training on reasoning. Developmental Psychology, 26(6), 952.
  • Lehman et al. (1990) wanted to determine how undergraduate education impacted students’ statistical and conditional reasoning skills. The team selected first year undergraduates who were planning to pick a major in one of the four disciplines of interest: humanities, social science, natural science and psychology. The students complete a measure assessing their statistical and conditional reasoning skills. Lehman et al (1990) then reached out to these students again during their fourth year and if they were still in one of the four disciplines, they were asked to complete the test again. The results showed that after four years of undergraduate training, students studying social science or psychology improved on the measure of statistical reasoning while students in natural sciences and humanities showed improvement on the conditional reasoning assessment. Improved statistical reasoning was associated with taking more statistics classes, while improved conditional reasoning was associated with taking more math and computer science classes. So undergraduate training can improve students’ reasoning skills depending on their area of study. 

• Nisbett, R. E., Fong, G. T., Lehman, D. R., & Cheng, P. W. (1987). Teaching reasoning. Science, 238(4827), 625-631.
  • The literature has been divided regarding people’s ability to effectively learn and apply inferential rules when dealing with logic and statistical problems. Nisbett et al. (1987) conducted a series of studies to assess the natural use or lack of use of inferential rules and how it may change after receiving instructive interventions. They compared both statistical and logical reasoning at the undergraduate and graduate level. For each iteration of the study, there were three intervention groups and one control group. The control group received no instruction about statistical or logical rules. The first intervention group received abstract instruction about statistical or logical rules, while the second group received instruction by examples. The third intervention consisted of both abstract and example rule usage. For statistical rule usage, Nisbett et al. (1987) found that all three intervention groups outperformed the control group on abstract statistical rule usage in novel situations. The performance declined after a two week time delay, but was still superior to the control group performance.  In contrast to the effectiveness of statistical training, performance on abstract logical rule usage did not improve for the intervention groups. But, participants could learn to apply practical logical rules. Ultimately, Nisbett et al. (1987) conclude that formal instruction in statistical rules is effective for increasing reasoning ability and transfer of knowledge to novel situations. The same conclusion is held for reacting practical logical rules too. 

• Sedlmeier, P. (1999). Improving statistical reasoning: Theoretical models and practical implications. Psychology Press.
  • Authored by Peter Sedlmeier, the book Improving statistical reasoning: Theoretical models and practical implications (1999), reviews empirical research on statistical literacy. The book identifies several models that have been the basis of statistical reasoning assessments, as well as theoretical models that have been used to develop interventions for teaching statistical reasoning. Some of the effective training methods identified by Sedlmeier include learning by doing and flexible training materials. The effectiveness of training methods also depends on the type of statistical reasoning being assessed: abstract-rules, conditional probabilities, and adaptive-reasoning. Sedlmeier concludes that statistical reasoning can be taught effectively and developed a comprehensive model of statistical reasoning based on the results of this review known as the PASS model.

Argument Generation by Students

• Sampson, V., & Clark, D. B. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. Science education, 92(3), 447-472.
  • Being able to provide a reasoned argument to support a claim is critical in science and science education. In this review, Sampson and Clark compared several different frameworks that researchers use to determine the quality of a student’s argument. The same argument could receive a different ranking of quality depending on the framework used to make the assessment since each framework has a different focus. They looked at two domain-general frameworks and four domain-specific frameworks. Some of the frameworks emphasized structure while others emphasized content, deductive validity, conceptual features and more. Each framework offers a unique approach of assessing the quality of an argument, but Sampson and Clark conclude that researchers should work to develop a framework that will take a holistic approach. 

• Woolley, J. S., Deal, A. M., Green, J., Hathenbruck, F., Kurtz, S. A., Park, T. K., … & Jensen, J. L. (2018). Undergraduate students demonstrate common false scientific reasoning strategies. Thinking skills and creativity, 27, 101-113
  • In order to understand the common errors in scientific reasoning made by undergraduate students, Woolley et al (2018) distributed an 80 question test assessing 9 scientific reasoning skills to undergraduates at 2 universities. Student responses were coded for errors (false strategies), and a small proportion of the students participated in follow up interviews. Woolley et al. (2018) identified systematic false strategies made by the undergraduates for each of the 9 scientific reasoning skills. The common use of these false strategies results in difficulty with STEM courses as students are required to apply the scientific reasoning skills in these courses.

Data Visualization

• Franconeri, S., Hullman, J., Padilla, L., Shah, P., & Zacks, J. (2021). The science of visual data communication: What works.  Psychological Science in the Public Interest, 22, 110-161. https://doi.org/10.1177/15291006211051956
  • Data visualizations can be highly effective at communicating key information, so long as they are designed properly. Franconeri et al. (2021) condensed the large literature about data visualizations and comprehension into their literature review. The research team then used the findings from the scientific literature to develop a set of guidelines for creating effective visuals. Some of these guidelines include avoiding the use of graphical illusions that can convey incorrect information about the data, avoiding overwhelming the viewer’s working memory capacity, and recognizing that individuals are faster at identifying broad patterns versus making specific comparisons.