Evaluating Research

Analyzing Research
(for people who aren’t researchers)

As you become more involved in education, you will frequently encounter discussions revolving around various methods and interventions, often backed by “research.” It’s important to understand that research can vary significantly in quality, design, and application. Some studies might lack a solid scientific foundation or have methodological limitations that affect their credibility.

When someone makes a claim based on “research,” it’s helpful to request a copy of the study in question. It’s not uncommon for people to vaguely remember a research finding, often from a news article or blog post, without having direct access to the original research. By examining the primary source, you can better assess the reliability and relevance of the research to your specific context.

In the event that someone provides you with the original study, it’s essential to evaluate its trustworthiness. This doesn’t imply that all research is inherently flawed; however, some studies might have limitations due to methodological issues or biases. It’s crucial to develop a mechanism for distinguishing reliable research from less credible studies, keeping in mind that no system is perfect and that it’s essential to consider multiple sources of evidence before drawing conclusions.

To effectively analyze research, I suggest using the PEARLS criteria: Participants, Execution, Analysis, Relevance, Links, and Scope. By examining these aspects, you can develop a more comprehensive understanding of the study’s trustworthiness and applicability to your specific situation. Remember that research is a valuable tool when used wisely and that understanding the nuances of various studies can help you make better-informed decisions for the students you serve.

To effectively analyze research, consider each of the PEARLS criteria:

Participants: Examine the study's sample size, demographic composition, and selection process. A larger, more diverse sample may offer more reliable and generalizable findings, while a smaller or less diverse sample may limit the study's applicability. Consider the target population and whether the participants are representative of that population.
Execution: Evaluate the study's methodology and design. Did the researchers use appropriate methods to answer their research question? Well-designed studies, such as randomized controlled trials (RCTs) and quasi-experimental designs (QEDs), can provide more robust evidence than less rigorous methods. Look for any potential flaws or biases in the study's execution that may affect its reliability.
Analysis: Assess how the researchers analyzed their data and drew their conclusions. Determine if they are implying or asserting correlation, causation, or neither. Causation can be challenging to establish, so be cautious when interpreting claims of cause and effect. Also, pay attention to any discrepancies between the results and the authors' interpretation of the results (e.g., are they overstating the significance of their findings?).
Relevance: Consider the study's timeliness and contextual relevance. When was the data collected and the study published? If the research topic has evolved or new developments have occurred since the study's publication, its relevance may be affected. Additionally, consider how the study's findings apply to your specific context or population.
Links: Investigate the funding source and any potential conflicts of interest. Who financed the research, and do they have financial, business, or ideological interests in the study's outcomes? Be cautious of studies funded by organizations that stand to benefit from specific findings, as this may introduce bias into the research.
Scope: Examine the breadth and depth of the study. Did the researchers investigate multiple variables or focus on a narrow set of factors? Consider how comprehensive the study is and whether it acknowledges potential limitations and alternative explanations for the observed results.

By evaluating each of these PEARLS criteria, you can gain a better understanding of a study’s trustworthiness and applicability. Keep in mind that not all studies will score highly in every category, and it’s important to consider the balance of strengths and weaknesses in each research report. By critically examining research through the PEARLS lens, you can make more informed decisions for the students you serve.

Consider the following examples. Evaluate the trustworthiness and applicability using the PEARLS criteria.

Practice #1

An article published in American Educator identified a 30 million word gap by the age of 3 – that an average child in a professional family will accumulate experience with almost 45 million words, an average child in a working-class family 26 million words, and an average child in a welfare family 13 million words.

Participants: Forty-two families were observed in homes with children learning to talk. They started when the children were 7-9 months old and followed them until they turned 3 years old. The sample included 13 families from upper socioeconomic status (SES), 10 were middle SES, 13 were lower SES, and 6 were on welfare. There were African-American families in each SES category. Of the 42 children, 17 were African-American and 23 were female.
Execution: Researchers observed, recorded, and analyzed more than 1,300 hours of casual interactions between parents and their language-learning children. Interactions were then disassembled into several dozen molecular features that could be reliably coded and counted.
Analysis: Researchers found that 86-98% of the words recorded in each child’s vocabulary were also recorded in the parent’s vocabulary. By the age of 34-36 months, the children were talking and using numbers of different words very similar to the averages of their parents. Researchers found that children from families on welfare had smaller vocabularies and added words more slowly than did children of the same age in professional families.
Relevance: The study was published in 1995 after researchers observed families for 2.5 years for an hour each month.
Links: No source reported.
Scope: The study focused on the word gap between children from different socioeconomic backgrounds and its potential impact on early language development.

After evaluating each study using the PEARLS criteria, consider how trustworthy and applicable the findings are. Use the information to inform your understanding of the study’s strengths and weaknesses, as well as its potential impact on decision-making processes. Now imagine a scale of 1-5 where 1 means completely untrustworthy and 5 means completely trustworthy. I tend to assume everything is a 3 until I have some evidence to suggest that it should be higher or lower. As you review this example, what 1 to 5 rating would you give it for each criteria?

Now add those six numbers together. This gives you a PEARLS rating between 6 and 30. I generally consider anything below 20/30 to be untrustworthy for decision making purposes and anything above 26/30 to be trustworthy for decision making purposes.

Practice #2

A study published in the American Journal of Clinical Nutrition showed that drinking Concord grape juice daily for 3-4 months could improve spatial memory, as well as driving performance, compared with a placebo group.

Participants: A total of 25 mothers, aged 40-50 years, with at least 1 child under age 13, a BMI of 18-29, working > 30hrs/week, with an adequate understanding of verbal and written English, possession of a full driving license for >5yrs, and having driven >5000 miles in the past year, participated in the study.
Execution: Subjects were randomly assigned, in a double-blind study, to drink 12 ounces of Concord grape juice or a placebo "juice."
Analysis: The researchers reported a small, but significant, improvement in scores for motor skills, executive function, verbal recall, attention, and blood pressure. They also saw an improvement in driving as measured by "car tracking" and steering control for the subjects consuming the Concord grape juice. They concluded that grape flavonoids could improve performance on everyday tasks.
Relevance: The study was published in 2016, and the trial ran for 12 weeks.
Links: The study was funded by Welch Foods, the maker of Welch's grape juice.
Scope: The study examined the potential benefits of Concord grape juice on cognitive functions and driving performance in a specific population of mothers.

Practice #3

A case series published in the Lancet demonstrated that the measles, mumps, and rubella (MMR) vaccine predisposes children to behavioral regression and pervasive developmental disorder in children.

Participants: Twelve children (mean age 6 years; range 3-10; 11 boys) were referred to a pediatric gastroenterology unit with a history of normal development followed by loss of acquired skills, including language, together with diarrhea and abdominal pain. The one girl in the study was noted to be a slow developer compared to her older sister.
Execution: Children underwent gastroenterological, neurological, and developmental assessment and review of developmental records. Ileocolonoscopy and biopsy samplings, magnetic-resonance imaging (MRI), electroencephalography (EEG), and lumbar puncture were done under sedation. Barium follow-through radiography was done where possible. Biochemical, hematological, and immunological profiles were examined.
Analysis: The researchers, using select data from the investigation, identified gastrointestinal disease and developmental regression in a group of previously normal children, which was generally associated in time with the vaccine as an environmental trigger.
Relevance: The study was published in 1998 and took place while each child was admitted to the ward for 1 week, accompanied by their parents.
Links: The study was funded by attorneys who, in addition to other legal representations, also represented parents in lawsuits against vaccine-producing companies.
Scope: The study investigated a potential link between the MMR vaccine and behavioral regression and pervasive developmental disorder in a small group of children.