6 tips to help you avoid overgeneralizing research findings

If researchers were to visit a chapter of the Veterans of Foreign Wars in West Virginia and ask men between the ages of 45 and 65 their opinions on gun ownership, chances are slim their answers would represent the views of all men in this age group nationwide. In this hypothetical scenario, it’s fairly obvious that what researchers would learn from a group of military veterans in a politically conservative state where hunting is deeply ingrained in local culture and history could not be applied to all men across the U.S.

In real-life research involving people, however, it’s often not as clear whether a study’s results can be generalized to a larger population. Many journalists don’t know how to tell when a study’s findings only apply to the individuals who were involved in the study and when findings can be generalized to a much larger group — people across a city, state or country, for example. This, undoubtedly, leads to mistakes in news stories, which can have serious consequences.

Members of the public often rely on news about research to help them make important decisions on issues such as health care, household finances and their children’s safety and education. Medical professionals who do not read academic articles regularly sometimes rely on news reports for new information about medical treatments and other topics to help them do their jobs.

“The media that reports scientific findings shapes the public’s understanding of the relevance and limitations of scientific findings,” a group of scholars write in an April 2025 analysis of news coverage about behavioral science research. “When media reports omit crucial details and present research findings as universally applicable, practitioners may be misled into applying interventions in contexts where they are less likely to be effective.”

This tip sheet aims to help journalists avoid overgeneralizing — reporting that a study’s findings apply to a much larger group than they actually do. We created this tip sheet after consulting with three scholars with expertise in academic research methods and science communication.

The six tips below are based on advice and insights from:

Turhan Canli, a professor of psychology and psychiatry at Stony Brook University. He’s also a faculty affiliate of the university’s Alan Alda Center for Communicating Science and the founder and director of Stonybrook’s SCAN (Social, Cognitive, and Affective Neuroscience) Center.
Corinne Huggins-Manley, a professor of research and evaluation methodology at the University of Florida and the director of UF’s School of Human Development and Organizational Studies in Education.
Maya Sen, a professor of public policy at Harvard Kennedy School who is a faculty affiliate of Harvard’s Institute for Quantitative Social Science, the Taubman Center for State and Local Government and the James M. and Cathleen D. Stone Program in Wealth Distribution, Inequality& Social Policy.

1. Pay close attention to the study’s sample.

For research involving human subjects, the sample is the group of people researchers studied. Samples can vary considerably in size and composition. A sample could be 100 business owners in Boston, 1,000 students who attend the same university in Ohio, or 10 million U.S. adults who took weight-loss medication last year.

But it isn’t necessarily the sample’s size that determines whether findings can be generalized beyond the people in the sample. What’s key is how well the sample represents the larger population to which researchers want to extrapolate the results. If the majority of a study’s participants are white women, the results can’t be applied to all women. Likewise, if a sample is mostly made up of adults with bachelor’s degrees or adults who live in one region of the country, what researchers learn cannot be generalized to adults nationwide.

Huggins-Manley adds that what researchers learn about people in one country cannot be applied to people in another.

“No matter how well I do a study in the U.S., I cannot use statistics to tell you what’s going on in China,” she says.

2. Look for the term “representative sample.”

A nationally representative sample mirrors the characteristics of a nationwide group of people. If, for example, researchers want to know whether registered voters in the U.S. support a certain policy proposal, they could survey a subset of registered voters whose demographics match registered voters nationally. If a nationally representative sample of registered voters dislikes the policy proposal, it’s highly likely that registered voters, as a whole, dislike it.

If researchers surveyed a group of registered voters that is not nationally representative, it would be inaccurate for journalists to report that the results apply to registered voters nationwide.

Likewise, if researchers had studied soda consumption among kindergartners or the dating habits of older Americans, journalists should not report that the findings apply to those groups of people at a national level unless the researchers relied on nationally representative samples.

“Any study worth anything will aspire to recruit from their target population and have a representative sample of that population,” Canli, the Stony Brook University professor, wrote to The Journalist’s Resource by email.

Researchers may use other phrases to signal whether their study sample is representative of a larger group. A researcher might write that a sample of homeowners in Los Angeles County, California “closely matches” or is “an accurate reflection” of all homeowners in the county. In that case, what the researchers learned could be generalized to registered voters across Los Angeles County.

If a research team asserts that its sample of Texas residents “mirrors the statewide population’s characteristics” or is a “representative sample” of Texas residents, its findings can be generalized to Texas residents.

3. Be aware that some studies are so narrowly focused, their results only apply to people in very specific scenarios or contexts.

The results of some studies will only apply to people with very specific characteristics, such as Black women aged 25 to 45 who live in New York and had an emergency cesarean section to deliver their first child.

Some findings will only apply in specific scenarios or contexts. Sen points to research on nudging as an example.

Nudges are interventions aimed at changing people’s behavior — for example, sending criminal defendants text message reminders of court hearings they must attend or placing bananas near cash registers in high school cafeterias to encourage teenagers to eat more fruit.

Journalists should keep in mind that an intervention can be successful under one set of conditions but fail in another.

“In the social sciences, a lot of well-published RTCs [randomized controlled trials] are very specific to the context in which the researchers ran the experiment and cannot be generalized outside that context,” Sen notes.

4. Don’t assume national averages can be applied to people in individual states.

Many academic studies and government reports estimate national averages, such as the percentage of Americans who have access to fluoridated water or unemployment rates for native-born and immigrant workers. But just as it’s wrong to extrapolate local findings to a national population, it’s also wrong to assume the national average reflects what is going on in an individual state or any other part of the country, Huggins-Manley warns. In your state or city, the percentage of residents who have access to fluoridated water might be much higher or much lower than the national average.

Unfortunately, many analyses that provide national averages don’t provide state-level or regional-level estimates, which makes it tough for journalists to compare individual states to one another and the national average.

However, you can contact the authors of the analysis and ask if they have state-level data or data broken down by geographic regions, Huggins-Manley says. If they do have that information, they might share it with you.

5. When in doubt, ask experts for help.

If you aren’t sure whether you have interpreted a study’s findings correctly, Canli recommends reaching out to the authors for guidance.

“In general, I think the journalist should ideally talk directly to the study author(s) to learn what they think the main take home message is,” he wrote by email.

Canli and Sen advise journalists to also seek feedback from researchers with expertise in the topic who were not involved in the study. Sometimes, scholars don’t see their own blind spots, Sen points out. She adds that researchers are trained and even encouraged to scrutinize one another’s work.

“If it’s [news coverage] that’s going to be a major headline, it does not hurt at all to call someone else who was not involved,” she says. “They can tell you very quickly whether there are glaring errors or overgeneralizations.”

6. Don’t trust artificial intelligence tools to accurately summarize research findings.

When you use an AI chatbot or search tool to summarize research, odds are high there will be mistakes. AI tools powered by large language models are known to fabricate information. That’s why Canli and Huggins-Manley urge journalists not to trust the summaries they produce.

Huggins-Manley also points out that a large language model may not be trained to differentiate between high-quality and low-quality research. AI chatbots can draw information from flawed studies, including retracted studies, as well as from working papers with tentative findings and other research that has not been peer reviewed, she warns.

“It’s going to be learning from that problematic corpus of text,” she says.

Researchers have discovered that AI chatbots can overgeneralize research findings, too. Scholars Uwe Peters and Benjamin Chin-Yee tested 10 widely used large language models to see how well they summarize research published in top science and medical journals. They tested older and newer models — including versions of ChatGPT — to produce thousands of summaries.

A key takeaway: Most of the tested models frequently overgeneralized research results, the two scholars conclude in a paper published earlier this year in the journal Royal Society Open Science.

“Analyzing nearly 5,000 LLM-generated summaries, we found rates of such over-generalizations as high as 73 percent for some models,” Peters, an assistant professor in philosophy at Utrecht University in the Netherlands, and Chin-Yee, an assistant professor in the departments of pathology and laboratory medicine, medicine, and philosophy at Western University in Canada, explain in an April 2025 essay for The Conversation.

“Strikingly, when we compared LLM-generated summaries to ones written by human experts, chatbots were nearly five times more likely to produce broad generalizations,” they write. “But perhaps most concerning was that newer models — including ChatGPT-4o and DeepSeek — tended to generalize more, not less.”

6 tips to help journalists avoid overgeneralizing research findings