)
Discussion Guide and Transcript
Season Three - Episode Five
Research Ethics Reimagined Podcast Season Three: Episode Five "The Science of Being Counted with David Dutwin, PhD"
- In this episode of PRIM&R’s podcast, "Research Ethics Reimagined," we explore the methodology, ethics, and future of public opinion research with David Dutwin, PhD, Executive Director and Senior Vice President of AmeriSpeak at NORC at the University of Chicago. Dr. Dutwin is a senior fellow with the program for opinion research and election studies at the University of Pennsylvania. He is a nationally recognized survey methodologist. For more than 20 years, he has taught courses in survey research and design, political polling, research methods, rhetorical theory, media effects, and other courses as an adjunct professor at the University of Pennsylvania, the University of Arizona, and West Chester University. He discusses how probability-based sampling underpins scientifically valid survey research, the safeguards that protect participants in sensitive surveys, and how artificial intelligence is reshaping the field. Listen on Spotify | Listen on Apple| Listen on Amazon
Discussion Questions
- 1.) Representation and Declining Trust in Survey Research
- Dutwin explains that telephone survey response rates have dropped from 30-40% in the 1970s to just 3-4% today, and that "low trust individuals are not participating in surveys" at the same rates as others. How should the research community respond when the people opting out of surveys differ systematically from those who participate, and what are the implications for public health and policy decisions built on that data?
- He describes designing recruitment materials with low word counts, plain language, and a deliberately broad sampling of past clients - from Harvard to the American Bible Society to the American Enterprise Institute - to avoid signaling political or ideological alignment. What lessons does this approach hold for researchers in other fields trying to recruit participants across diverse communities and ideological perspectives?
2.) Protecting Participants in Sensitive Research
- Dutwin notes that NORC’s IRB requires parental notification when researchers want to ask teen panelists about sensitive topics like sexual violence, bullying, or drug use, and that adults must complete several surveys to build trust before being asked whether their teen might join. How can institutions balance the value of including young people’s voices in research with robust protections for minors, especially on emerging issues like AI use and social media harms?
- He describes how Certificates of Confidentiality allow NORC to refuse federal subpoenas seeking participant identities, and notes this protection has actually been tested in real cases. How should researchers communicate both the strength and the limits of confidentiality protections to participants asked to disclose information about health conditions, criminal victimization, or intimate partner violence?
3.) AI, Synthetic Data, and the Future of Survey Research
- Dutwin describes emerging "synthetic data" products in which large language models generate responses rather than human participants, and argues that "fit for purpose" should determine when this approach is appropriate. How should researchers and IRBs evaluate the use of synthetic respondents, particularly for studies that might inform public health policy or clinical decisions?
- He emphasizes the importance of sharing study results back with participants, noting most panelists join because "they want to be heard" and want to see their contributions help the public. How can researchers and institutions build sustainable practices for returning results to participants, and what role should this play in maintaining trust in research over time?
Key Terms
Probability-Based Panel: A research panel whose members are recruited through random sampling from a documented frame (such as a postal address database), enabling results to be generalized to a larger population. This contrasts with nonprobability panels composed of self-selected respondents recruited through advertisements or partnerships.
Sample Frame: The documented list from which a sample is drawn. NORC’s National Frame begins with the US Postal Service database of all US addresses and is supplemented by field enumeration in hard-to-reach areas to achieve approximately 97-98% coverage of US households. Survey Weighting: A statistical technique that adjusts survey results to correct for differences between who actually responded and the population overall - for example, counting underrepresented groups more heavily so the final sample reflects census demographics. Panel Conditioning: The phenomenon in which repeated survey participation may change how respondents behave. Researchers watch for the “three S’s” - speeding, skipping, and straight-lining (selecting the same answer down a list) - as signs of disengagement. Certificate of Confidentiality: A formal protection issued by the Department of Health and Human Services that allows researchers to refuse to disclose identifying information about study participants, even when subpoenaed. Synthetic Data: Survey-like data generated by a large language model rather than collected from human respondents - for example, an LLM producing answers to a thousand survey questions as if it were a thousand individual people.
Sample Frame: The documented list from which a sample is drawn. NORC’s National Frame begins with the US Postal Service database of all US addresses and is supplemented by field enumeration in hard-to-reach areas to achieve approximately 97-98% coverage of US households. Survey Weighting: A statistical technique that adjusts survey results to correct for differences between who actually responded and the population overall - for example, counting underrepresented groups more heavily so the final sample reflects census demographics. Panel Conditioning: The phenomenon in which repeated survey participation may change how respondents behave. Researchers watch for the “three S’s” - speeding, skipping, and straight-lining (selecting the same answer down a list) - as signs of disengagement. Certificate of Confidentiality: A formal protection issued by the Department of Health and Human Services that allows researchers to refuse to disclose identifying information about study participants, even when subpoenaed. Synthetic Data: Survey-like data generated by a large language model rather than collected from human respondents - for example, an LLM producing answers to a thousand survey questions as if it were a thousand individual people.
Additional Resources
- NORC at the University of Chicago - The independent research organization affiliated with the University of Chicago where Dutwin leads AmeriSpeak.
- AmeriSpeak - NORC’s probability-based panel platform, including specialized panels for teens, veterans, and Asian American respondents.
- PRIM&R's Research Ethics Timeline - A resource for exploring the milestones of research ethics, including developments in federal research protections
Transcript
Please note, a transcript generator was used to help create written show transcript.
The transcript of this podcast is approximate, condensed, and not meant for attribution. Listen to the full conversatio on PRIM&R’s Research Ethics Reimagined podcast.
Catherine Batsford: Welcome to Research Ethics Reimagined. I'm your host, Catherine Batsford.
Dan McLean: And I'm Dan McLean.
Catherine Batsford: Today, we’re discussing who gets counted in public health, data trust, and the ethics of representation with Dr. David Dutwin, executive director and senior vice president of AmeriSpeak, NORC at the University of Chicago’s premier multiclient panel-based research platform. He is a senior fellow in the Program for Opinion Research and Election Studies at the University of Pennsylvania, a nationally recognized survey methodologist, and also teaches at the University of Arizona and West Chester University.
We are delighted to have you on our podcast to discuss how representation, participation, and trust influence the data that ultimately shapes public health policy.
First, we’d like to start by asking how you found yourself where you are today. Could you tell us a little about your career path and how you became interested in the field?
David Dutwin: Sure. I think a lot of people probably have unique stories to tell, and mine is similarly unique. I went back to graduate school after working on Capitol Hill. At the time, I thought I was going to become a political speechwriter, so I was taking communication and rhetoric courses.
Finally, in my last semester, I waited until the very last minute to take that dreaded statistics course. I also happened to take a course on public opinion, and it all just clicked. I realized, “Wow, I actually like this statistics stuff, and I’m good at it.” Who knew?
After that, I went on to another school to earn my PhD, focusing entirely on public opinion research and statistics. I really caught the bug, and the rest is history, I guess.
Dan McLean: Before we get too far into the details—and there are a lot of interesting studies and projects you’ve all worked on that we’ll explore today—could you explain the relationship between NORC at the University of Chicago and the University of Chicago?
I also know that NORC is spelled N-O-R-C. Could you elaborate a bit on the origins of the name as well?
David Dutwin: Sure. So a little over eighty years ago, the National Opinion Research Center was founded at the University of Chicago. Like a lot of academic institutions, it was a small survey shop to to do research for professors and whatnot. And about fifty years ago, NORC had gotten so big that the university basically said, it doesn't make sense for you to be solely and universally under our umbrella. And so NORC split off, but basically the agreement was that we would still be affiliated with the university. At least half of our board has to be university administrators, professors, etc.. But we really are quite autonomous. We have a major office in downtown Chicago, as well as one in in DC. The NORC name is funny depending on, the age. Sometimes we were the National Opinion Research Center and other times it's just NORC. Right? And it sort of depends on whatever marketing firm we have at the time recommended would be the best foot forward, essentially. So, yes, we are we are officially NORC as of the, about a decade ago. Dan McLean: AmeriSpeak, which is really that's your domain inside of NORC. Right? How does that fit into to NORC overall?
David Dutwin: So NORC at the University of Chicago generally is a research organization that conducts large-scale federal, state, local, and international research. That work includes surveys and focus groups, but also direct policy research and program evaluation. For example, NORC may evaluate educational systems in countries such as Pakistan and make recommendations for improvement. Similarly, in the US, NORC evaluates public policies and programs and provides evidence-based recommendations.
AmeriSpeak was developed, in part, to diversify that work and expand research opportunities with foundations, academic institutions, and for-profit organizations. To support that effort, NORC established a probability-based survey panel.
Survey research itself has evolved significantly over time. In the 1950s, 1960s, and into the 1970s, door-to-door interviewing was the primary survey method. Later, advances in technology made it possible to randomly sample telephone numbers, and telephone surveys became the predominant approach. In the 1970s, response rates of 30% to 40% were common, meaning a substantial proportion of people contacted by phone would participate in a survey.
As cell phones became more common, survey researchers shifted to cell phone sampling frames. At the same time, the rise of spam calls and telemarketing changed how Americans responded to unknown phone numbers. In the 1970s, an unexpected call might reasonably have been a survey request. Today, most people assume unknown calls are telemarketing or spam. As a result, telephone survey response rates have declined dramatically and are now often closer to 3% to 4%.
That decline has two major consequences. First, surveys become less reliable when only a small percentage of people participate. Second, telephone surveys become much more expensive because researchers must contact hundreds of people to secure a single completed response.
Probability-based panels emerged over the past decade as a new survey methodology designed to address some of these challenges. Researchers recruit participants in advance and ask them to agree to take surveys using scientifically random sampling methods. AmeriSpeak, specifically, uses the United States Postal Service database of residential addresses in the United States as its sampling frame. NORC then randomly selects households and recruits participants through mailed invitations, text messages, phone calls, and door-to-door outreach.
Although door-to-door recruitment is expensive, it remains highly effective. Approximately 30% of people contacted through in-person recruitment agree to participate, compared with response rates closer to 4% or 5% for many other survey methods. Because the panel is built from a random sample, the methodology remains scientifically rigorous and nationally representative. Dan McLean: One of the studies that caught my attention as I was reviewing your work was an annual nationally representative study of young people designed to help inform solutions that support their well-being. The study began in 2024 and is ongoing.
It examines the experiences and perspectives of approximately 1,500 young people across four major areas: sociopolitical divisiveness and healing, civic education, artificial intelligence, and mental health. I was wondering if you could share a little more about the study, including how the AmeriSpeak teen panel helped inform the results.
This feels like an issue that has been top of mind for many people over the last several years, so I’d also be interested to hear what insights you’ve gained from the research. David Dutwin: Sure. AmeriSpeak is really a suite of panels. The primary AmeriSpeak panel includes adults age 18 and older, but we also have a teen panel, a veterans panel, and several state and regional panels that allow us to drill down into more localized data. For example, we have ChicagoSpeaks, as well as panels in Texas and Ohio. We also have an Asian American panel that conducts interviews in four languages in addition to English.
With the teen panel specifically, we worked hard to build it thoughtfully. When someone joins AmeriSpeak, we ask a range of profile questions about topics such as politics, finances, health, education, travel, technology, and social media use. We also collect household information, so we know whether there is a teenager in the home.
Initially, we would immediately ask whether the teenager would also like to join the panel, but participation rates were not very strong. We realized we first needed to establish trust with families. Now, we allow adult participants to complete several surveys so they become familiar with who we are and how the process works. Once that trust has been established, we ask whether their teenager would be interested in joining the separate teen panel.
Today, we have several thousand teens on the panel and approximately 50,000 adults overall. That’s important because, like much of survey research today, there are easier but less rigorous ways to recruit teenagers through what are known as nonprobability panels, or convenience panels. A teenager may see an advertisement on Instagram or another social media platform asking if they want to participate in a survey and simply opt in.
The challenge with those approaches is that they are self-selected and not scientifically random. The participants tend to be teens who are already active on social media and willing to take surveys, which can limit how representative the results actually are. The challenge with those methods is that they are not scientifically rigorous because they rely on self-selection. The teens who participate are generally those who are already active on social media and willing to take surveys.
One of the reasons random sampling is so important is that randomization is really at the core of scientific research. If you randomly divide 500 people into two groups, give one group a placebo and the other a new medication, and the group receiving the medication experiences significantly different outcomes, most people would recognize that as a scientifically valid result.
Survey research works in much the same way. We begin with a national list of residential addresses in the United States and randomly select households to contact. That randomization process helps us produce scientifically rigorous and highly valid results.
We also work carefully to build trust with families. AmeriSpeak has an institutional review board, or IRB, that reviews every project to ensure participants are protected. That becomes especially important when researchers want to study sensitive topics involving teenagers, such as bullying, drug use, sexual violence, or mental health. In some cases, the IRB requires additional parental notification or consent before those surveys can move forward.
Right now, for example, we are conducting research examining the potential harms associated with social media, including bullying and other online experiences affecting teens. Artificial intelligence is also becoming a major area of interest. Researchers increasingly want to understand how teens are using AI and the extent to which some may begin treating AI systems almost like another person or source of emotional support. As a society, it is important that we better understand these behaviors so we can develop thoughtful policies, communication strategies, and support systems that help young people navigate an increasingly complex digital environment.
Catherine Batsford: When you think about public health data, what does that look like from a data collection perspective? Is it the same type of process you’ve been describing, where researchers come to you with specific projects and use your panels to conduct studies? Or are there also broader public health data collection initiatives that NORC works on directly? David Dutwin: NORC conducts a significant amount of public health research, particularly for the federal government. We manage projects such as the National Immunization Survey, along with a number of other studies involving Medicaid recipients and other populations. This work is important because the federal government relies on accurate public health data to understand issues such as disease prevalence, health care access, and how Americans use health insurance and medical services.
Earlier in my career, I was part of a team that conducted a large health interview survey in Massachusetts over many years. The survey helped policymakers better understand how residents were accessing health care and what types of insurance coverage they had. Those findings contributed to the development of what was known at the time as “RomneyCare,” which later informed elements of the Affordable Care Act. It was not the only factor involved, of course, but the survey data played an important role in helping policymakers shape those policies.
I think many people underestimate how central survey research is to public policy and decision-making in the United States and around the world. When most people think about surveys, they think about election polling and questions about who may win an election. In reality, political polling represents only a very small percentage of all survey research conducted in the United States. Most surveys are focused on areas such as public health, health insurance, education, crime statistics, and other issues that help governments and organizations better understand societal needs and develop more effective policies.
That mission of providing nonpartisan, evidence-based insight has always been central to NORC at the University of Chicago.
During the COVID-19 pandemic, for example, AmeriSpeak conducted multiple studies each week to help track rapidly changing public health conditions and public attitudes. Probability-based panels are particularly valuable during fast-moving events such as COVID-19, avian influenza, SARS, or other emerging health concerns because they allow researchers to move very quickly. A large federal survey can take months to design and launch, but probability panels can often begin collecting data within days, providing near real-time insight into rapidly developing situations. Dan McLean: I wanted to go back to something you mentioned several times about building trust with participants and their families. That’s something PRIM&R has spent a great deal of time thinking about in recent years—particularly how to strengthen trust in the scientific research process more broadly.
I was also reading about how your IRB operates and the fact that NORC at the University of Chicago provides certificates of confidentiality to participants in its panels.
When you approach families—and, in some cases, their children—how do you explain the process in a way that helps build trust? And could you also explain what a certificate of confidentiality is and why it matters? It seems especially important if you want participants to feel comfortable speaking openly and honestly about sensitive topics. David Dutwin: At the beginning of every survey, we include introductory language that explains participants’ rights and how their information will be protected. Because AmeriSpeak is a panel, many participants have heard this information before, but we still review it carefully for every study. We want respondents to understand that participation is voluntary, they may decline to answer any question, and the survey is not an interrogation or deposition.
We also make it very clear that their data will remain anonymous and confidential. Once a survey is completed, identifying information is removed and replaced with a case identification number. Researchers working with NORC at the University of Chicago do not receive personally identifiable information unless there is a specific and approved reason to do so, and participants are informed in advance if that situation arises.
Certificates of confidentiality are another important layer of protection. These certificates formally reinforce our commitment to maintaining participant confidentiality, which is especially critical in public health research involving sensitive information. Many surveys involve topics such as health conditions, interpersonal violence, prior victimization, mental health, or other deeply personal experiences.
In those cases, participants need to feel confident that their information will remain protected. Even if a government agency were to request identifying information through legal channels, a certificate of confidentiality issued through the US Department of Health and Human Services provides strong protections against releasing personally identifiable data linked to survey responses.
Those protections are essential to building trust and ensuring participants feel comfortable responding honestly to sensitive questions. Catherine Batsford: So you said before that the populations that you're surveying is fairly randomized, but do you do you feel like there's certain populations that are perhaps asked questions more frequently? Is there a methodology for not having study fatigue by certain populations that researchers could be thinking about? David Dutwin: I would answer that in two ways. First, with regard to survey fatigue, the numbers are generally on our side. There are currently more than 267 million adults in the United States, so the likelihood that someone in the general population is repeatedly contacted for surveys is relatively low.
The exception, of course, is for people who intentionally join a panel such as AmeriSpeak, where participation in surveys is expected. Even then, we work carefully to keep participants in what I would call a “Goldilocks zone.” If people are not engaged often enough, they may lose interest and disengage from the panel. On the other hand, if they receive too many survey requests, they can experience survey fatigue and eventually leave.
To avoid that, we try to maintain a panel large enough that participants are invited to complete no more than about one survey per week. We also study what is known as “panel conditioning,” which refers to the concern that repeated participation in surveys could change how people respond over time.
Some worry that frequent survey participation could lead to lower-quality responses through behaviors we sometimes refer to as the “three S’s”: speeding through surveys, skipping questions, or straight-lining responses by selecting the same answer repeatedly without carefully considering the questions. However, research generally shows that panel conditioning effects are relatively small. In some cases, experienced panelists actually become more attentive and thoughtful respondents.
Even so, data quality remains a major priority. We use quality-control measures to identify patterns suggesting that someone may not be engaging seriously with a survey. For example, if a survey designed to take 15 minutes is completed in only four minutes, we may follow up with the participant and remind them that careful responses are important. Continued low-quality participation can result in removal from the panel.
We also work to keep participants engaged by sharing newsletters and research findings with them. One of the most common reasons people say they join the panel is because they want their voices heard and want to contribute to research that benefits the public. We try to reinforce that connection by showing participants how the studies they contributed to are being used.
More broadly, I do worry about declining survey response rates. Whenever I speak publicly, I encourage people to participate in legitimate surveys when they have the opportunity. Surveys are one of the ways people can contribute their perspectives and experiences to public discussions and policymaking.
Unfortunately, years of spam calls and aggressive telemarketing have made many people understandably skeptical of being contacted by strangers. But when conducted responsibly, surveys remain an important way for individuals to make their voices heard and contribute to research that can inform public policy and improve society. Dan McLean: A couple of things came to mind as you were speaking. First, you mentioned the importance of sharing research results back with participants, which is something we’ve been discussing quite a bit recently. In fact, the director of the National Institutes of Health has emphasized the importance of returning research results to participants across the broader human subjects research enterprise. It was interesting to hear you highlight that as well.
I also wanted to go back to the structure of your panels and how you ensure they remain representative. AmeriSpeak describes itself as a probability-based panel in which households are selected from documented sample lists, often referred to as a “sample frame.”
Could you explain what that means in practice? How are these panels designed to ensure they reflect a broad population rather than a single niche group of participants? David Dutwin: That’s a great question. I think the best place to start is with the fact that there are generally two types of survey panels: probability-based panels and nonprobability panels.
Nonprobability panels are self-selected. For example, someone might see an online advertisement asking if they want to participate in surveys and simply opt in. I was once invited to one because I used Amtrak frequently and belonged to its rewards program. That panel company also partnered with airline loyalty programs such as American Airlines.
The challenge with that approach is representativeness. If you recruit participants primarily through travel rewards programs, you have to ask whether the resulting sample truly reflects the broader population, especially if you are conducting research on travel behaviors or public opinion.
Probability-based panels work differently. With AmeriSpeak, we begin with a national sample frame built from the United States Postal Service database of residential addresses in the United States. That database covers approximately 92% of US households. The remaining households may include people who rely solely on post office boxes or individuals living in extremely rural areas, such as parts of Appalachia or Alaska, where addresses can be more difficult to identify and deliver to.
To improve coverage, NORC supplements the postal database with additional fieldwork to identify households that may not appear in standard address files. As a result, coverage increases to approximately 97% to 98% of US households.
From there, the process relies on random sampling. We randomly select households from that national frame and invite them to participate. The goal is to create a sample that reflects the broader population as accurately as possible.
One of the major challenges in survey research today is declining response rates. Decades ago, telephone surveys often achieved response rates in the 30% range. Today, participation rates are often in the single digits. The critical question becomes whether the people who participate are meaningfully different from those who do not.
Historically, differences in participation could often be adjusted statistically using demographic information such as age, race, gender, and education level. For example, if younger adults were underrepresented in a survey sample, researchers could weight their responses more heavily to better reflect the overall population.
However, researchers now recognize that participation is influenced by more than demographics alone. Trust in institutions and trust in science also appear to affect whether individuals choose to participate in surveys. People with lower levels of institutional trust are often less likely to respond, which can create additional representational challenges.
To address this, we work carefully on recruitment strategies and communication. We aim to use accessible, straightforward language and ensure participants understand that surveys are conducted for a broad range of organizations and perspectives. The goal is to make participation feel inclusive and approachable for people from many different backgrounds and viewpoints.
Ultimately, high-quality survey research depends on both rigorous sampling methods and thoughtful efforts to build trust and participation across diverse populations. We work very intentionally to recruit participants in ways that feel inclusive and welcoming to a broad range of people, not just older or highly educated populations that have historically been more likely to participate in surveys.
At the same time, the survey research field is developing more advanced statistical weighting methods to help ensure that individuals with lower levels of institutional trust are still adequately represented in survey samples.
Ultimately, high-quality surveys require substantial effort to follow best practices, maintain strong response rates, and ensure that the people who participate resemble, as closely as possible, the people who do not. Research continues to show that carefully conducted probability-based surveys can still produce highly representative and reliable results.
By contrast, lower-quality or “quick and easy” survey approaches are more likely to experience representational problems because individuals with lower levels of trust are often less likely to participate. Addressing that challenge has become one of the major focuses of the survey research industry over the past decade. Catherine Batsford: Let’s say I am a researcher designing a study today. What is one change I could make immediately to improve representation and build trust in that study? David Dutwin: That’s a great question. First, using a probability-based approach is extremely important. It is also critical to work with researchers or survey organizations that have a strong track record of conducting high-quality studies and following established best practices.
In the case of survey panels, much of the work involved in creating a representative sample occurs before an individual study even begins. But beyond methodology, communication also matters. Researchers need to think carefully about how they introduce a study and how participants experience the recruitment process.
For example, in the past it was common for surveys to identify themselves immediately as being conducted on behalf of a particular news organization or institution. Today, that approach can sometimes discourage participation because people may have strong feelings—positive or negative—about certain organizations before they even hear what the survey is about.
Similarly, with mailed surveys, best practices have changed over time. Researchers once often labeled envelopes with phrases such as “Survey Enclosed.” Now, many survey researchers prefer very plain envelopes that contain only the address information. The idea is to encourage recipients to open the envelope before making assumptions about the survey or deciding whether to participate.
At the same time, researchers also want to create a sense of connection and relevance once participants engage with the study materials. Don Dillman at Washington State University, who is widely considered one of the foundational figures in survey methodology, emphasized designing recruitment materials that feel relatable and locally meaningful to participants.
There is a tremendous amount of research devoted to understanding how to improve participation and representativeness in survey research. But at a high level, the key is combining rigorous methodology with thoughtful, inclusive communication strategies that make participation accessible and welcoming to the broadest possible population. Dan McLean: I had a question about some of the case studies and research projects you’ve worked on. In several of them, it looks like part of the data comes from the AmeriSpeak panel, while another portion comes from more traditional survey methods or additional data sources.
I was curious about how you determine the balance between those different sources of information and how those blended methodologies work in practice.
One example that stood out to me was the study examining the political spectrum in the United States, where participants were grouped into five different segments based on their perspectives and behaviors. Another example was a study on food allergies, which appeared to use a similar blended approach.
Could you explain how you decide when to supplement panel data with other survey methods and how those different data sources are integrated into the final research? David Dutwin: In a perfect world, we would conduct all research entirely through the AmeriSpeak panel because we know the panel has been recruited using very rigorous methods, including door-to-door recruitment in some cases. However, there are situations where the panel alone cannot provide enough participants for a particular study.
For example, in studies involving relatively small or specialized populations—such as individuals with severe food allergies—we may not have enough eligible respondents within the panel itself. AmeriSpeak includes approximately 50,000 adult participants, but only a subset of those individuals may meet the criteria for a specific study. In addition, participation in any survey is voluntary, and typically only a portion of invited panelists choose to respond.
In cases like that, researchers may supplement the AmeriSpeak panel with additional data collection methods. That can include partnering with other panels, conducting fresh cross-sectional surveys through phone or mail outreach, or incorporating data from nonprobability panels. While nonprobability panels generally have lower methodological rigor, statistical techniques can be used to help reduce potential bias when combining those data sources with higher-quality probability-based samples.
There is another common situation where blended approaches become necessary: studies requiring extremely large sample sizes. For example, some projects aim to produce highly detailed state-level or metropolitan-level estimates and may require tens of thousands of respondents nationwide. Even a large panel such as AmeriSpeak may not be sufficient on its own to reach those numbers quickly enough.
In general, though, the preference is always to rely as heavily as possible on the AmeriSpeak panel because of the quality and rigor of the recruitment methodology. Additional sources are typically used only when necessary to meet the scale or specificity required for a particular research project. Catherine Batsford: We always like to end by looking ahead a bit. If we were to revisit this conversation in five, 10, or even 20 years, what do you think survey technology and public opinion research will look like in the future? David Dutwin: This is a good opportunity to talk about what is probably the biggest topic in survey research right now: artificial intelligence. One of the major developments emerging in the field is the use of synthetic data products.
A synthetic data product is essentially a survey generated by a large language model, such as ChatGPT or Claude, rather than by real human respondents. In practice, that means creating a survey dataset in which AI-generated “respondents” answer survey questions based on patterns learned from existing data.
The conversation around synthetic data reminds me somewhat of the early discussions surrounding nonprobability surveys 15 or 20 years ago. Whenever a new technology emerges, there is often an initial belief that it will completely replace existing methods. Over time, though, researchers usually recognize that new technologies have strengths and limitations, and that there are specific situations where they may or may not be appropriate.
I do not believe major federal statistical surveys will move entirely to synthetic data anytime soon, if ever. Many public policy decisions informed by these surveys involve billions of dollars in government spending and significant public health implications. In those cases, policymakers want the highest-quality human-generated data possible because the cost of conducting rigorous surveys is relatively small compared with the scale of the decisions being made.
That said, AI is likely to become an increasingly important tool in survey research. As with most AI applications, the quality of synthetic data depends heavily on the quality of the underlying data used to train the models. High-quality datasets, such as those produced through rigorous probability-based research, are likely to be essential for developing useful synthetic data systems. Poor-quality input data, on the other hand, will simply produce poor-quality outputs.
Overall, AI is creating an entirely new frontier for survey research and data collection. More organizations are relying on data to inform decisions than ever before, and that trend is likely to continue. AI may accelerate the speed and scale at which data can be generated and analyzed, but it also raises important methodological and ethical questions about accuracy, representation, and reliability.
Another area researchers are beginning to explore is the use of AI-generated interviewers. Instead of using live interviewers for telephone surveys, some organizations are testing whether AI systems could conduct those interviews directly. That is another example of how rapidly the field is evolving. Dan McLean: It’s funny hearing you talk about synthetic data and AI because I have to admit I’ve experimented with my own informal AI “polls.” I’ll ask a chatbot something like, “If you asked 100 experts in this field this question, what would they say?”
The AI usually gives the caveat that it cannot actually survey those experts, but then it will generate a response such as, “Eighty experts would likely say this, while 20 would say that.”
It’s obviously not real survey data, but it does create an interesting kind of pulse check or thought exercise. David Dutwin: Your example really gets to one of the core principles in survey research: fitness for purpose. The appropriate methodology depends on how the data will ultimately be used and what the consequences are if the results are wrong.
If someone casually asks an AI system a question out of curiosity and the answer is imperfect, the consequences are probably minimal. But there are many situations where accuracy matters significantly more.
For example, years ago I worked on a study asking whether people intended to purchase a new car or light truck within the next year. We conducted the research using both probability-based and nonprobability methods. The nonprobability sample produced estimates roughly twice as high as the probability-based sample.
Now imagine an automobile manufacturer using that information to decide whether to invest billions of dollars in building a new manufacturing plant. If the survey overestimates consumer demand, that could lead to extremely costly business decisions based on inaccurate data.
That is why high-stakes policy, public health, and economic decisions still rely heavily on rigorous human-based survey methods rather than synthetic data alone. In those contexts, researchers need the most reliable and representative data possible.
So ultimately, it comes down to fit for purpose. Different tools may be appropriate for different kinds of questions, depending on the level of precision and reliability required. And in your case, Dan, I think your use of AI was relatively low stakes, so we will not shame you for it. Fortunately, billions of dollars were probably not riding on the outcome of your informal AI poll. Dan McLean: I’ve even asked AI questions like whether I should replace my brake pads and when. I’ll phrase it as, “If you asked 100 mechanics, what would they recommend?” It’s interesting to see how it weighs different perspectives and provides general guidance.
David Dutwin: Absolutely.
Catherine Batsford: Well, thank you so much for joining us today. I learn something new every time I hear you speak, and it’s fascinating to better understand everything happening behind the scenes in survey research and public opinion data collection.
And we’ll repeat the message one more time: if you are invited to participate in a survey, consider taking part. It is one way to make your voice heard.
David Dutwin: Thank you for having me on the podcast.
Dan McLean: This was wonderful. Thank you again for your time and for sharing your insights with us today.
We are delighted to have you on our podcast to discuss how representation, participation, and trust influence the data that ultimately shapes public health policy.
First, we’d like to start by asking how you found yourself where you are today. Could you tell us a little about your career path and how you became interested in the field?
David Dutwin: Sure. I think a lot of people probably have unique stories to tell, and mine is similarly unique. I went back to graduate school after working on Capitol Hill. At the time, I thought I was going to become a political speechwriter, so I was taking communication and rhetoric courses.
Finally, in my last semester, I waited until the very last minute to take that dreaded statistics course. I also happened to take a course on public opinion, and it all just clicked. I realized, “Wow, I actually like this statistics stuff, and I’m good at it.” Who knew?
After that, I went on to another school to earn my PhD, focusing entirely on public opinion research and statistics. I really caught the bug, and the rest is history, I guess.
Dan McLean: Before we get too far into the details—and there are a lot of interesting studies and projects you’ve all worked on that we’ll explore today—could you explain the relationship between NORC at the University of Chicago and the University of Chicago?
I also know that NORC is spelled N-O-R-C. Could you elaborate a bit on the origins of the name as well?
David Dutwin: Sure. So a little over eighty years ago, the National Opinion Research Center was founded at the University of Chicago. Like a lot of academic institutions, it was a small survey shop to to do research for professors and whatnot. And about fifty years ago, NORC had gotten so big that the university basically said, it doesn't make sense for you to be solely and universally under our umbrella. And so NORC split off, but basically the agreement was that we would still be affiliated with the university. At least half of our board has to be university administrators, professors, etc.. But we really are quite autonomous. We have a major office in downtown Chicago, as well as one in in DC. The NORC name is funny depending on, the age. Sometimes we were the National Opinion Research Center and other times it's just NORC. Right? And it sort of depends on whatever marketing firm we have at the time recommended would be the best foot forward, essentially. So, yes, we are we are officially NORC as of the, about a decade ago. Dan McLean: AmeriSpeak, which is really that's your domain inside of NORC. Right? How does that fit into to NORC overall?
David Dutwin: So NORC at the University of Chicago generally is a research organization that conducts large-scale federal, state, local, and international research. That work includes surveys and focus groups, but also direct policy research and program evaluation. For example, NORC may evaluate educational systems in countries such as Pakistan and make recommendations for improvement. Similarly, in the US, NORC evaluates public policies and programs and provides evidence-based recommendations.
AmeriSpeak was developed, in part, to diversify that work and expand research opportunities with foundations, academic institutions, and for-profit organizations. To support that effort, NORC established a probability-based survey panel.
Survey research itself has evolved significantly over time. In the 1950s, 1960s, and into the 1970s, door-to-door interviewing was the primary survey method. Later, advances in technology made it possible to randomly sample telephone numbers, and telephone surveys became the predominant approach. In the 1970s, response rates of 30% to 40% were common, meaning a substantial proportion of people contacted by phone would participate in a survey.
As cell phones became more common, survey researchers shifted to cell phone sampling frames. At the same time, the rise of spam calls and telemarketing changed how Americans responded to unknown phone numbers. In the 1970s, an unexpected call might reasonably have been a survey request. Today, most people assume unknown calls are telemarketing or spam. As a result, telephone survey response rates have declined dramatically and are now often closer to 3% to 4%.
That decline has two major consequences. First, surveys become less reliable when only a small percentage of people participate. Second, telephone surveys become much more expensive because researchers must contact hundreds of people to secure a single completed response.
Probability-based panels emerged over the past decade as a new survey methodology designed to address some of these challenges. Researchers recruit participants in advance and ask them to agree to take surveys using scientifically random sampling methods. AmeriSpeak, specifically, uses the United States Postal Service database of residential addresses in the United States as its sampling frame. NORC then randomly selects households and recruits participants through mailed invitations, text messages, phone calls, and door-to-door outreach.
Although door-to-door recruitment is expensive, it remains highly effective. Approximately 30% of people contacted through in-person recruitment agree to participate, compared with response rates closer to 4% or 5% for many other survey methods. Because the panel is built from a random sample, the methodology remains scientifically rigorous and nationally representative. Dan McLean: One of the studies that caught my attention as I was reviewing your work was an annual nationally representative study of young people designed to help inform solutions that support their well-being. The study began in 2024 and is ongoing.
It examines the experiences and perspectives of approximately 1,500 young people across four major areas: sociopolitical divisiveness and healing, civic education, artificial intelligence, and mental health. I was wondering if you could share a little more about the study, including how the AmeriSpeak teen panel helped inform the results.
This feels like an issue that has been top of mind for many people over the last several years, so I’d also be interested to hear what insights you’ve gained from the research. David Dutwin: Sure. AmeriSpeak is really a suite of panels. The primary AmeriSpeak panel includes adults age 18 and older, but we also have a teen panel, a veterans panel, and several state and regional panels that allow us to drill down into more localized data. For example, we have ChicagoSpeaks, as well as panels in Texas and Ohio. We also have an Asian American panel that conducts interviews in four languages in addition to English.
With the teen panel specifically, we worked hard to build it thoughtfully. When someone joins AmeriSpeak, we ask a range of profile questions about topics such as politics, finances, health, education, travel, technology, and social media use. We also collect household information, so we know whether there is a teenager in the home.
Initially, we would immediately ask whether the teenager would also like to join the panel, but participation rates were not very strong. We realized we first needed to establish trust with families. Now, we allow adult participants to complete several surveys so they become familiar with who we are and how the process works. Once that trust has been established, we ask whether their teenager would be interested in joining the separate teen panel.
Today, we have several thousand teens on the panel and approximately 50,000 adults overall. That’s important because, like much of survey research today, there are easier but less rigorous ways to recruit teenagers through what are known as nonprobability panels, or convenience panels. A teenager may see an advertisement on Instagram or another social media platform asking if they want to participate in a survey and simply opt in.
The challenge with those approaches is that they are self-selected and not scientifically random. The participants tend to be teens who are already active on social media and willing to take surveys, which can limit how representative the results actually are. The challenge with those methods is that they are not scientifically rigorous because they rely on self-selection. The teens who participate are generally those who are already active on social media and willing to take surveys.
One of the reasons random sampling is so important is that randomization is really at the core of scientific research. If you randomly divide 500 people into two groups, give one group a placebo and the other a new medication, and the group receiving the medication experiences significantly different outcomes, most people would recognize that as a scientifically valid result.
Survey research works in much the same way. We begin with a national list of residential addresses in the United States and randomly select households to contact. That randomization process helps us produce scientifically rigorous and highly valid results.
We also work carefully to build trust with families. AmeriSpeak has an institutional review board, or IRB, that reviews every project to ensure participants are protected. That becomes especially important when researchers want to study sensitive topics involving teenagers, such as bullying, drug use, sexual violence, or mental health. In some cases, the IRB requires additional parental notification or consent before those surveys can move forward.
Right now, for example, we are conducting research examining the potential harms associated with social media, including bullying and other online experiences affecting teens. Artificial intelligence is also becoming a major area of interest. Researchers increasingly want to understand how teens are using AI and the extent to which some may begin treating AI systems almost like another person or source of emotional support. As a society, it is important that we better understand these behaviors so we can develop thoughtful policies, communication strategies, and support systems that help young people navigate an increasingly complex digital environment.
Catherine Batsford: When you think about public health data, what does that look like from a data collection perspective? Is it the same type of process you’ve been describing, where researchers come to you with specific projects and use your panels to conduct studies? Or are there also broader public health data collection initiatives that NORC works on directly? David Dutwin: NORC conducts a significant amount of public health research, particularly for the federal government. We manage projects such as the National Immunization Survey, along with a number of other studies involving Medicaid recipients and other populations. This work is important because the federal government relies on accurate public health data to understand issues such as disease prevalence, health care access, and how Americans use health insurance and medical services.
Earlier in my career, I was part of a team that conducted a large health interview survey in Massachusetts over many years. The survey helped policymakers better understand how residents were accessing health care and what types of insurance coverage they had. Those findings contributed to the development of what was known at the time as “RomneyCare,” which later informed elements of the Affordable Care Act. It was not the only factor involved, of course, but the survey data played an important role in helping policymakers shape those policies.
I think many people underestimate how central survey research is to public policy and decision-making in the United States and around the world. When most people think about surveys, they think about election polling and questions about who may win an election. In reality, political polling represents only a very small percentage of all survey research conducted in the United States. Most surveys are focused on areas such as public health, health insurance, education, crime statistics, and other issues that help governments and organizations better understand societal needs and develop more effective policies.
That mission of providing nonpartisan, evidence-based insight has always been central to NORC at the University of Chicago.
During the COVID-19 pandemic, for example, AmeriSpeak conducted multiple studies each week to help track rapidly changing public health conditions and public attitudes. Probability-based panels are particularly valuable during fast-moving events such as COVID-19, avian influenza, SARS, or other emerging health concerns because they allow researchers to move very quickly. A large federal survey can take months to design and launch, but probability panels can often begin collecting data within days, providing near real-time insight into rapidly developing situations. Dan McLean: I wanted to go back to something you mentioned several times about building trust with participants and their families. That’s something PRIM&R has spent a great deal of time thinking about in recent years—particularly how to strengthen trust in the scientific research process more broadly.
I was also reading about how your IRB operates and the fact that NORC at the University of Chicago provides certificates of confidentiality to participants in its panels.
When you approach families—and, in some cases, their children—how do you explain the process in a way that helps build trust? And could you also explain what a certificate of confidentiality is and why it matters? It seems especially important if you want participants to feel comfortable speaking openly and honestly about sensitive topics. David Dutwin: At the beginning of every survey, we include introductory language that explains participants’ rights and how their information will be protected. Because AmeriSpeak is a panel, many participants have heard this information before, but we still review it carefully for every study. We want respondents to understand that participation is voluntary, they may decline to answer any question, and the survey is not an interrogation or deposition.
We also make it very clear that their data will remain anonymous and confidential. Once a survey is completed, identifying information is removed and replaced with a case identification number. Researchers working with NORC at the University of Chicago do not receive personally identifiable information unless there is a specific and approved reason to do so, and participants are informed in advance if that situation arises.
Certificates of confidentiality are another important layer of protection. These certificates formally reinforce our commitment to maintaining participant confidentiality, which is especially critical in public health research involving sensitive information. Many surveys involve topics such as health conditions, interpersonal violence, prior victimization, mental health, or other deeply personal experiences.
In those cases, participants need to feel confident that their information will remain protected. Even if a government agency were to request identifying information through legal channels, a certificate of confidentiality issued through the US Department of Health and Human Services provides strong protections against releasing personally identifiable data linked to survey responses.
Those protections are essential to building trust and ensuring participants feel comfortable responding honestly to sensitive questions. Catherine Batsford: So you said before that the populations that you're surveying is fairly randomized, but do you do you feel like there's certain populations that are perhaps asked questions more frequently? Is there a methodology for not having study fatigue by certain populations that researchers could be thinking about? David Dutwin: I would answer that in two ways. First, with regard to survey fatigue, the numbers are generally on our side. There are currently more than 267 million adults in the United States, so the likelihood that someone in the general population is repeatedly contacted for surveys is relatively low.
The exception, of course, is for people who intentionally join a panel such as AmeriSpeak, where participation in surveys is expected. Even then, we work carefully to keep participants in what I would call a “Goldilocks zone.” If people are not engaged often enough, they may lose interest and disengage from the panel. On the other hand, if they receive too many survey requests, they can experience survey fatigue and eventually leave.
To avoid that, we try to maintain a panel large enough that participants are invited to complete no more than about one survey per week. We also study what is known as “panel conditioning,” which refers to the concern that repeated participation in surveys could change how people respond over time.
Some worry that frequent survey participation could lead to lower-quality responses through behaviors we sometimes refer to as the “three S’s”: speeding through surveys, skipping questions, or straight-lining responses by selecting the same answer repeatedly without carefully considering the questions. However, research generally shows that panel conditioning effects are relatively small. In some cases, experienced panelists actually become more attentive and thoughtful respondents.
Even so, data quality remains a major priority. We use quality-control measures to identify patterns suggesting that someone may not be engaging seriously with a survey. For example, if a survey designed to take 15 minutes is completed in only four minutes, we may follow up with the participant and remind them that careful responses are important. Continued low-quality participation can result in removal from the panel.
We also work to keep participants engaged by sharing newsletters and research findings with them. One of the most common reasons people say they join the panel is because they want their voices heard and want to contribute to research that benefits the public. We try to reinforce that connection by showing participants how the studies they contributed to are being used.
More broadly, I do worry about declining survey response rates. Whenever I speak publicly, I encourage people to participate in legitimate surveys when they have the opportunity. Surveys are one of the ways people can contribute their perspectives and experiences to public discussions and policymaking.
Unfortunately, years of spam calls and aggressive telemarketing have made many people understandably skeptical of being contacted by strangers. But when conducted responsibly, surveys remain an important way for individuals to make their voices heard and contribute to research that can inform public policy and improve society. Dan McLean: A couple of things came to mind as you were speaking. First, you mentioned the importance of sharing research results back with participants, which is something we’ve been discussing quite a bit recently. In fact, the director of the National Institutes of Health has emphasized the importance of returning research results to participants across the broader human subjects research enterprise. It was interesting to hear you highlight that as well.
I also wanted to go back to the structure of your panels and how you ensure they remain representative. AmeriSpeak describes itself as a probability-based panel in which households are selected from documented sample lists, often referred to as a “sample frame.”
Could you explain what that means in practice? How are these panels designed to ensure they reflect a broad population rather than a single niche group of participants? David Dutwin: That’s a great question. I think the best place to start is with the fact that there are generally two types of survey panels: probability-based panels and nonprobability panels.
Nonprobability panels are self-selected. For example, someone might see an online advertisement asking if they want to participate in surveys and simply opt in. I was once invited to one because I used Amtrak frequently and belonged to its rewards program. That panel company also partnered with airline loyalty programs such as American Airlines.
The challenge with that approach is representativeness. If you recruit participants primarily through travel rewards programs, you have to ask whether the resulting sample truly reflects the broader population, especially if you are conducting research on travel behaviors or public opinion.
Probability-based panels work differently. With AmeriSpeak, we begin with a national sample frame built from the United States Postal Service database of residential addresses in the United States. That database covers approximately 92% of US households. The remaining households may include people who rely solely on post office boxes or individuals living in extremely rural areas, such as parts of Appalachia or Alaska, where addresses can be more difficult to identify and deliver to.
To improve coverage, NORC supplements the postal database with additional fieldwork to identify households that may not appear in standard address files. As a result, coverage increases to approximately 97% to 98% of US households.
From there, the process relies on random sampling. We randomly select households from that national frame and invite them to participate. The goal is to create a sample that reflects the broader population as accurately as possible.
One of the major challenges in survey research today is declining response rates. Decades ago, telephone surveys often achieved response rates in the 30% range. Today, participation rates are often in the single digits. The critical question becomes whether the people who participate are meaningfully different from those who do not.
Historically, differences in participation could often be adjusted statistically using demographic information such as age, race, gender, and education level. For example, if younger adults were underrepresented in a survey sample, researchers could weight their responses more heavily to better reflect the overall population.
However, researchers now recognize that participation is influenced by more than demographics alone. Trust in institutions and trust in science also appear to affect whether individuals choose to participate in surveys. People with lower levels of institutional trust are often less likely to respond, which can create additional representational challenges.
To address this, we work carefully on recruitment strategies and communication. We aim to use accessible, straightforward language and ensure participants understand that surveys are conducted for a broad range of organizations and perspectives. The goal is to make participation feel inclusive and approachable for people from many different backgrounds and viewpoints.
Ultimately, high-quality survey research depends on both rigorous sampling methods and thoughtful efforts to build trust and participation across diverse populations. We work very intentionally to recruit participants in ways that feel inclusive and welcoming to a broad range of people, not just older or highly educated populations that have historically been more likely to participate in surveys.
At the same time, the survey research field is developing more advanced statistical weighting methods to help ensure that individuals with lower levels of institutional trust are still adequately represented in survey samples.
Ultimately, high-quality surveys require substantial effort to follow best practices, maintain strong response rates, and ensure that the people who participate resemble, as closely as possible, the people who do not. Research continues to show that carefully conducted probability-based surveys can still produce highly representative and reliable results.
By contrast, lower-quality or “quick and easy” survey approaches are more likely to experience representational problems because individuals with lower levels of trust are often less likely to participate. Addressing that challenge has become one of the major focuses of the survey research industry over the past decade. Catherine Batsford: Let’s say I am a researcher designing a study today. What is one change I could make immediately to improve representation and build trust in that study? David Dutwin: That’s a great question. First, using a probability-based approach is extremely important. It is also critical to work with researchers or survey organizations that have a strong track record of conducting high-quality studies and following established best practices.
In the case of survey panels, much of the work involved in creating a representative sample occurs before an individual study even begins. But beyond methodology, communication also matters. Researchers need to think carefully about how they introduce a study and how participants experience the recruitment process.
For example, in the past it was common for surveys to identify themselves immediately as being conducted on behalf of a particular news organization or institution. Today, that approach can sometimes discourage participation because people may have strong feelings—positive or negative—about certain organizations before they even hear what the survey is about.
Similarly, with mailed surveys, best practices have changed over time. Researchers once often labeled envelopes with phrases such as “Survey Enclosed.” Now, many survey researchers prefer very plain envelopes that contain only the address information. The idea is to encourage recipients to open the envelope before making assumptions about the survey or deciding whether to participate.
At the same time, researchers also want to create a sense of connection and relevance once participants engage with the study materials. Don Dillman at Washington State University, who is widely considered one of the foundational figures in survey methodology, emphasized designing recruitment materials that feel relatable and locally meaningful to participants.
There is a tremendous amount of research devoted to understanding how to improve participation and representativeness in survey research. But at a high level, the key is combining rigorous methodology with thoughtful, inclusive communication strategies that make participation accessible and welcoming to the broadest possible population. Dan McLean: I had a question about some of the case studies and research projects you’ve worked on. In several of them, it looks like part of the data comes from the AmeriSpeak panel, while another portion comes from more traditional survey methods or additional data sources.
I was curious about how you determine the balance between those different sources of information and how those blended methodologies work in practice.
One example that stood out to me was the study examining the political spectrum in the United States, where participants were grouped into five different segments based on their perspectives and behaviors. Another example was a study on food allergies, which appeared to use a similar blended approach.
Could you explain how you decide when to supplement panel data with other survey methods and how those different data sources are integrated into the final research? David Dutwin: In a perfect world, we would conduct all research entirely through the AmeriSpeak panel because we know the panel has been recruited using very rigorous methods, including door-to-door recruitment in some cases. However, there are situations where the panel alone cannot provide enough participants for a particular study.
For example, in studies involving relatively small or specialized populations—such as individuals with severe food allergies—we may not have enough eligible respondents within the panel itself. AmeriSpeak includes approximately 50,000 adult participants, but only a subset of those individuals may meet the criteria for a specific study. In addition, participation in any survey is voluntary, and typically only a portion of invited panelists choose to respond.
In cases like that, researchers may supplement the AmeriSpeak panel with additional data collection methods. That can include partnering with other panels, conducting fresh cross-sectional surveys through phone or mail outreach, or incorporating data from nonprobability panels. While nonprobability panels generally have lower methodological rigor, statistical techniques can be used to help reduce potential bias when combining those data sources with higher-quality probability-based samples.
There is another common situation where blended approaches become necessary: studies requiring extremely large sample sizes. For example, some projects aim to produce highly detailed state-level or metropolitan-level estimates and may require tens of thousands of respondents nationwide. Even a large panel such as AmeriSpeak may not be sufficient on its own to reach those numbers quickly enough.
In general, though, the preference is always to rely as heavily as possible on the AmeriSpeak panel because of the quality and rigor of the recruitment methodology. Additional sources are typically used only when necessary to meet the scale or specificity required for a particular research project. Catherine Batsford: We always like to end by looking ahead a bit. If we were to revisit this conversation in five, 10, or even 20 years, what do you think survey technology and public opinion research will look like in the future? David Dutwin: This is a good opportunity to talk about what is probably the biggest topic in survey research right now: artificial intelligence. One of the major developments emerging in the field is the use of synthetic data products.
A synthetic data product is essentially a survey generated by a large language model, such as ChatGPT or Claude, rather than by real human respondents. In practice, that means creating a survey dataset in which AI-generated “respondents” answer survey questions based on patterns learned from existing data.
The conversation around synthetic data reminds me somewhat of the early discussions surrounding nonprobability surveys 15 or 20 years ago. Whenever a new technology emerges, there is often an initial belief that it will completely replace existing methods. Over time, though, researchers usually recognize that new technologies have strengths and limitations, and that there are specific situations where they may or may not be appropriate.
I do not believe major federal statistical surveys will move entirely to synthetic data anytime soon, if ever. Many public policy decisions informed by these surveys involve billions of dollars in government spending and significant public health implications. In those cases, policymakers want the highest-quality human-generated data possible because the cost of conducting rigorous surveys is relatively small compared with the scale of the decisions being made.
That said, AI is likely to become an increasingly important tool in survey research. As with most AI applications, the quality of synthetic data depends heavily on the quality of the underlying data used to train the models. High-quality datasets, such as those produced through rigorous probability-based research, are likely to be essential for developing useful synthetic data systems. Poor-quality input data, on the other hand, will simply produce poor-quality outputs.
Overall, AI is creating an entirely new frontier for survey research and data collection. More organizations are relying on data to inform decisions than ever before, and that trend is likely to continue. AI may accelerate the speed and scale at which data can be generated and analyzed, but it also raises important methodological and ethical questions about accuracy, representation, and reliability.
Another area researchers are beginning to explore is the use of AI-generated interviewers. Instead of using live interviewers for telephone surveys, some organizations are testing whether AI systems could conduct those interviews directly. That is another example of how rapidly the field is evolving. Dan McLean: It’s funny hearing you talk about synthetic data and AI because I have to admit I’ve experimented with my own informal AI “polls.” I’ll ask a chatbot something like, “If you asked 100 experts in this field this question, what would they say?”
The AI usually gives the caveat that it cannot actually survey those experts, but then it will generate a response such as, “Eighty experts would likely say this, while 20 would say that.”
It’s obviously not real survey data, but it does create an interesting kind of pulse check or thought exercise. David Dutwin: Your example really gets to one of the core principles in survey research: fitness for purpose. The appropriate methodology depends on how the data will ultimately be used and what the consequences are if the results are wrong.
If someone casually asks an AI system a question out of curiosity and the answer is imperfect, the consequences are probably minimal. But there are many situations where accuracy matters significantly more.
For example, years ago I worked on a study asking whether people intended to purchase a new car or light truck within the next year. We conducted the research using both probability-based and nonprobability methods. The nonprobability sample produced estimates roughly twice as high as the probability-based sample.
Now imagine an automobile manufacturer using that information to decide whether to invest billions of dollars in building a new manufacturing plant. If the survey overestimates consumer demand, that could lead to extremely costly business decisions based on inaccurate data.
That is why high-stakes policy, public health, and economic decisions still rely heavily on rigorous human-based survey methods rather than synthetic data alone. In those contexts, researchers need the most reliable and representative data possible.
So ultimately, it comes down to fit for purpose. Different tools may be appropriate for different kinds of questions, depending on the level of precision and reliability required. And in your case, Dan, I think your use of AI was relatively low stakes, so we will not shame you for it. Fortunately, billions of dollars were probably not riding on the outcome of your informal AI poll. Dan McLean: I’ve even asked AI questions like whether I should replace my brake pads and when. I’ll phrase it as, “If you asked 100 mechanics, what would they recommend?” It’s interesting to see how it weighs different perspectives and provides general guidance.
David Dutwin: Absolutely.
Catherine Batsford: Well, thank you so much for joining us today. I learn something new every time I hear you speak, and it’s fascinating to better understand everything happening behind the scenes in survey research and public opinion data collection.
And we’ll repeat the message one more time: if you are invited to participate in a survey, consider taking part. It is one way to make your voice heard.
David Dutwin: Thank you for having me on the podcast.
Dan McLean: This was wonderful. Thank you again for your time and for sharing your insights with us today.
Research Ethics Reimagined guests are esteemed members of our community who generously share their insights. Their views are their own and do not necessarily reflect those of PRIM&R or its staff.