Crowdsourced Data Collection for Public Health: A Comparison with Nationally Representative, Population Tobacco Use Data

Introduction

Internet crowdsourcing, defined as a “distributed problem-solving and production model that leverages the collective intelligence of online communities,” is a tool with potential to address public health challenges.¹ Crowdsourcing offers an efficient way to obtain information from online respondents more quickly than some conventional data collection tools. Crowdsourcing applications entail varying involvement from participants, including answering questions (e.g., surveys); providing feedback on concepts (e.g., policies, programs); coding data (e.g., images); and creating user-generated content (e.g., communication messages).¹ Researchers are increasingly using crowdsourcing data collection, particularly in social and behavioral sciences.²

Crowdsourcing has value because data collection is efficient and relatively low cost and participants are readily available without geographic constraints.^1,3,4 Examples of research using crowdsourcing include tobacco control,^5–8 skin cancer prevention,⁹ and sexual behavior.¹⁰ Research using crowdsourcing includes observational studies to characterize specific constructs such as health beliefs, and correlational investigations of how exposures such as health messaging relate to outcomes such as beliefs or behavior.^5–10 Peer-reviewed papers using a single crowdsourcing platform (Amazon Mechanical Turk) increased from 61 in 2011 to 1,120 in 2015.² Increasing interest has spurred development of methodological tools for researchers,¹¹ and crowdsourcing is appearing in funding opportunities from agencies such as the National Institutes of Health.

Crowdsourcing platforms can replicate data collected using validated behavioral measures and tasks,^12,13 and crowdsourcing samples may provide greater demographic diversity than traditional convenience samples (e.g., college students).^12,14 There are concerns about generalizability since crowdsourced participants are those with technology access who are motivated to engage in research, and because there may be relatively small numbers of individuals in crowdsourced participant pools who meet specific eligibility criteria at any given time.^1,15 There is also some evidence indicating that crowdsourced samples may differ from the population on measures relevant to public health research, such as political beliefs.¹⁶

Several recent studies have used crowdsourcing to examine questions aimed at informing Food and Drug Administration (FDA) tobacco regulations.^5–8,17 Such an efficient data collection approach has potential value for tobacco regulatory science because FDA is charged with supporting regulations using population data and often such data need to be generated within a short timeframe to inform regulations.¹⁸ Crowdsourcing also provides the capability to reach priority groups to inform tobacco regulations, such as tobacco users and nonusers and specific demographic groups.¹⁸ Although crowdsourcing is increasingly used for tobacco research, as noted above prior studies have used crowdsourcing for topics ranging from skin cancer prevention⁹ to sexual risk behavior,¹⁰ and its use is increasing overall.² Empirical evidence on how crowdsourcing can be used to inform tobacco regulation as a case study can guide research in other public health domains as well.

This study empirically examined the potential contributions of crowdsourced data in public health research by comparing crowdsourced data to nationally representative U.S. survey data. The study focuses on tobacco use as an example because deidentified, nationally representative data from the National Adult Tobacco Survey (NATS) are publicly available from the U.S. Centers for Disease Control and Prevention (CDC).¹⁹ This study additionally focused on young adults ages 18 to 30 years because they are defined as a priority population for tobacco research²⁰ and can be accessed through crowdsourcing platforms where participation is limited to adults ages 18 and older. Our aim was to determine if crowdsourced data provides similar estimates of demographics, tobacco use behavior, and tobacco risk perceptions by comparing data collected through Amazon Mechanical Turk to the NATS. Additionally, we aimed to determine if correlational analyses in crowdsourced data are similar to population data, drawing from research indicating exposure to tobacco warning labels can affect risk perceptions.^5–8 Our study focused on these specific measures because monitoring population-level trends in tobacco use behavior, perceptions, and exposure to interventions such as warning labels are critical to tobacco regulatory decision-making.¹⁸ We tested the study aims by comparing data collected through Amazon Mechanical Turk to the NATS on demographics, tobacco use behaviors and perceptions, and exposure to tobacco warnings using parallel measures and sample weighting strategies for crowdsourced data.

Methods

Results

Discussion

This study compared crowdsourced data on young adult tobacco use to a population survey using parallel measures and weighting procedures for crowdsourced data. The findings highlight strengths and limitations of crowdsourced data that can be considered when conducting and interpreting crowdsourced research and areas of future research.

We recruited a crowdsourced sample of nearly 4,000 young adults in less than one week. After weighting, the crowdsourced sample was comparable to the NATS data on demographics involved in the weighting process, as expected. The samples were weighted to be comparable on smoking prevalence, and both reflected the general finding of prior studies that smokers are more likely than non-smokers to endorse non-cigarette tobacco use.^27,28 We found comparably high proportions of young adults perceived smoking to be harmful and addictive and similar patterns where more frequent exposure to tobacco warnings was reported among current users than nonusers in both samples. Finally, we observed similar trends overall that participants were more likely to perceive smoking is harmful with greater reported exposure to cigarette warning labels. There were also notable differences between the samples. Household income diverged between the samples and the proportion of respondents reporting use of most non-cigarette tobacco products was greater in the crowdsourced sample. The NATS employs a national sampling frame and methods to ensure correct sampling of cellular phone-only households.²¹ Compared to U.S. Census Bureau estimates, NATS’ young adult income distribution appears representative.²⁹ It is possible income distributions differed between samples because individuals with lower household incomes are overrepresented in crowdsourcing platforms or due to differing measurement approaches used in the NATS and crowdsourced data. A greater proportion of NATS respondents declined reporting household income, which may reflect different comfort in responding to an anonymous online survey versus a governmental telephone interview. ³⁰Weighted income distribution in the crowdsourced sample was similar to research in an online nationally representative young adult sample, further supporting this possibility.³¹ Prior studies have also shown that crowdsourced samples tend to over-represent respondents with lower incomes, a finding consistent with our data.^14,32

Research comparing telephone probability samples, Internet probability samples, and Internet convenience samples suggests telephone and Internet probability samples are more demographically representative of the U.S. population even with post-hoc weighting of non-probability data.³³ However, non-probability Internet sampling was also more likely to draw participants familiar with the survey subject.³³ This is important with respect to our finding that non-cigarette tobacco use was more prevalent in the crowdsourced sample, and this difference that was even more pronounced among current smokers. This suggests framing of the crowdsourced survey (a survey on tobacco) may have contributed to the higher proportion of crowdsourced respondents endorsing non-cigarette tobacco use by making it more likely that tobacco users would participate. This could be addressed by framing descriptions for crowdsourced data collection more generally (e.g., a survey on health) to reduce the likelihood that those familiar with the survey subject will be more likely to respond. Research examining effects of different framing of crowdsourced studies is an important avenue for further study.

Although we observed similar patterns in perceived harms of smoking, exposure to tobacco warnings, and correlations between these two variables among smokers and non-smokers in the samples, some estimates differed significantly. Generally, in the crowdsourced sample reported exposure to tobacco product warnings was lower among tobacco users and the correlation between exposure frequency and perceived harms of smoking was attenuated. This finding is difficult to interpret given evidence that text-only tobacco warning labels used in the U.S. are infrequently attended to and have limited effects on tobacco perceptions.³⁴ Differences in mode of survey administration across the studies may have also affected participants’ responses—for example a telephone survey may lead to more socially desirable reporting of warning exposure and perceptions than an anonymous, online survey. This finding is also consistent with research demonstrating variance in perceptions of smoking across probability and crowdsourced samples⁸ and may be a feature of greater measurement error in non-probability samples.³⁵

Our findings should be interpreted in light of limitations. We focused on young adult tobacco use illustratively; additional research should determine the strengths and limitations of crowdsourced data collection in other populations and for other public health issues. Data collection relied on self-reported measures administered in English only. Although all measures used have been validated they are subject to potential reporting biases. As noted above, it is also possible the study findings were influenced by different modes of survey administration. Post-hoc weighting of crowdsourced data was informed by the NATS’ approach but relied on a limited set of variables. Weighting based on other variables (e.g., geographic location) may improve crowdsourced data’s representativeness. Population-level trends in tobacco use may have contributed to differences observed between the data sets. NATS data were collected 3 years earlier; the prevalence of young adult non-cigarette tobacco product use, including electronic cigarettes and waterpipe, increased modestly during this period.^22,25 The prevalence of non-cigarette tobacco use in other recent population surveys of U.S. adults was similar to the NATS,³⁶ and increases in non-cigarette tobacco use prevalence were observed over among U.S. youth during this time, some of whom may have reached young adulthood and contributed to observed differences.³⁷ Given the time lag between federal agencies’ collection of population survey data and its release to the public this limitation is difficult to directly address. However, the magnitude of population increases in prevalence is insufficient to account for observed differences between the NATS and crowdsourced samples. This suggests the possibility that tobacco users may be overrepresented in crowdsourced data. Finally, our approach does not account for all potential unmeasured confounders that may have affected observed differences between the samples. As evidence on the characteristics of crowdsourced samples grows, this will help to better understand the potential influence of such confounding variables across studies.¹⁶ It will be important to address these methodological limitations through continued comparisons of crowdsourced and population-based data with careful attention to the timing of data collection, implementing additional measures of confounding variables, and expanding analyses to other population data sources to better understand if the findings observed were specific to the NATS.

Despite these limitations, our study provides insights into the strengths and limitations of crowdsourced data related to tobacco and potentially other public health domains. Even after post-hoc sample weighting, crowdsourced samples may differ demographically and may not provide generalizable estimates of behaviors such as tobacco use relative to population data. These findings suggest crowdsourced data collection likely provides an efficient means of gathering data at the stages of idea development, designing public health interventions, and ascertaining feedback on alternative approaches, as is illustrated in other recent studies as well.^5–7,17 Our data and other recent work⁷ also suggest crowdsourcing data collection may be valuable when targeted samples are desired, given for example the relatively higher prevalence of tobacco users in the crowdsourced data. However, for the purposes of generalizing to larger populations the adequacy of this sampling method is in need of further testing. This includes research to understand the potential value of crowdsourced data for purposes that require greater generalizability, such as monitoring of population trends in behaviors and other outcomes. Continued comparisons of crowdsourced data to population data sources including the NATS and other nationally representative, such as the Behavioral Risk factor Surveillance Surveys or National Health Interview Surveys, will be informative. Additional research on methodological approaches to improve sampling (e.g., using quotas) and post-hoc adjustments through weighting and other means will advance this research area as well.

Supplementary Material