Non-inclusive language in human subjects questionnaires: addressing racial, ethnic, heteronormative, and gender bias

Abstract

Background

Questionnaires for research that involve diverse populations require inclusive language. There are few guidelines to assist researchers in minimizing social and cultural biases in data collection materials; such biases can result in harm and negatively impact data integrity.

Methods

We describe an approach to evaluating language in data collection forms reflecting racial, ethnic, heteronormative, and gender bias using the Environmental influences on Child Health Outcomes (ECHO)-wide Cohort Study (EWC) as a case study. The 245 data collection forms were used by 69 cohorts in the first seven years of the (ECHO)-wide Cohort Study (EWC). A diverse panel of reviewers (n = 5) rated all forms; each form also was rated by a second student. Items identified as reflecting bias were coded as to the specificity of the bias using nine categories (e.g., racial bias, heteronormative assumptions) following whole panel discussion. We provide recommendations for conducting inclusive research to the scientific community.

Results

Thirty-six percent (n = 88) of the data collection forms were identified as containing biased language. In total, 137 instances of bias were recorded, eight instances of racial or ethnic bias, 56 instances of bias related to sex, gender identity and sexual orientation and 73 instances of bias related to universal assumptions. Seventy-three percent (n = 64) of forms with biased language are validated measures. The review culminated in recommended revisions to forms used by ECHO and the general scientific community.

Conclusion

Adverse health outcomes disproportionately affect marginalized populations. Utilizing culturally and socially conscious research materials that are inclusive of various identities and experiences is necessary to help remediate these disparities. Our review finds compelling evidence of bias in many widely used data collection instruments. Recommendations for conducting more inclusive science are discussed.

Introduction

Culturally and socially conscious research has the potential to contribute to a nation free of inequitable health outcomes among individuals from racial, ethnic, sexual and gender orientation groups in the minority who have experienced and continue to experience discrimination and marginalization by others (hereafter, “marginalized groups”) [1]. However, there are few guidelines to assist researchers in conducting inclusive science. More specifically, there is an urgent need for utilization of data collection materials that are both methodologically sound and contain minimal bias. Historically marginalized groups are more likely to experience preventable differences in the burden of disease, injury, violence, and related harm [2,3,4]. These disparities are pervasive across a range of conditions, including diabetes, hypertension, obesity, asthma, heart disease, cancer, and preterm birth [4]. For example, Black persons continue to have higher rates of morbidity and mortality than white persons across most physical health indicators [5, 6]. Hispanic individuals also have excessive morbidity, particularly diabetes, hypertension, and certain cancers [7,8,9]. Members of lesbian, gay, bisexual, transgender, queer/questioning, intersex, asexual/agender (LGBTQIA +) communities experience worse health outcomes in several areas, including mental health, sexual health, and health care access including preventive health care services [10,11,12].

These differential health outcomes are not caused by innate biological or genetic differences among groups, rather, they are physiological responses to complex mechanisms derived from discrimination, allostatic load, and intergenerational exposures across the life course [3, 13]. Since these marginalized groups do not represent innate biological differences, observed disparities in health outcomes are modifiable and preventable. Historically, scientists have justified the separation and categorization of humans, leading to erroneous belief in social hierarchies in which specific lives are seen as less valuable than others [2]. The Tuskegee Syphilis Study and the case of Henrietta Lacks exemplify egregious practices that have been made in scientific research studies [14, 15]. Modern day examples of unethical research practices include an experiment where Black or Hispanic boys were given intravenous doses of the now banned diet drug fenfluramine and the misuse of blood samples from Havasupai tribal members [16, 17]. Such violations of informed consent contribute to ongoing mistrust of research institutions among marginalized groups. Ethical concerns in research still exist, especially related to the more subtle but very real impacts of bias, associations, and stereotypes that encourage prejudice.

Reducing health disparities and inequalities among ethnically and gender-diverse populations is a top public health priority in the United States [4]. Thus, socially and culturally conscious research methodology and protocols are critical for capturing the mechanisms by which these differential health outcomes occur. Moreover, culturally sensitive participant-facing materials should be used to enhance the recruitment and retention of marginalized groups [18].

This paper aims to describe approaches to identifying instances of bias in data collection instruments and encourages scientists to consider how administering questionnaires containing biased language will affect their participants, data, and results. Our objectives are to: (1) Describe language reflecting racial, ethnic, heteronormative and gender bias in participant-facing data collection forms used in the first seven years of the Environmental influences on Child Health Outcomes (ECHO) Program as a case study. (2) Provide language recommendations for conducting inclusive research to the scientific community.

While there have been several small studies within a specific discipline or health outcome area that have evaluated data collection instruments for inclusivity or use in diverse populations, to our knowledge, this is the first comprehensive review of data collection forms widely used in epidemiologic research studies for racial, ethnic, heteronormative, and gender bias [19,20,21]. Results of the review process will inform the greater scientific and public health community of potential sources of bias in human subjects research protocols and recommended revisions will be provided to reduce harm in data collection.

Methods

The ECHO cohort study is a nationwide study of more than 50,000 children and their families [22]. This study focuses on five primary childhood outcome areas: neurodevelopment, airways, obesity, pre-, peri-, and postnatal and positive health outcomes and examines the effects from a multitude of exposures in utero and during childhood, including the physical, chemical, societal, psychosocial, behavioral and biological environments. Given the size and scope of the ECHO cohort and its objectives, the ECHO Diversity, Equity, Inclusion and Accessibility (DEIA) Working Group examined the protocol for inclusivity as part of its charge.

The ECHO cohort study collected new data using the standardized ECHO-wide Cohort data Collection Protocol (EWCP) and enrolled new pregnant and/or child participants from extant cohort studies [22]. Western IRB (WIRB), now WCG™, the contracted single institutional review board approved version 1.2 of the EWCP in 2019 which was collaboratively developed by all cohort and component principal investigators and then each local IRB also approved the protocol for each individual cohort site in the first Cycle of ECHO. The following individual IRBs approved the ECHO protocol for the first Cycle: Baystate Medical Center Institutional Review Board, Boston Children's Hospital Committee on Clinical Investigations, Boston Medical Center IRB, Care New England-Women & Infants IRB, Children’s Hospital of Philadelphia IRB, Cincinnati Children’s Hospital Medical Center IRB, Columbia University Human Research Protection Office IRB, Drexel University Office of Research, East Carolina University Office of Integrity and Compliance, Emory University IRB, Harvard Pilgrim Health Care Institutional Review Board, Henry Ford Health System IRB, Human Subjects Division at the University of Washington, Indiana University IRB, UNC IRB and the Office of Human Research Ethics, Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University IRB, Kaiser Permanente of Southern California IRB, Lifespan IRB, Marshfield Clinic Research Foundation's IRB, Michigan State University IRB, New York State Psychiatric Institute Office of Research Oversight, New York University School of Medicine IRB, NYU Medical Center/Bellevue Hospital Institutional Review Board, Partners Human Research Committee, Phoenixville Hospital Office of Regulatory Affairs, Seattle Children's Hospital IRB, Spectrum Health IRB, Tufts Medical Center IRB, University of California San Francisco IRB, University of Chicago IRB, University of Florida Jacksonville IRB, University of Illinois at Urbana-Champaign, University of Oregon Committee for the Protection of Human Subjects (CPHS), University of Pittsburgh IRB, University of Puerto Rico Human Research Subjects Protection Office, University of Rochester, University of Southern California IRB, University of Tennessee IRB, University of Utah IRB, University of Washington Office of Research Human Subjects, University of Wisconsin Health Sciences IRB, Vanderbilt University Human Research Protections Program, Wake Forest University Health Sciences IRB, Washington University Human Research Protection Office, William Beaumont Hospital Human Investigation Committee, and WIRB-Copernicus Group (WCG) IRB. WCG™ is the sole IRB for the second Cycle of ECHO and no individual level IRBs are involved.

A total of 260 data collection forms were part of the first 7 year cycle of the EWCP [23] and were used by 69 cohorts nationwide. Fifteen data collection instruments were excluded from our analysis because they involved computer adaptive testing (CAT) or were used solely for data abstraction purposes by study personnel (e.g., forms for capturing data from the electronic medical record). The remaining participant-facing data collection forms (n = 245) comprised the final review (Supplemental Table). A diverse panel (n = 5) of volunteer interns not affiliated with the ECHO Program but completing internships with one of the ECHO cohorts completed the review of all data collection forms. The interns were selected as part of competitive internship programs for undergraduate and graduate students from groups underrepresented in the sciences at the University of Southern California and California State University, Northridge (a federally designated Hispanic Serving Institution) and were asked to participate in this review for protocol inclusivity. Members of the panel represented a broad age range (20–30 years old), an age range similar to that of the caregiver participants in ECHO. The panel represented Hispanic, African American, LGBTQIA +, neurodiverse, disabled, and socioeconomically disadvantaged communities. Several of the panelists self-identified with two or more of these identities. While members of the panel may not reflect lower education levels, data collection forms were reviewed to ensure an appropriate reading level prior to implementation in the protocol.

The panelists met with members of the ECHO DEIA Working Group and were provided an overview of the protocol as well as the objective of the review. Importantly, the review was agnostic and panel-led which was an important consideration for the DEIA Working Group as the panel represented members of marginalized groups who had limited input on the selection of the protocol instruments when the ECHO protocol was established. The panelists conducted a brief review of the literature to identify current definitions and examples of heteronormative, racial, ethnic and ableist bias. The panel then met to examine definitions and examples of these types of bias before reviewing the data collection protocol. The definitions and examples that guided the review are provided in Table 1. Following this meeting, protocol data collection forms were then evenly distributed to each panelist for review and annotation. In the first iteration of the review, members examined protocol instruments for biased language, keeping in mind that the focus of the review was on language that is non-inclusive or potentially harmful (i.e., language that is offensive or otherwise increases burden for specific groups). At least two members of the team independently reviewed each form and extracted items they considered to be biased, logging their findings in a shared spreadsheet.

Table 1 Definitions and examples of bias

After the initial review, the panel met and collectively determined that the following nine designations were appropriate for coding forms: racial bias, non-inclusive anthropometric measures, non-inclusive personal care product options, gender bias, heteronormative assumptions, transgender bias, pronoun usage, restricted response options for sex, and assumption about family structure. Data collection forms and individual items were then coded with one or multiple of these designations in the second iteration of the review. Further details regarding these designations are presented in Table 2. In total, all forms were reviewed a minimum of three times. A final meeting was held to discuss and come to consensus on each item flagged in the data collection instrument review and the final recommendations were provided to the DEIA Working Group and the larger consortium.

Table 2 Identified problematic areas and examples from the ECHO program review

Results

A comprehensive list of all reviewed forms with their respective categorizations is shown in the Supplemental Table. Table 2 lists each of the nine subcategories of bias (racial or ethnic bias, non-inclusive anthropometric measures, non-inclusive personal care product options, gender bias, heteronormative assumptions, transgender bias, pronoun usage, restricted choices for sex, assumptions about family structure), their description, and an example from a reviewed form. Table 3 provides a list of all identified problematic data collection forms with corresponding recommendations for inclusive language.

Table 3 Identified problematic data collection forms with recommendations for inclusive language

Thirty-six percent (n = 88) of the data collection forms were identified as containing biased language in one of the nine previously outlined subcategories of bias (Table 3). Of the 88 total forms identified as containing biased language, 33 contained multiple distinct instances of bias (e.g., racial bias AND transgender bias). In total, 137 instances of bias were recorded. Eight instances of racial or ethnic biases (e.g., language that targets or is not representative of specific racial or ethnic groups) were discovered. Fifty-six instances of bias related to sex, gender identity and sexual orientation (e.g., gender bias, restricted choices for sex, heteronormative assumptions, transgender bias) were discovered. Seventy-three instances of universal assumptions (e.g., binary pronoun usage, assumption about family structure, non-inclusive anthropometric measures, non-inclusive personal care product options). We did not find instances of ableist bias in any of the forms. A flow-chart representation of these results is provided in Fig. 1. Seventy-three percent (n = 64) of forms with biased language were validated or published measures used in multiple studies.

Fig. 1

Analysis of Biased Language in Data Collection Forms: Subcategories and Instances. Of the 260 total data collection forms used in the initial 7-year cycle of the ECHO-wide Cohort Data Collection Protocol (EWCP), 15 were excluded due to computer adaptive testing (CAT) or exclusive data abstraction use. This left 245 participant-facing forms for review. Among these, 88 forms exhibited biased language across nine subcategories, with 33 containing multiple instances, totaling 137 instances. Eight instances of racial or ethnic bias, 56 instances related to sex, gender identity, and sexual orientation bias, and 73 instances of universal assumptions were identified

Discussion

The ECHO Cohort is a large, highly diverse research program that brings together pregnant persons, caretakers, children, and teenagers of different backgrounds from across the United States. However, this review demonstrates the prevalence of potentially harmful data collection forms in large consortia such as the ECHO Program. Researchers must realize the length and time commitment required to complete a questionnaire is just one of many factors that contribute to participant burden. We assert that biased language evident in many widely used data collection instruments likely contributes to participant burden.

Utilizing inclusive terminology in data collection is complex due to the historical, social, and political implications of labels and verbiage [1]. Moreover, the iterative nature of language requires regular investigation to identify appropriate wording. We limit our discussion here to the most pervasive examples of bias found in the review and general guidelines for using inclusive language in participant facing data collection instruments.

Researchers should strive to avoid language that targets specific racial or ethnic groups or language that is not representative of a diverse participant population. For example, the Dietary Screener Questionnaire (DSQ), which was developed for inclusion in the 2009–2010 National Health and Nutrition Examination Survey, asks participants how often they eat ‘Mexican-type salsa’ [26]. In contrast, items that ask about foods such as ‘pizza’ or ‘spaghetti’ do not denote them as ‘Italian-type’ [26]. Though we acknowledge differences in nutrient profiles between different types of salsas, we recommend that ‘Mexican-type’ be removed from the question stem unless all food groups on the form are categorized by nationality.

In addition to avoiding language that is biased against certain racial or ethnic groups, researchers should consider where data collection tools fail to accurately capture the diversity of their study population. For example, an ECHO form created to collect personal product usage and chemical exposures fails to designate options for textured hair care products including hair relaxers, oils, lotions, gels, and leave-in conditioners [23]. Some of these products, which many individuals use daily, contain parabens, phthalates, and other chemicals that are known endocrine disruptors [29]. Thus, a significant source of chemical exposure is potentially missed because researchers do not use a form that lists a variety of personal care products, particularly those for textured hair.

Furthermore, many data collection instruments that measure race or ethnicity use the categories from the U.S. Census [1]. While these designations have improved with time, some categories, including Middle Eastern or Northern African, are not currently census options [30]. Burlew et al. point out that while some subgroups of heritage are differentiated, including Asian (Chinese vs. Japanese), American Indian/Alaska Native (tribe), Latino (Puerto Rican vs. Cuban), Black individuals who have diverse nationalities are not distinguished [1]. Therefore, U.S.-born Black, Caribbeans/West Indian, and South African persons, would all mark the same response on the Census [1]. As such, researchers must be mindful in collecting race/ethnicity in their studies, ensuring that they use proper measures given the participant population.

Use of inclusive terminology is of particular importance in understanding health inequities faced by lesbian, gay, bisexual, transgender and queer (LGBTQIA +) persons [31]. Without measures that facilitate an accurate understanding of an individual’s gender identity and sexual orientation, researchers may propagate heteronormative ideology and miss relevant differences in participant life experiences [24]. While we acknowledge that increasing the number of categories in an individual variable can produce statistical challenges—especially in smaller studies—collecting data on participant gender identity and sexual orientation is essential for researchers to identify health disparities that exist within or across LGBTQIA + communities and generate actionable information [32]. Moreover, the ability to confidently state which communities a study’s results may or may not generalize to requires careful characterization of the study’s population. However, our review found many data collection instruments do not offer gender designations outside of binary male/female (Table 3). Respondents may need guidance to clarify sexual orientation and gender identity terminology, thus specific definitions of sexual and gender minority identities should be provided [32]. Additionally, “unsure” answer choices could be made more specific by providing options including “I have not decided what my gender is" or “I am not sure which gender I am attracted to” and “I am not sure what this question is asking”. Allowing participants to mark “prefer not to answer” is also important. In forms in which caretakers report on their child’s gender identity researchers may learn from asking how the caretaker describes their child’s gender in addition to how parents think the child would describe themselves. For example, “When it comes to describing the child’s gender, what word or words are closest to what you use?” – or just “How do you describe the child’s gender?”. In age groups where information from the child is not collected, researchers may consider asking caretakers questions to elicit the child’s view of their gender, i.e., “does the child describe their gender the same way that you do?”, and if no, asking “When it comes to describing the child’s gender, what word or words are closest to what the child uses?’.

Emerging tools for conceptualizing and measuring gender identity in children utilize a dual identity approach [33]. Young children vary in how they view themselves in relation to other genders, such that all gender groups serve as important reference groups when measuring gender identity [33]. Thus, using a multidimensional gender identity measure is important to understand the implications of gender identity as it relates to health and development. Without a dual approach, researchers are unable to capture distinctions between children who feel very similar to their own gender from those who feel similar to other genders [33]. Martin et al. propose a graphical measure that asks how similar children feel to girls and boys on a range of domains of gender typing [33]. Evidence from this measure suggests that children as early as first grade are able to use both own- and other- gender comparisons to inform gender identity [33]. Implementation of a dual identity approach for measuring gender identity in children allows for assessment of a fuller range of gender identities and facilitates an understanding of the development of gender identity in children.

Implementation of gender-inclusive pronouns within data collection instruments can benefit participants and help foster welcoming and affirming research environments [34]. The review herein found binary pronoun usage to be the most common instance of problematic language, with 68/245 (28%) of reviewed forms using binary pronouns. We recommend changing wording in which binary pronouns are used and replacing with inclusive pronouns (they/them) or specifically referring to the subject of the question (the child, the biological mother, etc.).

Questionnaires designed to elicit familial relationships require revision to capture the growing diversity of American families. Recent data indicates that 26% of children live with one parent, up from 22% in 2000. Sixteen percent of children live in “blended families- households with a stepparent, stepsibling, or half-sibling [35]. Though counting same-sex couples in the U.S. remains a challenge, research indicates a growing number of children are being raised by same-sex parents [36]. Despite these figures, data collection instruments that fail to accommodate diverse family structures are still in wide use. The Alabama Parenting Questionnaire was developed in 1991 to be administered to children for the purpose of evaluating familial relationships [28]. One question on this form asks the respondent about both their mom and their dad, neglecting that the child may live with a single parent, other caretaker, or have same sex parents. The form also assumes a dual parent household in other questions by using the wording ‘parents’ rather than parent(s). Our panel concurred that this language is outdated and potentially insufficient for assessing familial relationships and even harmful to children whose family structures are not represented in the questionnaire.

In 2021, a Subgroup on Individuals with Disability within the NIH Working Group on Diversity published a set of recommendations for disability inclusion and most notably recommended that there is a critical need to expand efforts to include perspectives of individuals with disabilities in the scientific workforce and in research studies [37]. Our panel did not find any instances of ableism in the data collection instruments; however, this is in part due to the primary focus on evaluating language in data collection instruments. The ECHO data collection protocol does not include translation or accommodations for visually- or hearing-impaired individuals; however, there is guidance provided to cohorts for best practices in consenting individuals with hearing or vision impairments or those with limited literacy levels.

Conducting inclusive science is essential to maintaining robust, informative, and diverse research cohorts. Researchers must strive to be as accurate as possible when collecting data to promote health equity among underserved and under-researched populations. Data that is collected without cultural consideration may result in inaccurate assumptions that do little to mitigate health disparities in marginalized communities [1]. If participants are not able to accurately reflect their experiences and identity, because options don’t exist for them to choose, then data quality will undoubtedly suffer. Thus, researchers should avoid boxing participants into an option that does not accurately reflect who they are whenever possible. Scientists must uphold the highest possible standard for research, meaning that inclusivity in the research process is paramount. This way, participants are not being harmed in the research process — which is supposed to mitigate health disparities, not reinforce them.

To our knowledge, this is the only review that has evaluated a protocol of this magnitude for racial, ethnic, heteronormative and gender bias. This kind of review is essential for continually improving the quality of data collection, as participants can more accurately represent their identities when inclusive language is used. In turn, researchers are better equipped to analyze diverse populations, which is a public health priority. The evolving nature of language, particularly language that emerges as we adapt to a diversifying society, necessitates regular reviews of this kind. We emphasize that the recommendations included in this work are meant to be flexible and iterative, to be built upon by the scientific community as new terminology emerges or current language becomes obsolete [38].

This review culminated in recommended revisions to data collection instruments used by the ECHO Program and many of these revisions were adopted into the protocol for Cycle 2 of the program. Additionally, several publishers were approached with suggested revisions for proprietary measures and they showed a generally positive willingness to implement the recommended changes.

However, there are several limitations in our approach. We acknowledge that the committee was not selected specifically for this purpose and did not have representation from all marginalized groups. There was limited representation from the disability community. The group also was primarily tasked with reviewing the protocol as it was provided to them, rather than to make recommendations for what was not included in the protocol or alteration in how the protocol could be delivered.

Ensuring adequate representation of children and families who have been shown to experience higher levels of persistent environmental stressors and/or who are at higher risk for suboptimal health, such as individuals who are gender non-conforming or identify as LGBTQIA +, is an important goal of the ECHO program. Thus, engaging in efforts to increase their participation is indicated. It also is possible, but is not known, that children in families resistant to inclusive language and hesitant to participate in research that is inclusive also are at risk for suboptimal health. However, families are instructed that they may skip any questions that make them feel uncomfortable.

Creating culturally and socially conscious research materials that are inclusive of a range of identities and experiences may help reduce burden and remediate disparities among diverse populations. We recommend that this framework be used by other large consortia to evaluate their own data collection forms for inclusivity.

Conclusion

Adverse health outcomes disproportionately affect diverse populations. Utilizing culturally and socially conscious research materials that are inclusive of various identities and experiences is necessary to help remediate these disparities. Our review finds compelling evidence of non-inclusivity in many widely used data collection instruments and suggests that similar reviews be undertaken by other large consortia or multi-site studies.