Skip to main navigation Skip to search Skip to main content

Suitability of Bag-of-Words and Transformer Models as Health Literacy Assessment Tools

Research output: Contribution to conference › Poster › peer-review

Abstract

Background. Public health leaders have identified the need to improve institutional communications to increase health information uptake in the wake of the COVID-19 pandemic.1-2 Readability assessments offer numerical benchmarks that can be used to assess written materials’ contributions to this aim. However, the hurdles that health communication materials present can involve more than just long sentences and words. Health literacy assessments like the CDC Clear Communication Index3 (CDC Index) offer similar insight into broader domains like actionability. However, as the CDC Index exemplifies, these assessments are human resource intensive, resulting in study designs resembling qualitative assessments despite an orientation toward quantitative results. The CDC Index comprises 20 binary ratings based on health communication guidelines. The end result is a percentage score that streamlines interpretation. At the same time, quantification sacrifices the nuanced examination that goes into close reading of the materials. For example, a material that executed every communication guideline exceptionally well could receive the same score as one that required more deliberation to reach final decisions. This loss of nuance is expected in a quantitative assessment of materials, but the CDC Index rating process requires training, close reading, and deliberation among raters that impedes study designs that can provide generalizable knowledge about a corpus of texts. While there are benefits to more qualitative approaches to assessments of materials, there is still a gap in the field around scalable health literacy assessments beyond readability. The aim of this study is to examine the suitability of applying supervised machine learning to scale up the CDC Index to address this methodological gap.
Data. We filtered a larger Tweet dataset4 to focus on 31,664 Tweets from state public health agencies, in English, containing a COVID-19 pandemic keyword, containing no URL errors, largely comprising unique text, posted between January 1 and December 10, 2020. We created “timeline” versions of Quote Tweets by appending the full text of the referenced Tweet after the commentary provided by a state health agency account, if applicable. Similarly, we analyzed the full text of Retweets and original Tweets by state health agencies. Among these, we focused on Tweets >87 characters long, filtering out the shortest 3% of Tweets in the dataset. We drew a state- and time-stratified sample of 1,900 Tweets for model training, 200 Tweets for model validation, and 400 for training human raters.

Methods. We used LASSO regression to create candidate models to predict health communication specialists’ ratings of Tweets via 8 CDC Index guidelines. These guidelines focused on: presence of a main message; presence of a call to action; use of active voice;
use of familiar words; presence of a behavioral recommendation, and an explanation of the importance of a behavioral recommendation (if applicable); presence of a discussion of risk, and an explanation of the nature of risk (if applicable). During an 11-week period in summer 2023, two raters each single-rated 950 Tweets to create a model training data set. After excluding Tweets with data labeling errors, we had a training data set of 1,895 Tweets. The raters also double-rated 300 Tweets to create the model validation data set. After excluding Tweets with data labeling errors, we had a validation data set of 298 Tweets. We used consensus scores where both raters aligned. We created forced consensus scores via a 3rd rater settling any conflicts. We used a bag-of-words approach and 5-fold cross-validation to create a LASSO regression model to predict raters’ binary ratings of Tweets meeting or failing to meet one of eight CDC Index guidelines. We assessed model performance in the validation data set by comparing predictions to the gold-standard consensus scores. This allowed us to use inter-rater agreement metrics, recognizable within public health, to describe the alignment between model predictions and human ratings.

Findings. The data labeling process revealed several changes necessary to apply the CDC Index to short social media posts at scale. For example, one guideline asks a rater to determine whether a material’s call to action uses active voice. This requires alignment on determining that the Tweet has a call to action, specifying the location of the call to action, and assessing the main point of the guideline: use of active voice. We streamlined this guideline to focus on the message as a whole rather than just the call to action. We argue this was a reasonable modification considering a call to action is likely to comprise a significant portion of a Tweet. The data labeling process and model validation results highlighted some guidelines as particularly promising for a supervised machine learning approach, namely: inclusion of a main message, inclusion of a call to action, and inclusion of a behavioral recommendation. In this exploratory study, these guidelines displayed acceptable human-human and human-computer inter-rater agreement, while also relying on specialized health communication knowledge. This contrasts with guidelines around the use of active voice and the use of familiar words. Both guidelines displayed low human-human and human-computer inter-rater agreement, while not relying on specialized health communication knowledge. Tools to identify active voice have been developed outside of public health. And a dictionary-based approach may be more appropriate to assess the use of common words for the broad audiences a state health agency communicates with. The results further illustrate the risk of stigma from automated health literacy assessments of materials, as some ratings were associated with specific place names. While there are ways to exclude such potentially stigmatizing associations from models, this issue is not straightforward due to the use of place names in compound phrases that carry semantic meaning, e.g. #MaskUpMA. Finally, a LASSO regression approach on bags-of-words has several qualities that align with common practices in public health. Training the models is feasible within the commonly used R language via existing packages. Interpreting regression model outputs is a key professional skill gained through public health training. The presence of individual tokens as model coefficients results in the kind of usable knowledge expected within health literacy research. For example, in our model predicting human raters’ determination of the presence of a call to action, the most predictive tokens included commands like “read” and “wear”, as well as the word “please.” This can contribute to the development of best practices for meeting CDC Index guidelines, e.g. using “please” to soften the tone of a behavioral recommendation while still including a clear command verb. Overall, this study points to the utility of such an approach to scaling up health literacy assessments as a real-time public health monitoring tool, though not necessarily for every guideline in the CDC Index or for every health topic.
Original languageEnglish
Publication statusPublished - 19 Jul 2024
Externally publishedYes
EventInternational Conference on Computational Social Science - University of Pennsylvania, Philadelphia, United States
Duration: 17 Jul 2024 → 20 Jul 2024
Conference number: 10
https://ic2s2-2024.org/

Conference

ConferenceInternational Conference on Computational Social Science
Abbreviated titleIC2S2
Country/TerritoryUnited States
CityPhiladelphia
Period17/07/24 → 20/07/24
Internet address

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

  1. SDG 3 - Good Health and Well-being
    SDG 3 Good Health and Well-being

Fingerprint

Dive into the research topics of 'Suitability of Bag-of-Words and Transformer Models as Health Literacy Assessment Tools'. Together they form a unique fingerprint.

Cite this