Mohit Chandra

Mohit Chandra

Researcher | Photographer | Football Aficionado | Part-Time Home Chef :)

Lived Experience Not Found

Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions

Authors

Mohit Chandra1, Siddharth Sriraman1, Gaurav Verma1, Harneet Singh Khanuja1, Jose Suarez Campayo2, Zihang Li3, Michael L. Birnbaum4, Munmun De Choudhury1

1College of Computing, Georgia Institute of Technology 2Hospital General Universitario Gregorio Marañón 3Hofstra University 4Columbia University

Paper on arXiv Code Repository

Introduction

Overview of work; we present two tasks in this work– ADR detection and multiclass classification (RQ1), and Expert-LLM response alignment (RQ2).
Figure 1: Overview of work; we present two tasks in this work– ADR detection and multiclass classification (RQ1), and Expert-LLM response alignment (RQ2).

Adverse Drug Reactions (ADRs) from psychiatric medications are one of the leading causes of hospitalizations among mental health patients. With healthcare systems and online communities facing limitations in resolving ADR-related issues, Large Language Models (LLMs) have the potential to fill this gap. However, despite the increasing capabilities of LLMs, past research has not explored their capabilities in detecting ADRs related to psychiatric medications or in providing effective harm reduction strategies. To address this, in this work, we introduce the Psych-ADR benchmark and the Adverse Drug Reaction Response Assessment (ADRA) framework to systematically evaluate LLM performance in detecting ADR expressions and delivering expert-aligned mitigation strategies. The proposed Psych-ADR benchmark includes 239 Reddit posts, labeled across two hierarchical levels for ADR detection and multiclass classification along with expert-written responses to queries. The proposed framework evaluates LLM-generated responses against those of medical experts, focusing on four assessment axes: (a) text readability, (b) emotion and tone expression, (c) alignment of harm-reduction strategies, and (d) actionability of suggested strategies.

Psych-ADR Benchmark

Figure 2: (Left)Class-wise distribution of examples in thePsych-ADR benchmark dataset; % w.r.t. N = 239. (Right) Sample answer representing the structure of answers provided in the Psych-ADR benchmark dataset.
Figure 2: (Left)Class-wise distribution of examples in thePsych-ADR benchmark dataset; % w.r.t. N = 239. (Right) Sample answer representing the structure of answers provided in the Psych-ADR benchmark dataset.

Psych-ADR benchmark is a novel dataset with 239 reddit posts related to psychiatric medications and symptoms with annotations and replies provided by medical experts. The dataset contains two-hierarchical labels depicting -- 1) Presence/Absence of ADR, 2) Category of ADR. A key aspect of the Psych-ADR benchmark is the inclusion of expert-written responses to queries in the ADR labeled posts. We identified and articulated the logical structure where each response in our dataset begins with empathizing with the patient, followed by information on diagnosis, request for additional information, proposing harm reduction strategies to mitigate the ADR, and concluding with a final set of questions.

ADRA Alignment Framework

Evaluation of long-form text generation is an open problem and involves many challenges like isolating the stylistics from the semantics. However, in the context of responses to ADR queries, we propose abstracting out the LLM generations and ground-truth expert responses to four key components– (1) emotion and tone, (2) text readability, (3) harm reduction strategy, and (4) actionability of proposed strategies. Via this abstraction to key components, our alignment evaluations focus on specific aspects that contribute towards an ideal response to ADR queries.


  • Emotion and Tone Alignment: We measured the alignment of LLM-generated and expert written answers across 8 relevant emotional and tonal categories identified from prior literature.

  • Readability Alignment: To evaluate the difficulty level of reading both LLM-generated content and expert-written text, we employed the SMOG Index—a widely recognized measure commonly used to assess health literacy materials.

  • Harm Reduction Strategy Alignment: In cases of adverse reactions to psychiatric medications, suggesting safe medical interventions is crucial to prevent further harm. We operationalized these interventions using harm reduction strategies (HRS), aimed at minimizing the negative effects of medications that one is reliant on. Ideally, LLMs should propose strategies that align with the expert's responses.

  • Actionability of Proposed Strategies: Actionability in the responses of healthcare professionals enables greater engagement and encourages increased action from patients. To this end, we designed an approach to measure the alignment between LLM responses and expert responses along the actionability dimension using four (4) sub-criteria.

Findings

Table 1: Performance of different models on Binary Detection and Multiclass Classification tasks under Zero-Shot and 5-Shot scenarios. We report the accuracy score(Acc.) and weighted F1 score as(F1) with the best and second-best performing model metrics in each scenario highlighted in bold and underline, respectively.
Table 1: Performance of different models on Binary Detection and Multiclass Classification tasks under Zero-Shot and 5-Shot scenarios. We report the accuracy score(Acc.) and weighted F1 score as(F1) with the best and second-best performing model metrics in each scenario highlighted in bold and underline, respectively.

Finding 1: Larger models typically perform better for ADR detection tasks,but this trend does not hold for ADR multiclass classification

As expected, larger models(by parameter size) outperformed their smaller counterparts in the ADR detection task within their respective families,with Claude3 Opus achieving the highest accuracy at 77.41%, followed by GPT-4o and GPT-4Turbo at 72.03%. Interestingly, specialized medical models(OpenBioLLM-Llama3-70B and Llama3Med42v2-70B)struggled in this task. However, for ADR multiclass classification,we did not observe any clear pattern between model size and performance. GPT-4 Turbo was the best performing model with an accuracy of 57.58%, followed by Llama3-Med42v2-70B at 56.40%. All models struggled with multiclass classification,likely due to the complexity of distinguishing between ADR types.

Finding 2: Models exhibited a “risk-averse” tendency, and prone to commit false-positive errors

In both ADR detection and multiclass classification tasks, all models displayed “risk-averse" behavior, often mislabeling posts without ADRs as positive for ADRs. In zero-shot settings,Claude Opus had a false-positive rate of 42% for ‘ADR-No’ labels, while Claude3 Haiku’s false positive rate was as high as 97%(see paper). Similarly, in ADR multiclass classification, models struggled to distinguish between non-dose-related, dose-related, and time-related ADRs. GPT-4Turbo misclassified 51% of non-dose-related and 50% of time-related ADRs in zero-shot settings.

Finding 3: In-context learning enhances model performance but not in every case

We observed that in-context learning in general improved performance of models for both ADR detection and multiclass classification tasks, with a more significant impact on the latter task. For multiclass classification, we observed an average increase of 18.14 and 23.06 points in weighted F1 score among model performance using least-similar and most-similar example prompting respectively. However, this pattern was not observed in the ADR detection task. Claude 3 Opus outperformed other models in the ADR detection, achieving an F1 score of 76.44 with zero-shot prompting. In ADR multiclass classification, Llama-3.5-405B performed best with most-similar examples (F1 76.69).

Finding 4: LLMs generate harder to read replies as compared to experts

Figure 3: Mean SMOG Scores and 95% Confidence Intervals for Various Models (lower values are better).
Figure 3: Mean SMOG Scores and 95% Confidence Intervals for Various Models (lower values are better).

LLM-generated responses tend to be more complex, reflected in higher SMOG scores compared to those written by experts (SMOG mean11.02) in the Psych-ADR benchmark. Welch’s t-test (Welch, 1947) further revealed that the SMOG scores of expert-written responses were significantly lower than those of any LLMgenerated responses. We also observed that more capable models produced more readable responses (with Claude 3 Opus being an exception). Similar to the findings on emotional alignment, Llama3Med42v2-70B showed the lowest alignment with the expert-written responses, producing the most complex responses, likely due to a major portion of instruction-tuning data coming from medical and biomedical scientific literature.

Finding 5: LLMs struggle to provide expert-aligned harm reduction strategies with the best model providing ~71% alignment with experts

Table 2: Alignment of harm reduction strategies of various models with the expert’s response.We report the mean and standard deviation for the AlignScore metric GPT-4o score, with the best (bold) and second-best (underline) performing model in each metric highlighted.
Table 2: Alignment of harm reduction strategies of various models with the expert’s response.We report the mean and standard deviation for the AlignScore metric GPT-4o score, with the best (bold) and second-best (underline) performing model in each metric highlighted.

More capable models Llama 3.1-405B Instruct and Claude 3.5 Sonnet in their respective families tended to produce strategies less aligned with the expert than their smaller counterparts, validating a previously observed pattern of LLM performance in responding to open-ended clinical questions (Kanithi et al., 2024). While the open-weights models performed on par or better than proprietary models across both alignment metrics, the best performing medical model (OpenBioLLM-Llama3-70B) aligned with expert harm reduction strategies for 70.86% of the cases, highlighting the need for further fine-tuning for specialized domains such as psychiatry.

Finding 6: While LLMs provide less practical and relevant advice, their advice is more clear and specific.

Table 3: Mean actionability alignment scores of harm reduction strategies (last column),computed as average of practicality, relevance, specificity, and clarity scores.
Table 3: Mean actionability alignment scores of harm reduction strategies (last column),computed as average of practicality, relevance, specificity, and clarity scores.

We noted that expert responses scored the highest on overall actionability in comparison to all the LLMs(0.46). Nonetheless, medical models like OpenBioLLMLlama3-70B and Llama3-Med42v2-70B demonstrated reasonable actionability scores(0.44), followed by other proprietary and open-weights models (0.35 to 0.43). While expert responses were rated considerably better than all LLM responses in terms of the practicality (0.83) and contextual relevance (0.73) of the harm reduction strategies,their specificity (0.17) and clarity (0.13) are relatively lacking. This indicates that while LLMs tend to demonstrate greater specificity and clarity in their harm reduction strategy, the recommended strategies may often not be feasible and contextually relevant,considering the users personal circumstances,such as physical ability,financial resources, and time constraints.

Broader Implications of Our Work

Need for going beyond the choice-based medical benchmarks

LLMs have achieved near-perfect scores on popular medical benchmarks (Nori et al., 2023a; Singhal et al., 2023b), however, these evaluations typically focus on multiple-choice or case based questions,which don’t reflect the nuanced understanding required in real-world scenarios like mental health. Despite their strong performance on medical tasks, Llama3-Med42v2-70B and OpenBioLLM-Llama3-70B struggled with detecting ADRs and providing aligned and actionable HRS, highlighting the need to move beyond standard benchmarks towards more holistic alignment evaluation paradigms.

Focusing on empowering experts rather than replacing them

While LLMs did not match expert performance in our analysis, they showed a potential to enhance healthcare by providing clearer, more actionable responses. Given the global shortage of mental health professionals (Kazdin and Rabbitt, 2013), LLMs could expand access to mental healthcare and support experts with further finetuning and alignment with expert reasoning.

Disentangling inclusion of humanistic features in LLMs and advocacy for inclusion of lived experience

While our work provides evidence of the lack of lived experience, which is essential for understanding the nuances of a complex task such as ADR detection and for proposing mitigation strategies in model responses, we do not advocate for increasing human-like features in LLMs. Previous studies have suggested that heightened anthropomorphism, independent of whether it is accompanied by enhanced capabilities, can increase trust among individuals (Natarajan et al., 2020; Chen et al., 2021). In contrast, we advocate for approaches that align with previous research, which has shown that the efficacy of LLMs in the healthcare domain can be enhanced through fine-tuning on specialized data or by incorporating useful features into the model, without introducing human-like features.

Citation

@article{chandra2024lived,
    title     = {Lived Experience Not Found: LLMs Struggle to Align with Experts on 
                Addressing Adverse Drug Reactions from Psychiatric Medication Use},
    author    = {Chandra, Mohit and Sriraman, Siddharth and Verma, Gaurav and 
                Khanuja, Harneet Singh and Campayo, Jose Suarez and Li, Zihang and 
                Birnbaum, Michael L and De Choudhury, Munmun},
    journal   = {arXiv preprint arXiv:2410.19155},
    year      = {2024}
}