Rules of engagement: how sensitive concerns are hidden from drug trials

by

Part 2 of a series. Part 1 is Polls vs. official data on side effect rates

The year is 1995. A 28-year-old man, whom we’ll call Alan, is participating in a large clinical trial of Propecia. Alan has been losing his hair in the last couple of years, so he brings high hopes that the new drug will stop hair loss, and maybe even lead to regrowth. Alan is in the group taking finasteride, not placebo (although he does not know this). Three months after starting the drug, Alan shows up for a study appointment.

The setting resembles a medical office. A study investigator, trained as a nurse, examines Alan’s scalp and hair, recording measurements on a form. Among other questions, she asks Alan if he experienced any side effects. None, he replies. He is then given a form which might have looked like this, in part:

SEXUAL FUNCTION QUESTIONNAIRE

Over the last month, when you had erections with sexual stimulation, how often were your erections hard enough for penetration?

Over the last month, during sexual intercourse, how difficult was it to maintain your erection to completion of intercourse?

Over the last month, how much have you enjoyed sexual intercourse?

Over the last month, when you had sexual stimulation or intercourse, how often did you ejaculate?

Over the last month, when you had sexual stimulation or intercourse, how often did you have the feeling of orgasm (with or without ejaculation)?

Over the last month, how do you rate your confidence that you can get and keep your erection?

Selected questions from International Index of Erectile Function, 15 questions (IIEF-15).

Alan has a shy and polite demeanor, and does not want to talk about his sex life in the study. If he recently experienced sexual dysfunction, he might prefer not to think about it, or suppose it was a psychological issue. Since he wants to stay on the drug and see if it works, he sees no benefit in mentioning side effects. He also does not want to disappoint the investigator or create an awkward situation.1

Alan is thus far from an open book in the trial. He is negotiating between his internal experience and the rules and expectations of the trial. (This recalls Pierre Bourdieu’s concept of the habitus, described as “norms and expectations…predisposing us to act…within the constraints of particular social fields.”2) As we move through different settings in daily life—a restaurant, a city bus, a sporting event, a clinical study appointment—the rules of engagement change. In each setting, conventions suggest when to sit or stand, cheer or be quiet, bring up personal concerns or not. Breaking these conventions could create friction and embarrassment.

Still other layers of context may be in play. The investigator’s demeanor, time constraints or broader cultural norms could encourage or discourage Alan from bringing up safety concerns. All these factors help explain how a safety concern could be experienced, but never expressed in any form.

Figure 1 shows a two-stage model of how trials can exclude and discount safety signals. First, the signal is prevented from emerging by personal, social and cultural factors. Second, the sponsor has opportunities to discount signals that are expressed by manipulating the data.

Figure 1. The concealment and discounting of sensitive adverse effects. (Open in a new tab) (A) represents a “true” rate of adverse effects. 44% is a provisional number drawn from an analysis of informal polls. (B) Contextual factors discourage signals from being expressed at all. For example, explicit language about sexual activity might seem inappropriate in the clinical setting. Cultural taboos around sexuality remain in place within the examination room (and might be heightened). (C) Only a subset of adverse events are disclosed. (D) The study design and decisions by the sponsor further discount the safety signal (for example, investigators determined whether an adverse effect was drug-related). (E) Safety data that is ultimately reported might drastically understate the actual risks of the drug.

Figure 2 shows how these two mechanisms operate at different points in the Propecia trials.

Figure 2. Losing the signal, step by step. (Open in a new tab) (A) An ideal, veridical signal of a drug’s risks. (B) At enrollment, prospective participants are turned away if they do not meet certain criteria. This could conceal safety concerns that would arise in actual use. (C) AEs may not enter the participant’s awareness (for example, if he does not engage in sexual activity), or he may decide to ignore them. (D) A participant may not attribute an AE to the drug. (E) Only certain types of AEs are measured systematically (for example, depression and anxiety were not measured systematically, but these are common in actual use). (F) The participant might choose not to disclose an AE (see Figure 1). (G) If an AE is disclosed, the participant might discount its severity to avoid displeasing the investigator, appearing less masculine, taking too much time, or causing an awkward situation. (H) In data preparation, the sponsor may apply a cutoff to a scaled score to determine whether it qualifies as an AE. (I) The investigator determines whether the AE is drug-related. (J) The denominator for AE rates is the number of participants who started the trial. Early dropouts from the treatment group tend to lower the overall rate of AEs. (K) Merck researchers carried out post hoc exclusions and transformations of sexual function questionnaire data.3 (L) When AEs are reduced in both groups, there may not be sufficient statistical power to distinguish AE rates in treatment and placebo groups (for example, within four subdomains of sexual function in Propecia Phase III trials). (M) The design of charts and tables, opaque language, and generalizations can conceal safety concerns. Abbreviation: AE, adverse effect.

Phase III Propecia trials were blinded, meaning that neither participants nor investigators knew whether a participant was taking finasteride or a placebo. This design is intended to eliminate bias, but if the trial is insensitive to adverse events, there may be a floor effect. That is, a greater proportion of safety events in the treatment group could fall below the threshold of sensitivity, reducing the difference in rates between groups. (For example, in a Propecia Phase III trial there was no statistical difference between groups in four subdomains of sexual function. In a more general statistic, 3.8% of the finasteride group had any sexual adverse event vs. 2.1% of the placebo group. This difference was statistically significant, but might have been greater if safety detection had been more sensitive.) Figure 3 illustrates the floor effect.

Figure 3. A floor effect collapses differences between groups. Trapezoidal shapes represent all sexual adverse events in each group. (A) Safety events that were disclosed by the participant, attributed to the drug by the sponsor, and reported. Merck reported that 3.8% of the finasteride group and 2.1% of the placebo group reported a sexual adverse event—a statistically significant difference. (B) Events that were disclosed by the participant but not reported because of study design factors and data manipulation (see Figure 2). (C) Safety concerns that were never expressed by the participant due to social and contextual factors (see Figure 1). Downward-pointing arrows indicate forces that reduce sensitivity to adverse events. If sensitivity to safety events is low, a greater proportion of adverse events in the treatment group may fall below the sensitivity threshold (a floor effect). This would conceal drug-related safety concerns and collapse differences between the groups.

A previous post reported a nearly sixfold difference (5.7x) between AE rates in informal polls (43.9%) and clinical trials (7.7%). In this post, we have reviewed two types of mechanisms that could explain this vast difference: 1) social, contextual and cultural factors that prevent the disclosure of sensitive concerns; and 2) study design, interpretations, and data manipulation by the sponsor. These factors are rarely mentioned when trial data is cited today, as if the trials are beyond criticism. There are two ironies in this. First, the very formality that makes trials seem authoritative, might reduce their sensitivity to safety events. Second, the scale and complexity of the trials provided many opportunities for Merck to exclude or discount safety concerns. It is not clear that FDA was aware of these mechanisms of exclusion when it reviewed and approved Propecia in 1997. We can only wonder whether, if the safety signal had been properly reported, the drug would have been approved in the first place.

Notes

1. In a focus group study, Akre et al. found that men ages 16–20 did not talk to anyone about sexual dysfunction (SD). They felt shame and uneasiness over this topic, since disclosing such problems could lead others to question their masculinity. Other barriers were concern about lack of confidentiality, and the view that “the role of a [general practitioner] is to take care of somatic problems but not sexual problems.” Source: Akre C, Michaud PA, Suris JC. “I’ll look it up on the Web first”: Barriers and overcoming barriers to consult for sexual dysfunction among young men. Swiss Med Wkly. 2010. doi:10.4414/smw.2010.12968.

2. Source: habitus. Oxford Reference. Accessed December 1, 2022. https://www.oxfordreference.com/view/10.1093/oi/authority.20110803095914456

3. See: United States Food and Drug Administration. Medical Review in Drug Approval Package: Propecia (Application No.: 020788). Approval Date: December 19, 1997.