Examination of Participant Flow in the CONSORT Diagram Can Improve the Understanding of the Generalizability of Study Results

Introduction

The results of a study are best generalized only to the population from which the sample of that study was drawn. So, a study conducted in young women will only provide guidance about outcomes in young women, and not outcomes in older women or in men of any age. Likewise, a randomized controlled trial (RCT) of a new antidepressant (AD) drug will not establish the efficacy of the drug in patients who are suicidal or those who have psychotic depression, nor its safety in patients with cerebrovascular or cardiovascular disease, because such patients would have been specifically excluded from the study sample (as an aside, to be strictly accurate, efficacy in psychotic or suicidal depression, and safety in patients with medical comorbidity, would not have been established even had the sample included such patients; this is because the study would not have been adequately powered for such subgroup analyses). The sample selection criteria are listed in the methodology section of a study, and conclusions about generalization of study results can be drawn from a scrutiny of these criteria.

Not all patients who meet study selection criteria are actually recruited, and, of those recruited, not all reach the end of the study. So, the ability to generalize from the study findings is further restricted. This article discusses the importance of the CONSORT diagram, or the study flowchart, as another means of determining the extent to which the findings of a study can be generalized. The subject is important in evidence-based mental health care because physicians must be able to gauge how applicable study findings are to the patients whom they see.

The CONSORT Diagram

The CONSORT diagram presents the flow of subjects through the trial, from screening to the study endpoint. The diagram provides information about the following elements:

1. How many patients (who seemed appropriate for the study) were screened.
2. How many patients (from among those screened) actually met the selection criteria and consented to participate in the study and how many patients were excluded for reasons such as failure to meet the selection criteria or refusal to participate.
3. How many patients were actually randomized to each intervention.
4. How many patients dropped out in each group, with reasons for dropout (eg, treatment inefficacy, experience of adverse events, withdrawal of consent, scheduling difficulties).
5. How many patients in each group formed the sample the data from which could be analyzed.

Generalization of study results could be challenging if there is considerable attenuation in patient numbers described from the start of the CONSORT diagram to its end. Why this is so will become clear from the examples provided in the next 2 sections.

Cognitive Therapy vs Antidepressant Drugs as Maintenance Treatment for Unipolar Depression

Kuyken et al1 described a large (N = 424), 2-year RCT that compared mindfulness-based cognitive therapy (MBCT) with maintenance AD treatment in the prevention of relapse or recurrence of depression in adults with recurrent major depressive disorder (MDD). Patients were required to have had at least 3 previous episodes of depression; almost half the sample, in fact, had had at least 6 previous episodes. The authors reported a 2-year relapse rate of 44% vs 47% in the MBCT versus AD groups, respectively; time to relapse did not differ significantly between the 2 groups. Secondary outcomes, such as depression-free days, residual depression at study endpoint, health care costs, and societal costs, also did not differ between groups.

A casual reader may be tempted to conclude that MBCT could be a good alternative to maintenance AD medication even in patients with a loaded past history of illness. Such a conclusion could, however, be premature. One reason is that 24% of the AD group did not remain on a therapeutic dose of AD medication, and about 30% of the MBCT group did not stop their AD medication; this could have provided the MBCT group with a relative advantage. Another reason relates to the recruitment of the sample, as evident from the study flowchart.

The flowchart, as supplemented by information from the text, was revealing. A computerized search had identified > 28,500 eligible patients from a general practitioner (GP) database; however, GPs excluded nearly 9,000 patients because they were apparently unsuitable. This straightaway introduced potential bias, because the excluded patients may have been more severely ill, or may have had other characteristics that the GPs thought would make them unsuited for MBCT. Note that patients would not have been excluded because of unsuitability for maintenance AD treatment because that was the default treatment.

Letters of invitation were sent to about 19,600 patients; most either did not respond or declined to participate further. Additionally, 89 patients actually self-referred themselves; their participation presumably indicated specific interest in MBCT or dissatisfaction with their ADs because, as already indicated, AD treatment was the default.

Out of 2,188 patients assessed for eligibility, 1,120 were considered ineligible, and 644 declined to participate. Thus, only 424 patients remained for randomization. This final sample is likely to have been favorably inclined toward MBCT and/or unfavorably inclined toward ADs because it is hard to imagine that a patient with recurrent MDD would willingly quit a well-tolerated and effective AD regimen. Therefore, whereas treatment with MBCT yielded outcomes similar to those with maintenance ADs, a 2-year follow-up program of MBCT may actually be appropriate for only a very small percentage of patients with recurrent MDD; the rest may not be considered suitable, or may just not be interested in MBCT.

A perceptive reader might inquire whether the unsuitability or disinterest of patients could be related to participation in the RCT rather than to participation in MBCT. This is possible, but the study was designed to be pragmatic. So, it does seem that most recurrent MDD patients receiving maintenance ADs do not want to consider MBCT or are considered by their GPs to be unsuitable for the intervention. In other words, we learn from the study flowchart (and from the explanations in the text) that the findings of the study can only be generalized to a small proportion of patients with recurrent MDD, and that these patients must favor MBCT and/or disfavor ADs, or, at least, they must be in equipoise with regard to MBCT and maintenance AD treatment. Whereas generalizability to patients in equipoise was warranted, in the opinion of the authors,1 surprisingly, an accompanying commentary sought to generalize the findings about MBCT to all patients with recurrent depression.2

Quetiapine vs Placebo as Maintenance Treatment for Bipolar Depression

How does the CONSORT diagram help in psychopharmacology RCTs in which patients are sampled in a more conventional manner than that in the study described by Kuyken et al1? Consider another large (N = 628), 2-year RCT, in which quetiapine and placebo were compared as augmentation agents during maintenance therapy in bipolar patients who were receiving lithium or divalproex.3 Quetiapine comprehensively outperformed placebo; mood events during the study period were experienced by 20% vs 52% of patients in the quetiapine versus placebo groups, respectively; and hazard ratios for time to recurrence of mania, depression, or any mood event were in the 0.30 to 0.33 range, favoring quetiapine over placebo.

How applicable are these findings to the average bipolar patient? The CONSORT diagram shows that 1,953 patients were enrolled into a prerandomization stabilization phase. However, less than a third of these patients could be randomized; of the 1,325 patients who prematurely discontinued the study; 404 dropped out due to the experience of adverse events, 329 were lost to follow-up, 269 withdrew consent, and the rest did not reach randomization for other reasons, including lack of treatment response and not meeting eligibility criteria. Thus, it appears that when quetiapine is used as an augmentation agent during maintenance treatment in bipolar disorder, most patients quit in the initial 12 weeks. So, the findings of the study can only be generalized to the patients who survive the initial 12 weeks of augmentation, which is not the same as concluding that quetiapine is an appropriate augmentation treatment for every bipolar patient receiving maintenance treatment with lithium or divalproex.

Unfortunately, the CONSORT diagram in this RCT3 did not indicate how many patients were screened; almost certainly a sizable number of patients would have been found ineligible, further limiting the generalizability of the findings in bipolar disorder.

Conclusions

The findings of a study can only be generalized to the population from which the sample was drawn. The characteristics of this population can be understood from a scrutiny of the study selection criteria, described in the study methods. Often, however, sample attenuation occurs between subject screening and final recruitment; this information can be obtained from the study flowchart or CONSORT diagram. Such sample attenuation can further limit the generalizability of the study outcomes. Readers should therefore look at both study selection criteria and the CONSORT diagram to understand the extent to which the study results apply to the patients whom they see.

Parting Notes

It is also important to examine the CONSORT diagram for what happens to patients after randomization. For example, dropout, and the reasons for dropout, can provide direct and indirect information about the safety, acceptability, and efficacy of the treatments being studied.

Each month in his online column, Dr Andrade considers theoretical and practical ideas in clinical psychopharmacology with a view to update the knowledge and skills of medical practitioners who treat patients with psychiatric conditions.

Department of Clinical Psychopharmacology and Neurotoxicology, National Institute of Mental Health and Neurosciences, Bangalore, India ([email protected]).
Financial disclosure and more about Dr Andrade.

REFERENCES

1. Kuyken W, Hayes R, Barrett B, et al. Effectiveness and cost-effectiveness of mindfulness-based cognitive therapy compared with maintenance antidepressant treatment in the prevention of depressive relapse or recurrence (PREVENT): a randomised controlled trial. Lancet. 2015;386(9988):63-73. PubMed doi:10.1016/S0140-6736(14)62222-4

2. Mulder R. Depression relapse: importance of a long-term perspective. Lancet. 2015;386(9988):10-12.PubMed doi:10.1016/S0140-6736(14)62448-X

3. Suppes T, Vieta E, Liu S, et al; Trial 127 Investigators. Maintenance treatment for patients with bipolar I disorder: results from a North American study of quetiapine in combination with lithium or divalproex (trial 127). Am J Psychiatry. 2009;166(4):476-488. PubMed doi:10.1176/appi.ajp.2008.08020189