Psychiatric Rehospitalization: Development of a Person-Level Indicator for Care Planning and Quality Assurance

This work may not be copied, distributed, displayed, published, reproduced, transmitted, modified, posted, sold, licensed, or used for commercial purposes.
By downloading this file, you are agreeing to the publisher’s Terms & Conditions.

Psychiatric Rehospitalization:

Development of a Person-Level Indicator
for Care Planning and Quality Assurance

Christopher M. Perlman, PhD; John P. Hirdes, PhD; and Simone Vigod, MD, FRCPC

ABSTRACT

Objective: Rehospitalization affects quality of life and health system efficiency. Although this outcome is a common quality indicator, there are few applications for linking evaluation to risk mitigation at the person level. This study examined risk factors for rehospitalization to develop an application for supporting care planning based on the interRAI Mental Health (MH), a commonly available assessment system.

Method: A retrospective analysis was performed of 53,538 psychiatric inpatients assessed with the interRAI MH in Ontario, Canada, between January 2010 and May 2014. The interRAI MH is a clinical system for assessing demographic variables, service utilization, functional status, and clinical needs. Logistic regression models and survival analysis were used to develop the Rehospitalization Clinical Assessment Protocol by predicting 90-day rehospitalization to any inpatient mental health bed.

Results: Variables found to significantly predict rehospitalization included 6 or more lifetime hospitalizations (odds ratio [OR] = 1.40), positive symptoms of psychosis (OR = 1.23), a secondary substance use disorder (OR = 1.13), and being at risk of harm to self (OR = 1.11). Using these variables, the Rehospitalization Clinical Assessment Protocol was derived whereby those at level 2 (highest) were 74% more likely to be rehospitalized within 90 days than those at level 0. By 1-year postdischarge, 30% at level 2 and 18% at level 0 were rehospitalized.

Conclusions: The Rehospitalization Clinical Assessment Protocol is an application supporting care planning for targeting risk of rehospitalization whenever a person is assessed with the interRAI MH. Further exploration is needed to understand how the use of this Clinical Assessment Protocol, service processes, and health system structures further mediate or moderate psychiatric rehospitalization risk.

Prim Care Companion CNS Disord
2015;17(4):doi:10.4088/PCC.15m01784

© Copyright 2015 Physicians Postgraduate Press, Inc.

Submitted: January 14, 2015; accepted March 24, 2015.

Published online: July 23, 2015.

Corresponding author: Christopher M. Perlman, PhD, School of Public Health and Health Systems, University of Waterloo, 200 University Ave W, Waterloo, ON, N1H 7W5 ([email protected]).

Rehospitalization is a commonly assessed indicator of health system performance1 as it has a significant impact on persons with mental health conditions2,3 and on health system utilization.4 However, the validity of rehospitalization as an indicator of performance has been questioned given the challenge in addressing risk at the person level.5 Prior hospitalizations are consistently the strongest predictors of future mental health hospitalizations.4,5 This pattern may reflect a range of issues, including the person’s illness severity, socioeconomic and health system supports for illness management, and maintenance of community residency.6–15 For instance, stability and social support in living situations,16,17 as well as opportunities for meaningful social and leisure engagement,9 are associated with decreased risk of psychiatric rehospitalization. While recognition of prior hospitalizations may inform overall risk, the identification of specific risk factors that are also amenable to intervention may have greater impact for prevention.

The utility of decision support systems for addressing rehospitalization risk relies on the availability of relevant clinical information. Rehospitalization risk indices have been implemented to varying degrees of accuracy in a number of health sectors.18–20 In mental health care, such indices have been uncommon, largely due to the lack of commonly available clinical assessment systems.4 Clinical data may be advantageous to administrative data for capturing a range of amenable factors for use in care planning.11

In several international jurisdictions, the interRAI Mental Health (interRAI MH) assessment has been implemented. In Canada, the Canadian Institute for Health Information (CIHI) oversees the collection and management of the interRAI MH across several provinces.21 The instrument assesses a wide range of demographic, clinical, and service utilization variables and is designed to support care planning, outcomes monitoring, case-mix identification, and quality measurement.22–24 The care planning applications of the interRAI MH include Clinical Assessment Protocols, specific decision support applications for informing care planning with patients.25 The Clinical Assessment Protocols combine item responses to identify potential problems or needs, such as education and employment,26 trauma,27 and risk of harm to others.28

The purpose of this study is to develop a psychiatric Rehospitalization Clinical Assessment Protocol for targeting care planning related to risk of rehospitalization. First, this study examines predictors of rehospitalization using data from a large sample of mental health inpatients in Ontario, Canada. Second, predictors are examined in combination to develop an algorithm to identify levels of risk, the Rehospitalization Clinical Assessment Protocol levels. Third, time-to-event analyses are used to examine the predictive validity of the Rehospitalization Clinical Assessment Protocol levels. Given the broad use of rehospitalization as a performance indicator and the international implementation of the interRAI MH, the development of a psychiatric Rehospitalization Clinical Assessment Protocol may be useful for linking care planning and quality improvement.

METHOD

Sample

The retrospective sample of 53,538 patients was drawn from interRAI MH data available from the CIHI Ontario Mental Health Reporting System (http://www.cihi.ca/CIHI-ext-portal/internet/EN/Home/home/cihi000001). This system includes interRAI MH data collected from every person admitted to an inpatient mental health bed across all hospitals in Ontario, Canada. The data for this study included all assessments between January 1, 2010, and May 25, 2014. The index hospitalization was based on the person’s first hospitalization in the database, as indicated by the first paired admission and discharge assessment. Index hospitalizations were excluded for discharges after February 25, 2014, to allow for exposure to 90-day rehospitalization. Data were not included for patients with lengths of stay less than 72 hours as they were not assessed with the complete interRAI MH assessment. The Office of Research at the University of Waterloo, Waterloo, Ontario, approved this study protocol and, in conjunction with the authors, was responsible for overseeing the ethical aspects of the project.

This study utilized an existing dataset that includes anonymized data collected as part of regular clinical practice in all inpatient psychiatry hospitals. Part of this practice includes routine submission of these data to the CIHI for reporting. The CIHI provides anonymized data to researchers. As such, no informed consent process was required between the researchers and those who were assessed.

Assessment Instrument

The interRAI MH includes over 400 items measuring health service utilization, clinical and functional status, provisional psychiatric diagnostic categories, harm to self and others, control interventions, substance use, social relationships, and vocational factors.22 Clinical staff overseeing the care of the person complete the interRAI MH based on interview, observation, and discussion with other informants including members of the care team and family. In Ontario, the interRAI MH is completed at admission and after 90 days in hospital and/or at discharge. The interRAI MH is licensed by interRAI free of charge to the Ontario Ministry of Health and Long-Term Care for use among all hospitals providing inpatient mental health services. Further information about copyright and licensing can be found at www.interrai.org.

In interrater reliability studies, the average agreement was 83% for all interRAI MH items, and the average weighted κ for all interRAI MH items was 0.70.29,30 Less than 15% of items on the interRAI MH had κ values below 0.60, and only 3 items had κ values below 0.4.30 Average κ values for selected variables include 0.78 for mental health service history, 0.66 for self-injury, 0.86 for physically abusive behavior, and 0.67 for instrumental activities of daily living.29 All variables examined in this study were drawn from the interRAI MH.

Independent Variables

Demographic variables that were included were gender, age, marital status, education, employment, living alone or with others, being homeless, having a partner/spouse, and having a confidant. Admission status variables that were included were patient type (coded as: acute, long-stay, geriatric), involuntary admission, reason for admission, and insight into mental health (assessed as none, limited, or full). Provisional diagnosis was based on Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) categories that were assigned by the attending psychiatrist. Each diagnostic category was coded as the first, second, or third most important diagnosis. The presence of a secondary mental health diagnosis was assigned for the presence of any 2 assigned diagnostic categories. A secondary substance use diagnosis was defined as the presence of a substance use disorder assigned as secondary in combination with any other DSM-IV diagnosis.

Prior mental health service utilization was assessed using 2 items: the number of hospitalizations from the prior 2 years (coded as 0 = none, 1 = 1 or 2, or 2 = 3 or more) and the number of lifetime hospitalizations (0 = none, 1 = 1 to 3, 2 = 4 to 5, and 3 = 6 or more). In Ontario, the interRAI MH coding of prior hospitalizations has been found to be 95% accurate compared to administrative discharge abstract data.31

Several summary scales are embedded in the interRAI MH that assess clinical characteristics. The Depression Severity Index summarizes 5 items into a scale ranging from 0 to 15, with higher scores indicating a higher number and frequency of depressive symptoms observed in the prior 3 days. The Depression Severity Index has demonstrated good internal consistency with a Cronbach α = 0.77, and scores greater than 3 are significantly related to the presence of a mood disorder.24 The Positive Symptom Scale summarizes the frequency of items assessing hallucinations, command hallucinations, delusions, and abnormal thoughts observed in the 3 days prior to assessment into a scale ranging from 0 to 12. The Cronbach α for the Positive Symptom Scale for this study sample was 0.77, demonstrating good internal consistency. The Cognitive Performance Scale incorporates items such as short-term memory and cognitive skills for daily decision-making into a scale ranging from 0 (cognitively intact) to 6 (very severe impairment); the Cognitive Performance Scale has been found to correlate strongly with other common measures of cognition in a mental health sample.32 The Activities of Daily Living Hierarchy Scale combines items measuring hygiene, mobility, toilet use, and eating into a scale ranging from 0 to 6, with higher scores indicating greater dependence in functioning.33

The Risk of Harm to Others, Self-Care Index, and Severity of Self-Harm scales were developed as indicators of risk to patient safety.25 These scales are predictive algorithms based on decision trees that use interRAI MH items to predict staff ratings of risk in these areas as well as future patient safety events. For example, the Risk of Harm to Others scale is scored based on the presence of current or past history of violence in combination with other behaviors, cognitions, and mental state indicators. The Risk of Harm to Others scale has been found to predict inpatient incidents of violence in a psychiatric hospital.28 The Severity of Self-Harm scale is scored using items on the presence of current or past history of self-harm and suicide attempt in combination with the presence of symptoms and family concern about risk. The scales range from scores of 0, indicating minimal risk, to 6, for very high/imminent risk.

Dependent Variable

The primary outcome was rehospitalization within 90 days of index discharge. In this study, rehospitalization refers to an admission to any designated inpatient mental health bed in Ontario. This is preferable to using only within-hospital readmissions, an approach that has been found to underestimate rehospitalization by 30%.34 The number of days between the date of index discharge and the date of next admission was calculated to assign outcome. Patients who were transferred to another psychiatric hospital within 15 days of the index hospitalization (n = 182) were not identified as being rehospitalized. A continuous variable of the total number of days between discharge and rehospitalization was also defined, as was a dichotomous variable indicating the presence of any rehospitalization, regardless of timeframe.

The 90-day outcome was chosen as the primary outcome to understand risk factors over a medium term. While 30-day rehospitalization is a common outcome1 it may not adequately reflect a system’s ability to respond to the needs of those at highest risk. Instead, 90-day risk may identify opportunities to plan for more stable supports to prevent rehospitalization across care transitions. The 90-day timeframe has been used in non–mental health fields focused on rehabilitation, such as cardiac care35; as recovery and rehabilitation are a primary focus of mental health care, reducing risk of 90-day rehospitalization should be the goal of a well-designed health system.

Analytic Approach

The development of the Rehospitalization Clinical Assessment Protocol was based on the premise that risk identification for care planning should utilize characteristics of the person at admission, rather than service or process variables. While process factors, such as discharges against medical advice, may be related to risk of rehospitalization, such factors cannot be used in care planning at admission. The goal of our analysis was to identify clinical characteristics to allow the care planning process to begin early in a person’s admission.

Univariate analyses were used to describe the variability in rehospitalization using the median, first quartile (Q1), and third quartile (Q3) days to rehospitalization. Chi-square analyses were used to examine differences in rates of rehospitalization among categorical variables. The mean, standard deviation (SD), and 95% confidence intervals (CI) were calculated for all summary scales to describe average severity and distribution of clinical characteristics among those rehospitalized. Odds ratios and 95% CIs were calculated using bivariate logistic regression to examine relationships between these scales and rehospitalization. Similar to analysis in prior research,11 3 forms of multivariate logistic regression models were evaluated: a basic administrative model testing the number of prior hospitalizations as the only predictor variables, an intermediate model that examined the addition of DSM-IV diagnostic categories, and a full clinical model that added sociodemographic and clinical characteristics. Preference was given in the final model selection to variables that could be amenable to intervention rather than static variables. Different combinations of remaining independent variables were examined to rule out order-of-entry, deletion effects, and colinearity.36 The c-statistic was used to examine the discriminatory power of the model such that a c-statistic below 0.7 is considered to have low range of accuracy.37 Confidence intervals for the c-statistics were calculated to compare the accuracy of the models.38

A 3-level Rehospitalization Clinical Assessment Protocol was developed based on variables identified by the final logistic model. Conceptual categories of risk, similar to a decision tree, were created by assigning cases first on the variable with the strongest effect followed by other variables predictive of rehospitalization. Combining variables into risk levels was done to support ease of interpretation in clinical decision-making. A second logistic regression model was then developed to assess the predictive validity of the Rehospitalization Clinical Assessment Protocol. Kaplan-Meier survival analysis was used to examine the proportion of patients at each level of the Rehospitalization Clinical Assessment Protocol over time, censoring at 1-year after index discharge.

RESULTS

The majority of the sample were designated as acute (79.9%), 15% were long-stay (length of stay expected to be > 30 days), and 5.9% were in designated geriatric programs. The rate of 90-day rehospitalization was 10.7% (n = 5,730), while 29.9% (n = 16,034) were rehospitalized at any point following index discharge. Among those rehospitalized within 90 days, the median number of days to rehospitalization was 28 (Q1 = 12, Q3 = 53). Among those who were rehospitalized at any point, the median number of days to rehospitalization was 108 (Q1 = 21, Q3 = 333).

Table 1 shows the distribution of sample characteristics, as well as rates of rehospitalization. Significant bivariate differences in the rate of rehospitalization were identified for all variables except gender; having no confidant; having an anxiety disorder, personality disorder, or secondary DSM-IV diagnosis (excluding substance use disorders); or those admitted due to a threat or danger to others. The majority of the sample was between ages 24 and 65 years, while rates of rehospitalization were higher for 18- to 24-year-old patients or those aged 65+ years. The highest rate of rehospitalization was 16.8% among patients with 3 or more admissions in the prior 2 years. Rates ranged from 11.3% to 14.3% among those who were homeless or live alone, involuntarily admitted, admitted due to a threat or danger to self, or assessed to have no insight into their mental health condition. Patients with schizophrenia or other psychoses or secondary substance use disorders had the highest rates of rehospitalization among diagnoses.

Table 2 lists the clinical summary scale scores from the index admission assessment among those who were and were not rehospitalized, as well as bivariate odds ratios for the likelihood of rehospitalization for each scale. The sample showed a high degree of depressive symptoms (average Depression Severity Index scores greater than 4). Comparing the overlap of CIs, similar average scores on the Depression Severity Index, Pain, Cognitive Performance Scale, and Activities of Daily Living Hierarchy measures were found between those who were and were not rehospitalized. Bivariate odds ratios indicated a significant relationship between having higher scores on the Positive Symptom Scale, Severity of Self-Harm scale, Risk of Harm to Others scale, or Self-Care Index and a likelihood of being rehospitalized within 90 days.

Table 3 shows logistic regression results for predicting rehospitalization. The first bivariate model found that prior psychiatric admissions were strongly associated with rehospitalization but provided limited discriminatory power (c = 0.55). Holding prior hospitalizations constant in model 2, mood disorders, schizophrenia or other psychoses, personality disorders, and secondary substance use disorders increased the risk of rehospitalization, while dementia or other cognitive disorders and primary substance use disorders decreased the risk. Patients with a primary substance use disorder were 37% less likely to be rehospitalized, while the risk increased by 36% among persons with schizophrenia and 11% among patients with a secondary substance use diagnosis. When comparing the c-statistics, the strength of model 2 remained modest but improved upon model 1. The addition of demographic and clinical characteristics in model 3 did not increase model strength; holding prior hospitalizations and diagnoses constant, greater odds of rehospitalization were found for being unemployed, having higher scores, and having higher Severity of Self-Harm scale scores. When diagnoses were excluded in model 4, the effect of prior hospitalizations, Positive Symptom Scale scores, and Severity of Self-Harm scale scores were magnified, but the model strength did not change. Patients with 6 or more lifetime hospitalizations had the greatest risk of rehospitalization with odds 40% higher than those with no prior admissions. Patients with Positive Symptom Scale scores of 3 or more were 23% more likely to be rehospitalized compared to those with Positive Symptom Scale scores of less than 3. Patients with a Severity of Self-Harm scale score greater than 0 were 11% more likely to be rehospitalized. Model 4 was retained for further analysis given the lack of difference in model fit with model 3 and utility of including clinical characteristics rather than diagnoses for informing care planning.

Figure 1 illustrates the assignment of risk levels to the Rehospitalization Clinical Assessment Protocol. Given the strength of the effect of prior hospitalizations, assignment began with a split on the number of psychiatric hospitalizations. The presence of any 1 of the clinical characteristics then determined assignment to levels 1 and 2. Rehospitalization rates increased in a linear direction, from 9.4% for level 0 to 15.4% for level 2. Similarly, the odds of rehospitalization increased by 35% between Clinical Assessment Protocol levels 0 and 1 and were 39% higher between levels 1 and 2.

Figure 2 shows the survival distribution function for the Rehospitalization Clinical Assessment Protocol levels across the number of days to rehospitalization within 1 year. The differentiation between the 3 Clinical Assessment Protocol levels increased as the number of days to rehospitalization increased. For instance, at 30 days, about 9% of those at level 2 compared to about 5% at level 0 were rehospitalized. This difference increased to 14% and 11% at 90 days, 22% and 14% at 180 days, and 30% and 18% by 365 days.

DISCUSSION

This study examined risk of psychiatric rehospitalization using a number of utilization, demographic, and clinical admission characteristics of a large inpatient mental health care sample. Prior hospitalizations, greater severity in several clinical conditions, presence of a secondary substance use diagnosis, and being unemployed predicted rehospitalization within 90 days. Using these variables, we were able to develop a risk indicator, the Rehospitalization Clinical Assessment Protocol, which increasingly differentiates risk across 3 levels over time following discharge.

Our findings expand upon previous literature examining rehospitalization. Consistent with previous studies, prior hospitalizations were identified as the strongest predictor of rehospitalization. This study also identified sociodemographic and clinical factors at admission that predict rehospitalization. Some studies have shown little added effect of clinical factors after considering prior hospitalizations12 while others have found characteristics such as symptom severity and chronicity to be predictive of rehospitalization.8–12 The variation in findings may be due to the way in which rehospitalization is defined (eg, time-to-event vs static definitions) and the availability of person-level data. The severity of positive symptoms or presence of self-harming behavior are indicative of severe conditions that, when combined with prior hospitalizations, reveal persistent symptomatology. The effect of unemployment may indicate a lack of engagement in meaningful activity that could be related to this persistent symptomatology or a lack of opportunities within community settings. Recognizing that full employment may not be possible for those with severe and persistent conditions, supported employment may represent a cost-effective39 opportunity to increase community engagement, provide financial means, and promote recovery.

The effect of substance use disorders and rehospitalization was particularly interesting. Although other diagnoses were excluded, we included primary and secondary substance use disorders as no indicators of addiction severity were available in the data. The impact of the relapsing nature of substance use disorders on repeat hospitalizations has been documented.40 However, our findings suggest that patients with a primary substance use disorder are about 40% less likely to be rehospitalized while the presence of a secondary substance use diagnosis increased risk. Further inquiry is needed to understand these differences. An ad hoc analysis revealed about 47% (n = 3,935) of patients with a primary substance use diagnosis in our sample had no secondary DSM-IV diagnosis. It may be that these patients are primarily engaged in inpatient substance use programs within specialized mental health hospitals, are higher functioning, and have more supports available upon discharge. Given that factors such as positive symptoms and self-harm behavior were held constant, our model suggests that the additional presence of substance use further increases risk of rehospitalization. These findings may illustrate the importance of integrating substance use care within mental health care planning at all levels of care.

The discriminatory strength of models predicting rehospitalization was modest and may be related to the contextual nature of hospitalizations. While the purpose of this study was not to identify all factors predictive of rehospitalization, the limited strength was surprising given the number of variables examined and large sample size. Variable estimates of discrimination have been previously identified in mental health care41 and non–mental health populations.42 It may be that clinical context predisposes risk of rehospitalization while factors at the provider, community, health system, and societal levels mediate or moderate risk. At the provider level, factors such as length of stay, service design and delivery, leaving against medical advice, and discharge disposition may all contribute to risk of further hospitalization, particularly among patients at heightened risk. Further research could explore the quality of care received by those triggering the Clinical Assessment Protocol to identify whether a relationship exists between risk, quality, and subsequent rehospitalization. This research could make use of existing quality indicators, including those based on the interRAI MH.24

Health system factors may also influence patterns of rehospitalization. Rehospitalization may relate to the availability and coordination of transitional support programs,42 psychiatrists in the community,43 and a variety of community-based interventions.44–48 Jurisdictional differences in financing and coordination of mental health services may also influence patterns of rehospitalization. In Ontario, most psychiatric hospitalizations are publically funded wherein the majority of admissions are for acute services (eg, crisis intervention and stabilization). Community-based programming, such as case management, for those with severe and persistent conditions is publically funded but subject to availability. The effectiveness of acute hospitalizations in combination with the availability and engagement of community services may additionally account for rehospitalizations. While programs such as Assertive Community Treatment may be cost-effective compared to hospitalization,39 enrollment capitations or regional variations in the availability of these services may limit their effectiveness in reducing hospitalizations. Further research exploring nested relationships between health system structures and processes, community characteristics, and person-level factors could lead to a greater understanding of preventable hospitalizations as well as surveillance mechanisms to support public mental health decision-making and system improvement.

This study utilized clinical and administrative factors to predict rehospitalization, a strength in comparison to indices using only administrative data.11 However, this study is limited in that it included clinical data from designated inpatient mental health services and not from other acute health settings or those staying in hospital less than 72 hours. Complete clinical data were not available for these settings most likely leading to underestimated rates of rehospitalization in this study. As well, deaths occurring postdischarge could not be adjusted in the estimated rates of rehospitalization. Additionally, other administrative data that may have increased the strength of our models, such as specific medical diagnoses, contact with specific community services, or resource utilization indicators such as adjusted diagnostic groups, were not available. These variables were not available in the data used for this study. Finally, rehospitalization could not be defined as predictable or unnecessary, although efforts were made to reduce confounding by excluding transfers between psychiatric hospitals in assigning rehospitalization. The topic of unnecessary hospitalizations requires further inquiry. While hospitalization due to extreme crisis may be a necessary modality, it may be reasonable to expect systems to prevent such crises following a recent discharge. Although the process of defining unnecessary hospitalization is highly complex, such definitions may improve the accuracy of health system indicators and targeted quality improvement initiatives.

CONCLUSION

Rehospitalization affects the quality of life of persons with mental health and addiction needs as well as the efficiency of the health system. While ongoing hospitalizations may promote safety and ongoing recovery, instances of rapid rehospitalization may be indicative of poor health system performance. The Rehospitalization Clinical Assessment Protocol as part of the interRAI MH assessment system presents a decision support application for identifying increased risk of psychiatric rehospitalization at the point of admission. This application may be useful for care planning to moderate the predisposition to rehospitalization. Further inquiry is needed to understand how a supportive and coordinated health system can prevent rehospitalization.

Author affiliations: School of Public Health and Health Systems, University of Waterloo, Waterloo, Ontario (Drs Perlman and Hirdes); Women’s College Research Institute, Women’s College Hospital, Toronto; and Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada (Dr Vigod).

Potential conflict of interest: Dr Hirdes is supported by the Ontario Homecare Research Chair of the Ontario Ministry of Health and Long Term Care, a sponsor of the study. Drs Perlman and Vigod report no potential conflicts of interest.

Funding/support: This study was part of a larger study funded by the Primary Health Care Transition Fund of the Ontario Ministry of Health and Long Term Care.

Role of the sponsor: This sponsor approved the overall design of this study, but had no influence on the collection, management, analysis, and interpretation of data or in the preparation, review, or approval of the manuscript.

Previous presentation: Pilot analyses related to this study were presented at the 62nd Institute on Psychiatric Services; October 14, 2010–October 17, 2010; Boston, Massachusetts.

Acknowledgments: We thank the Canadian Institute for Health Information and interRAI for their support in managing and sharing the data used in this study. We are also grateful to Jonathan Chen, MSc, at the University of Waterloo for data management support. Mr Chen reports no other conflicts of interest related to the subject of this article.

Additional information: The interRAI MH data used in this study are managed by the Canadian Institute for Health Information (CIHI) as part of their Hospital Mental Health Dataset (HMHD). These data constitute the Ontario Mental Health Reporting System, one of the datasets included in the HMHD. The data were shared with the researchers through a data sharing agreement between CIHI and the Canadian Collaborating Centre for interRAI at the University of Waterloo. Information on these data can be found at http://www.cihi.ca/CIHI-ext-portal/internet/EN/TabbedContent/types+of+care/specialized+services/mental+health+and+addictions/cihi021343.

REFERENCES

1. Hermann RC, Mattke S, Somekh D, et al. Quality indicators for international benchmarking of mental health care. Int J Qual Health Care. 2006;18(suppl 1):31–38. doi:10.1093/intqhc/mzl025 PubMed

2. Madi N, Zhao H, Li JF. Hospital readmissions for patients with mental illness in Canada. Healthc Q. 2007;10(2):30–32. doi:10.12927/hcq.2007.18818 PubMed

3. Lin E, Diaz-Granados N, Stewart DE, et al. Postdischarge care for depression in Ontario. Can J Psychiatry. 2011;56(8):481–489. PubMed

4. Bowersox NW, Saunders SM, Berger BD. Predictors of rehospitalization in high-utilizing patients in the VA psychiatric medical system. Psychiatr Q. 2012;83(1):53–64. doi:10.1007/s11126-011-9182-2 PubMed

5. Durbin J, Lin E, Layne C, et al. Is readmission a valid indicator of the quality of inpatient psychiatric care? J Behav Health Serv Res. 2007;34(2):137–150. doi:10.1007/s11414-007-9055-5 PubMed

6. Bobo WV, Hoge CW, Messina MA, et al. Characteristics of repeat users of an inpatient psychiatry service at a large military tertiary care hospital. Mil Med. 2004;169(8):648–653. PubMed

7. Suzuki Y, Yasumura S, Fukao A, et al. Associated factors of rehospitalization among schizophrenic patients. Psychiatry Clin Neurosci. 2003;57(6):555–561. doi:10.1046/j.1440-1819.2003.01167.x PubMed

8. Webb S, Yaguez L, Langdon PE. Factors associated with multiple re-admission to a psychiatric hospital. J Ment Health. 2007;16(5):647–661. doi:10.1080/09638230701494845

9. Sweetman C, Davies S. Factors involved in psychiatric hospital readmissions: a review of the current literature. Curr Med Lit (Psychiatry). 2004;15:61–66.

10. Hodgson RE, Lewis M, Boardman AP. Prediction of readmission to acute psychiatric units. Soc Psychiatry Psychiatr Epidemiol. 2001;36(6):304–309. doi:10.1007/s001270170049 PubMed

11. Hendryx MS, Russo JE, Stegner B, et al. Predicting rehospitalization and outpatient services from administration and clinical databases. J Behav Health Serv Res. 2003;30(3):342–351. doi:10.1007/BF02287322 PubMed

12. Kolbasovsky A, Reich L, Futterman R. Predicting future hospital utilization for mental health conditions. J Behav Health Serv Res. 2007;34(1):34–42. doi:10.1007/s11414-006-9044-0 PubMed

13. Perlick DA, Rosenheck RA, Clarkin JF, et al. Symptoms predicting inpatient service use among patients with bipolar affective disorder. Psychiatr Serv. 1999;50(6):806–812. doi:10.1176/ps.50.6.806 PubMed

14. Moran PW, Doerfler LA, Scherz J, et al. Rehospitalization of psychiatric patients in a managed care environment. Ment Health Serv Res. 2000;2(4):191–198. doi:10.1023/A:1010108402722

15. Bernardo AC, Forchuk C. Factors associated with readmission to a psychiatric facility. Psychiatr Serv. 2001;52(8):1100–1102. doi:10.1176/appi.ps.52.8.1100 PubMed

16. Averill PM, Hopko DR, Small DR, et al. The role of psychometric data in predicting inpatient mental health service utilization. Psychiatr Q. 2001;72(3):215–235. doi:10.1023/A:1010396831037 PubMed

17. Langdon PE, Yaguez L, Brown J, et al. Who walks through the ‘revolving-door’ of a British psychiatric hospital? J Ment Health. 2001;10(5):525–533. doi:10.1080/09638230120041281

18. van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182(6):551–557. doi:10.1503/cmaj.091117 PubMed

19. Cotter PE, Bhalla VK, Wallis SJ, et al. Predicting readmissions: poor performance of the LACE index in an older UK population. Age Ageing. 2012;41(6):784–789. doi:10.1093/ageing/afs073 PubMed

20. Braes T, Moons P, Lipkens P, et al. Screening for risk of unplanned readmission in older patients admitted to hospital: predictive accuracy of three instruments. Aging Clin Exp Res. 2010;22(4):345–351. doi:10.1007/BF03324938 PubMed

21. Canadian Institute for Health Information (CIHI). Mental health and addictions, 2014. CIHI website. http://www.cihi.ca/CIHI-ext-portal/internet/EN/TabbedContent/types+of+care/specialized+services/mental+health+and+addictions/cihi021343. Accessed April 24, 2015.

22. Hirdes JP, Marhaba M, Smith TF, et al; Resident Assessment Instrument-Mental Health Group. Development of the Resident Assessment Instrument–Mental Health (RAI-MH). Hosp Q. 2000-2001;4(2):44–51. PubMed

23. Gray LC, Berg K, Fries BE, et al. Sharing clinical information across care settings: the birth of an integrated assessment system. BMC Health Serv Res. 2009;9(1):71. doi:10.1186/1472-6963-9-71 PubMed

24. Perlman CM, Hirdes JP, Barbaree H, et al. Development of mental health quality indicators (MHQIs) for inpatient psychiatry based on the interRAI Mental Health assessment. BMC Health Serv Res. 2013;13(1):15. doi:10.1186/1472-6963-13-15 PubMed

25. Hirdes JP, Telegdi NC, Mathias K, et al. interRAI Mental Health Clinical Assessment Protocols (CAPs) for Use With Community and Hospital-Based Mental Health Assessment Instruments, 9.1, Canadian Edition. Ann Arbor, MI: interRAI, 2011.

26. Kehyayan V, Hirdes JP, Perlman CM. Education and employment needs and receipt of services in community and inpatient mental health settings. Community Ment Health J. 2014;50(6):637–645. doi:10.1007/s10597-014-9694-y PubMed

27. Mathias K, Hirdes JP, Pittman D. A care planning strategy for traumatic life events in community mental health and inpatient psychiatry based on the interRAI assessment instruments. Community Ment Health J. 2010;46(6):621–627. doi:10.1007/s10597-010-9308-2 PubMed

28. Neufeld E, Perlman CM, Hirdes JP. Predicting inpatient assaults using the RAI-MH Risk of Harm to Others Clinical Assessment Protocol: a tool for risk assessment and care planning. J Behav Health Serv Res. 2012;39:472–480. doi:10.1007/s11414-011-9271-x PubMed

29. Hirdes JP, Smith TF, Rabinowitz T, et al; Resident Assessment Instrument-Mental Health Group. The Resident Assessment Instrument-Mental Health (RAI-MH): inter-rater reliability and convergent validity. J Behav Health Serv Res. 2002;29(4):419–432. doi:10.1007/BF02287348 PubMed

30. Hirdes JP, Ljunggren G, Morris JN, et al. Reliability of the interRAI suite of assessment instruments: a 12-country study of an integrated health information system. BMC Health Serv Res. 2008;8(1):277. doi:10.1186/1472-6963-8-277 PubMed

31. Canadian Institute for Health Information (CIHI). Depression in Ontario: what predicts a first mental health rehospitalization? CIHI website. https://secure.cihi.ca/free_products/depression_in_ontario_aib_2010_e.pdf. Updated March 2, 2010. Accessed April 24, 2015.

32. Jones K, Perlman CM, Hirdes JP, et al. Screening cognitive performance with the Resident Assessment Instrument for Mental Health Cognitive Performance Scale. Can J Psychiatry. 2010;55(11):736–740. PubMed

33. Morris JN, Fries BE, Morris SA. Scaling ADLs within the MDS. J Gerontol A Biol Sci Med Sci. 1999;54(11):M546–M553. doi:10.1093/gerona/54.11.M546 PubMed

34. Vigod SN, Taylor VH, Fung K, et al. Within-hospital readmission: an indicator of readmission after discharge from psychiatric hospitalization. Can J Psychiatry. 2013;58(8):476–481. PubMed

35. Mudrick DW, Shaffer L, Lalonde M, et al. Cardiac rehabilitation participation reduces 90-day hospital readmissions after acute myocardial infarction or percutaneous coronary intervention. J Am Coll Cardiol. 2013;61(10):E1418. doi:10.1016/S0735-1097(13)61418-7

36. Hosmer DW, Lemeshow S. In: Shewhart WA, Wilks SS, eds. Applied Logistic Regression. 2nd ed. New York, NY: John Wiley & Sons, Inc.; 2005. doi:10.1002/0471722146

37. Streiner DL, Cairney J. What’s under the ROC? an introduction to receiver operating characteristics curves. Can J Psychiatry. 2007;52(2):121–128. PubMed

38. Lasko TA, Bhagwat JG, Zou KH, et al. The use of receiver operating characteristic curves in biomedical informatics. J Biomed Inform. 2005;38(5):404–415. doi:10.1016/j.jbi.2005.02.008 PubMed

39. Latimer EA. Economic considerations associated with assertive community treatment and supported employment for people with severe mental illness. J Psychiatry Neurosci. 2005;30(5):355–359. PubMed

40. Hawkins EJ, Malte CA, Baer JS, et al. Prevalence, predictors, and service utilization of patients with recurrent use of Veterans Affairs substance use disorder specialty care. J Subst Abuse Treat. 2012;43(2):221–230. doi:10.1016/j.jsat.2011.11.002 PubMed

41. Batalla A, Garcia-Rizo C, Castellví P, et al. Screening for substance use disorders in first-episode psychosis: implications for readmission. Schizophr Res. 2013;146(1–3):125–131. doi:10.1016/j.schres.2013.02.031 PubMed

42. Howell S, Coory M, Martin J, et al. Using routine inpatient data to identify patients at risk of hospital readmission. BMC Health Serv Res. 2009;9(1):96. doi:10.1186/1472-6963-9-96 PubMed

43. Vigod SN, Kurdyak PA, Dennis CL, et al. Transitional interventions to reduce early psychiatric readmissions in adults: systematic review. Br J Psychiatry. 2013;202(3):187–194. doi:10.1192/bjp.bp.112.115030 PubMed

44. Kurdyak PA, Stukel TA, Goldbloom D, et al. Universal coverage without universal access: a study of psychiatrist supply and practice patterns in Ontario. Open Med. 2014;8(3):
e87–e99. PubMed

45. Anderson K, Wing H, Womack M. Keys to reducing rehospitalization. Behav Health Manage. 2000;20:38–41.

46. Sledge WH, Dunn CL, Schmutte T. Surprising predictor of rehospitalization. Psychiatr Serv. 2008;59(5):577. doi:10.1176/appi.ps.59.5.577 PubMed

47. Maples NJ, Copeland LA, Zeber JE, et al. Can medication management coordinators help improve continuity of care after psychiatric hospitalization? Psychiatr Serv. 2012;63(6):554–560. doi:10.1176/appi.ps.201100264 PubMed

48. Min SY, Whitecraft J, Rothbard AB, et al. Peer support for persons with co-occurring disorders and community tenure: a survival analysis. Psychiatr Rehabil J. 2007;30(3):207–213. doi:10.2975/30.3.2007.207.213 PubMed