The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use, including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

×
New ResearchFull Access

Randomized Trial of an Electronic Personal Health Record for Patients With Serious Mental Illnesses

Abstract

Objective

The authors evaluated the effect of an electronic personal health record on the quality of medical care in a community mental health setting.

Method

A total of 170 individuals with a serious mental disorder and a comorbid medical condition treated in a community mental health center were randomly assigned to either a personal health record or usual care. One-year outcomes assessed quality of medical care, patient activation, service use, and health-related quality of life.

Results

Patients used the personal health record a mean of 42.1 times during the 1-year intervention period. In the personal health record group, the total proportion of eligible preventive services received increased from 24% at baseline to 40% at the 12-month follow-up, whereas it declined in the usual care group, from 25% to 18%. In the subset of patients with one or more cardiometabolic conditions (N=118), the total proportion of eligible services received improved by 2 percentage points in the personal health record group and declined by 11 percentage points in the usual care group, resulting in a significant difference in change between the two groups. There was an increase in the number of outpatient medical visits, which appeared to explain many of the significant differences in the quality of medical care.

Conclusions

Having a personal health record resulted in significantly improved quality of medical care and increased use of medical services among patients. Personal health records could provide a relatively low-cost scalable strategy for improving medical care for patients with comorbid medical and serious mental illnesses.

Patients with serious mental illnesses are at risk for elevated rates of medical comorbidity and adverse health outcomes, including premature mortality (1). One of the most important factors contributing to these poor health outcomes is deficits in the quality of medical care (2, 3). At the patient level, symptoms of psychiatric illness, such as amotivation (4), cognitive limitations (5), and low health literacy (6, 7), make it challenging for people with serious mental illness to effectively manage their illnesses and obtain needed care. At the provider and system levels, patients with comorbid mental and general medical conditions typically receive care across multiple locations, leading to challenges in coordination of care between mental health and general medical providers (8).

Personal health records hold the potential to improve quality and outcomes of care by providing patients and providers timely access to key health information. Whereas electronic medical records are most commonly developed by individual provider organizations, electronic personal health records shift the ownership and locus of health information from being scattered across multiple providers to the patient (9, 10).

For patients with serious mental disorders, personal health records may be able to provide particular benefits for improving care (11). Personal health records may help patients to better engage in care, facilitate communication across multiple providers, and provide a single record that follows patients across multiple settings. However, the same challenges that patients with mental disorders face in obtaining health services could also adversely affect their ability to use personal health records. To date, there has been almost no research testing the potential feasibility and benefits of personal health records to improve care for persons with mental disorders (12).

We report the results of a randomized trial testing the effects of a mental health personal health record in a sample of patients with serious mental illnesses and comorbid medical conditions. The results can help inform the use of this emerging technology in the care of people with mental disorders, as well as in other vulnerable populations.

Method

Overview

The project adapted an existing community-based personal health record to patients with serious mental illnesses. A 12-month randomized trial tested the effect of the record on the quality of medical care received in an urban community mental health center.

Intervention

My Health Record is an adaptation of the Shared Care Plan, a community-based personal health record developed by providers and by patients with chronic medical conditions to self-manage their illnesses and interact with the health system (13). The core features consist of personal details; diagnoses; goals and action steps; health indicators, including fields for blood pressure and cholesterol and glucose levels; medications and allergies; hospital visits; immunizations; and health and family health history. Prompts remind patients about routine preventive services.

To adapt the health record to the needs of patients with serious mental disorders, a series of focus groups consisting of mental health consumers (two groups), mental health providers (one group), and primary care providers (one group) presented the personal health record and identified potential modifications that were needed for this population. Based on the findings, the following changes were made: 1) rewriting all elements of the personal health record to a sixth-grade reading level to address limited health literacy in the population (7); 2) adding a section establishing mental health and health goals to help patients overcome amotivation and improve patient engagement with self-management and medical care; 3) adding a mental health advanced directive section addressing patient preferences for mental health care in cases in which the patient is unable to make mental health decisions (14); and 4) providing contact information about resources, including grocery stores, YMCAs, and safety net health providers in the patients’ neighborhoods. A screenshot of the My Health Record web interface is presented in Figure 1.

FIGURE 1. My Health Record Screenshot

Because of low levels of digital literacy, a 4-hour computer literacy training curriculum was provided to all patients in the personal health record arm to enable them to effectively use the computer and the personal health record.

Data from the personal health record were stored on an encrypted server, and passwords were required to log in. Clients were able to access the personal health record data with protected passwords from any computer with an Internet connection. Additionally, they were allowed to designate health partners (including physicians, other providers, and friends and/or family), who could obtain access to the personal health record, and to identify the fields that health partners could access. For those without access to a computer, a workstation was provided at the mental health clinic.

A study staff member was available to patients to help orient them to the use of the personal health record and to enter and retrieve data during the first 6 months of the intervention. An initial 1-hour visit orienting patients to the personal health record was followed by 30-minute follow-up visits as needed. On average, patients had 14.8 support visits (SD=5.53), for a total of approximately 8 hours of support time during the study period.

After entering the initial data, clients were encouraged to access the personal health record at least every other week. Each patient printed out a wallet-sized card that provided an overview of his or her medical history, laboratory workup, and medications. Before each medical appointment, patients also printed out a detailed, full-size printout for their providers.

Recruitment and Randomization

Patients with schizophrenia, schizoaffective disorder, bipolar disorder, major depression, or posttraumatic stress disorder and one or more chronic medical condition confirmed through chart review were eligible to participate. Patients were also required to have a primary care provider and to have a minimum of a sixth-grade reading level to ensure full participation in the project. Individuals were recruited through clinician referral or community mental health center waiting room screening or self-referred through recruitment flyers or word-of-mouth. Participants assigned to the usual care group continued to receive medical and mental health care as usual in the community and returned for follow-up interviews but were not provided a personal health record. After complete description of the study to the participants, written informed consent was obtained.

Outcome Assessment

Interviews and reviews of all medical and mental health charts were conducted at baseline and the 12-month follow-up. Quality and service use indicators were assessed through review of all medical and mental health charts; patient activation and health-related quality of life were assessed through patient interviews.

The primary study outcome was quality of medical care, which included 1) quality of preventive services and 2) quality of cardiometabolic care in the subset of individuals with cardiometabolic conditions. Quality of preventive services measures were obtained from the U.S. Preventive Services Task Force guidelines (15). These include four measures of physical examination, eight screening measures, seven vaccination measures, eight education measures, two measures of preventive services for men, and five preventive services measures for women. A score for each domain was calculated as the total number of preventive services for which an individual was eligible that were received by the individual. An aggregate score was developed as the total number of services across all domains for which an individual was eligible that were received by the individual.

Quality indicators for cardiometabolic risk factors were used as a second outcome measure because of the relatively high prevalence, clinical burden, and availability of indicators for this group of conditions. For the subset of individuals with these conditions (N=118), indicators were developed using the RAND Community Quality Index (1618). Cardiometabolic measures included 11 indicators for hypertension, seven measures for diabetes, and six measures for hyperlipidemia. As with preventive services, a score for each group of cardiometabolic indicators was calculated as the total number of services for which an individual was eligible that were received by the individual. An aggregate cardiometabolic quality score was developed as the total number of services across all domains for which an individual was eligible that were received by the individual.

Secondary outcomes included 1) health services use, including mental and medical inpatient, outpatient, and emergency department use; 2) patient activation, as measured by the Patient Activation Measure (19), a 22-item measure of patient skills and confidence in self-management behaviors; and 3) health-related quality of life, using the physical and mental component summary measures of the Medical Outcomes Study 36-Item Short-Form Health Survey (20).

Data Analysis

All analyses were conducted as intent-to-treat. General linear analyses were conducted using the SAS PROC GLM (SAS Institute, Cary, N.C.) procedure to model change in each outcome as a function of being in the personal health record group compared with the usual care group. For dichotomous outcomes, the baseline value was included as a covariate in the regression predicting the measure at the follow-up interview. For continuous outcomes, changes in scores between baseline and follow-up were specified as dependent variables. Two-tailed tests of significance were used for all analyses.

Results

Study Flow, Participant Characteristics, and Use of the Personal Health Record

Of a total of 644 individuals screened, 170 were eligible and randomly assigned; the most common reasons that patients were screened but not enrolled were 1) lack of a regular primary care (N=208) or mental health (N=102) provider and 2) lack of a comorbid chronic medical illness (N=75) (Figure 2). All individuals in the personal health record and usual care groups had complete baseline and 12-month chart data available for the primary quality outcomes; a total of 141 (82.9%) had 12-month interviews.

FIGURE 2. Study CONSORT Diagram

All demographic and clinical parameters were balanced between the intervention and usual care groups (Table 1). The mean age of participants was 49.3 years (SD=7.62). One-half (49.4%) of the participants in the sample were men, and a majority (83.5%) were African American. Most participants were poor, with a mean annual income in the sample of $6,966 (SD=$4,985.13). On average, participants had a mean of 2.3 comorbid medical conditions; the most common mental health conditions were major depression (47.6%) and schizophrenia (27.6%). A total of 118 patients (65.6% of the sample) had one or more cardiometabolic conditions.

TABLE 1. Demographic and Clinical Characteristics of the Study Samplea
CharacteristicStudy Arm
Analysis
Personal Health Record (N=85)Usual Care (N=85)
MeanSDMeanSDp
Age (years)49.37.149.38.10.98
Income (monthly [U.S. dollars])554.8379.4607.9448.60.41
Number of medical comorbidities2.31.52.31.30.74
N%N%p
Male4350.64148.20.76
Race/ethnicity
 White1315.389.40.24
 Black68807487.10.21
 Hispanic0022.40.15
Single4554.23844.70.21
Stable housing6679.57183.50.50
Stable employment4149.45261.20.12
Disability2327.12630.60.61
Medical diagnosis
 Asthma1922.42124.70.72
 Osteoarthritis2934.117200.03
 Chronic obstructive pulmonary disease89.41011.80.61
 Coronary artery disease1011.889.40.61
 HIV22.478.20.08
 Hepatitis1517.71922.40.44
 Active tuberculosis33.522.40.65
 Cardiometabolic conditions5868.26070.60.74
  Diabetes2023.52225.90.72
  Hyperlipidemia3642.42832.90.21
  Hypertension5564.75767.10.75
Mental diagnosis
 Schizophrenia2124.72630.60.39
 Bipolar disorder1416.567.10.06
 Depression4047.14148.20.89
 Substance use disorder22.478.20.09
 Other mental illness44.711.20.17

a For continuous variables, a t test was used, and a chi-square test was used for dichotomous variables.

TABLE 1. Demographic and Clinical Characteristics of the Study Samplea
Enlarge table

Based on data from the personal health record web server, participants used the personal health record a mean of 42.1 (SD=55.0) times during the 1-year intervention period.

Effects on Quality of Medical Care

The total proportion of eligible preventive services received increased in the personal health record group (from 24% at baseline to 40% at the 12-month follow-up, compared with a decline in the usual care group from 25% to 18%), resulting in a significant difference in change between the personal health record and usual care groups (p<0.001) (Table 2). Compared with the usual care group, the personal health record group had significantly greater improvements in rates of physical examination (p<0.001), screening (p=0.02), vaccination (p<0.001), and education (p<0.001). The overall rate of preventive service use in the personal health record and usual care groups is presented in Figure 3.

TABLE 2. Quality of Preventive and Cardiometabolic Care
VariableStudy Arm
Analysis
Personal Health Record
Usual Care
MeanSDMeanSDFdfpa
Quality of preventive services
Physical examination12.781690.0005
 Baseline0.530.180.550.14
 12-Month interview0.550.130.460.21
Screening5.761690.02
 Baseline0.290.130.330.16
 12-Month interview0.210.160.180.15
Vaccination20.13169<0.0001
 Baseline0.080.120.080.13
 12-Month interview0.190.20.060.09
Education153.82168<0.0001
 Baseline0.170.160.170.16
 12-Month interview0.730.340.150.16
Preventive care for women0.19850.66
 Baseline0.320.370.270.33
 12-Month interview0.270.310.270.3
Preventive care for men0.4830.53
 Baseline0.160.280.150.23
 12-Month interview0.140.270.090.19
Total percentage of eligible services receivedb99.35169<0.0001
 Baseline0.240.10.250.1
 12-Month interview0.40.140.180.11
Quality of cardiometabolic services
Hypertension9.981690.002
 Baseline0.730.240.780.2
 12-Month interview0.780.190.70.3
Hyperlipidemia0.04340.85
 Baseline0.870.340.920.28
 12-Month interview0.710.460.860.35
Diabetes3.02360.09
 Baseline0.420.250.580.25
 12-Month interview0.540.220.510.27
Total percentage of eligible services receivedb9.391690.003
 Baseline0.730.240.780.2
 12-Month interview0.750.20.670.31

a The data represent the values for group type (personal health record compared with usual care), the key independent variable of interest, without adjusting for the number of outpatient medical visits.

b The data indicate the proportion of services for which a participant was eligible and obtained.

TABLE 2. Quality of Preventive and Cardiometabolic Care
Enlarge table
FIGURE 3. Personal Health Record and Rate of Receipt of Indicated Preventive Care Services

In the sample of patients with cardiometabolic conditions (N=118), the total proportion of eligible cardiometabolic services received improved by 2 percentage points in the personal health record group but declined by 11 percentage points in the usual care group, resulting in a significant difference in change between the two groups (p=0.003) (Table 2). In the personal health record arm, there was a significantly greater improvement in care for hypertension (p=0.002) but not for diabetes or hyperlipidemia.

Effects on Secondary Outcomes

Participants in the personal health record group had a significant increase in the number of outpatient medical visits compared with those in the usual care group (mean increase: 14.9 [SD=10.71] compared with 0.5 [SD=11.15], p<0.001) (Table 3). There were no significant changes in other measures of inpatient, outpatient, or emergency department services.

TABLE 3. Service Use, Patient Activation, and Health-Related Quality of Life
VariableStudy Arm
Analysis
Personal Health RecordUsual Care
N%N%Fdfpa
Services use (dichotomous)
Inpatient mental health hospitalization1.031680.16
 Baseline11.211.2
 12-Month interview0022.4
Emergency department mental health visit3.681680.68
 Baseline2327.11416.5
 12-Month interview1011.878.3
Inpatient medical hospitalization3.011680.68
 Baseline33.544.7
 12-Month interview1011.81214.3
Emergency department medical visit3.61680.85
 Baseline5058.84148.2
 12-Month interview3945.93845.2
MeanSDMeanSDFdfpa
Services use (continuous)
Number of outpatient mental health visits2.651680.11
 Baseline12.18.111.48.1
 12-Month interview21.523.815.421.9
Number of outpatient medical visits73.36168<0.0001
 Baseline12.29.813.114.3
 12-Month interview27.111.513.712.2
Patient activation measure (100 possible score)0.021380.90
 Baseline56.114.355.614.4
 12-Month interview58.912.559.215.7
Medical Outcomes Study 36-Item Short-Form Health Survey composite indices
 Physical component measure0.851390.36
  Baseline33.411.232.79.2
  12-Month interview33.49.733.68.9
 Mental component measure0.641390.42
  Baseline33.410.933.811.1
  12-Month interview34.610.936.211.3

a The data represent the values for group type (personal health record compared with usual care), the key independent variable of interest.

TABLE 3. Service Use, Patient Activation, and Health-Related Quality of Life
Enlarge table

Both groups exhibited small improvements in patient activation, physical health-related quality of life, and mental health-related quality of life; however, none of these changes differed significantly between the two groups (Table 3).

Mediation Analysis

To explore whether the number of visits might explain a portion of the effect on quality measures, an exploratory mediation analysis was conducted examining whether controlling for the number of outpatient medical visits attenuated the association between the intervention and the quality scores. After adjusting for outpatient use of services, only education and total number of preventive services remained significant (p<0.001). The magnitudes of differences in preventive services changes and cardiometabolic care changes between the personal health record and usual care groups all decreased substantially, except for preventive services for women, preventive services for men, and quality of hyperlipidemia treatment (Table 4).

TABLE 4. Quality of Preventive and Cardiometabolic Care (Mediation Analysis)
Quality of CarePrediction of Service Change Between Baseline and Follow-Up Interview
Without Adjustment for the Number of Outpatient Medical Visits
With Adjustment for the Number of Outpatient Medical Visits
CoefficientapbCoefficientapb
Preventive services
Physical examination0.1180.00050.0210.58
Screening0.0710.020.0370.30
Vaccination0.128<0.00010.0380.24
Education0.587<0.00010.441<0.0001
Preventive care for women0.0430.66–0.0650.58
Preventive care for men0.0380.530.0600.42
Total percentage of eligible services receivedc0.228<0.00010.140<0.0001
Cardiometabolic services
Hypertension0.1290.0020.0740.12
Hyperlipidemia–0.0330.85–0.1940.28
Diabetes0.1520.090.1190.29
Total percentage of eligible services receivedc0.1270.0030.0820.09

a The data represent the coefficient of group type (personal health record compared with usual care).

b The values for group type are presented.

c The data indicate the proportion of services for which a participant was eligible and obtained.

TABLE 4. Quality of Preventive and Cardiometabolic Care (Mediation Analysis)
Enlarge table

Discussion

In a sample of patients with serious mental illnesses and comorbid medical conditions, having a personal health record was associated with improved quality of preventive care and cardiometabolic care, as well as increased use of general medical services. There were no evident benefits regarding patient activation, quality of life, or other measures of service use.

An increasing number of studies have found that personal health records can improve rates of routine preventive services in general medical populations (21, 22). Our study demonstrated that a poor and complex population of people with serious mental illnesses can derive similar benefits from these new technologies. In contrast to most personal health records, which are developed as extensions of electronic health records, the community-based personal health record in this study made it possible for patients to include information from multiple providers, a high priority for populations whose care may be scattered across multiple organizations.

While the study showed that implementing such programs is feasible, particularly for this population, they need to be implemented with adequate support. In this study, computer training helped address deficits in computer literacy, and technical assistance was important in helping patients enter data and access their personal health record. As with other health technology interventions, implementation studies are needed to identify which training and technical assistance approaches can most efficiently allow patients to reap the benefits (23).

What allowed patients with a personal health record to obtain higher quality of medical care? Analyses suggested that greater rates of medical utilization in the personal health record group appeared to be an important driver of higher quality of preventive and cardiometabolic services. Underuse of medical services commonly underlies quality deficits in this population (2); increasing patients’ use of services, in turn, may help foster improved quality of care.

Several limitations to this study should be noted. The study was conducted in a single, urban community mental health center. Further work is needed to establish generalizability to other mental health settings. Second, the study was only conducted among the subset of patients with a regular mental health and medical provider. For populations without a regular source of care, addressing access barriers may be needed before implementing interventions such as personal health records, in which the goal is to improve quality and engagement in treatment. Finally, reflecting the state of technology at the time the study was conducted, the web-based personal health record relied on participants to enter all data and access the record through desktop computers. Exchange of health information across providers (24), along with patient portals (25), will increasingly make it possible to directly synchronize with clinical and laboratory data, reducing patient burden and increasing utility of personal health records as tools for coordinating care. The increasing ubiquity of smartphones may further increase the ability of patients to access health records and communicate with providers whenever and wherever the need arises (26, 27). Further research is needed to examine the benefits of these new technologies in improving care.

This study demonstrated that personal health records hold potential to improve the quality of care among individuals with serious mental illnesses treated in public mental health settings. More generally, personal health records point to the promise of new health technologies for improving care in vulnerable populations who have traditionally not been included in developing the interventions or in evaluating their effectiveness. As these technologies are developed and disseminated in the coming years, it will be essential to ensure that they are available to, as well as tested in, patients with serious mental illnesses and other disadvantaged populations.

Patient Perspective

“Ms. J” is a 58-year-old African American woman with a history of schizophrenia, diabetes, and hypertension who participated in the personal health record arm of the study. She received her mental health care from a community mental health center and her medical care from a federally qualified health center in the neighborhood. She often left her appointments feeling overwhelmed and uncertain as to how to manage her health. She did not own a computer but had some experience in using e-mail and the web and was able to access a computer from the public library.

During the study, she accessed the personal health record on average once per week, using it to update her medications and health goals. She brought a printout of her personal health record to each of her medical and mental health appointments and began attending her primary care physician visits more regularly. At her 12-month chart review, her receipt of needed cardiovascular services increased from 40% to 75%; her mean blood pressure improved from 159/90 mmHg to 130/81 mmHg, and her fasting blood glucose levels went from 90 mg/dL to 78 mg/dL.

At her final interview, the patient described her experience with the record as follows: “My Health Record helped me understand my health conditions and helps me keep track of my weight and my blood pressure. I gave printed copies of my personal health record to all my providers, and this has made me more confident when talking to them. I feel better prepared and more organized now when I meet with my doctors.”

From the Rollins School of Public Health and the Department of Health Policy and Management, Emory University, Atlanta.
Address correspondence to Dr. Druss ().

The authors report no financial relationships with commercial interests.

Funded by a grant (R18HS017829) from the Agency for Healthcare Research and Quality.

References

1 Parks J, Svedsen D: Technical Report: Morbidity and Mortality in People with Serious Mental Illness. Alexandria, Va, National Association of State Mental Health Program Directors, Medical Directors Council, 2006Google Scholar

2 Mitchell AJ, Malone D, Doebbeling CC: Quality of medical care for people with and without comorbid mental illness and substance misuse: systematic review of comparative studies. Br J Psychiatry 2009; 194:491–499Crossref, MedlineGoogle Scholar

3 Mitchell AJ, Lord O: Do deficits in cardiac care influence high mortality rates in schizophrenia? a systematic review and pooled analysis. J Psychopharmacol 2010; 24(suppl):69–80Crossref, MedlineGoogle Scholar

4 Foussias G, Remington G: Negative symptoms in schizophrenia: avolition and Occam’s razor. Schizophr Bull 2010; 36:359–369Crossref, MedlineGoogle Scholar

5 Heinrichs RW, Zakzanis KK: Neurocognitive deficit in schizophrenia: a quantitative review of the evidence. Neuropsychology 1998; 12:426–445Crossref, MedlineGoogle Scholar

6 Dickerson FB, Kreyenbuhl J, Goldberg RW, Fang L, Medoff D, Brown CH, Dixon L: A 5-year follow-up of diabetes knowledge in persons with serious mental illness and type 2 diabetes. J Clin Psychiatry 2009; 70:1057–1058Crossref, MedlineGoogle Scholar

7 Dickerson FB, Goldberg RW, Brown CH, Kreyenbuhl JA, Wohlheiter K, Fang L, Medoff D, Dixon LB: Diabetes knowledge among persons with serious mental illness and type 2 diabetes. Psychosomatics 2005; 46:418–424Crossref, MedlineGoogle Scholar

8 Druss BG, Walker ER: Mental disorders and medical comorbidity. Synth Proj Res Synth Rep 2011; (21):1–26MedlineGoogle Scholar

9 Bonander J, Gates S: Public health in an era of personal health records: opportunities for innovation and new partnerships. J Med Internet Res 2010; 12:e33Crossref, MedlineGoogle Scholar

10 Fricton JR, Davies D: Personal health records to improve health information exchange and patient safety, in Advances in Patient Safety: New Directions and Alternative Approaches, vol. 4. Edited by Henriksen KBattles JBKeyes MAGrady ML. Rockville, Md, Technology and Medication Safety, 2008Google Scholar

11 Fetter MS: Personal health records. Issues Ment Health Nurs 2009; 30:652–654Crossref, MedlineGoogle Scholar

12 Ennis L, Rose D, Callard F, Denis M, Wykes T: Rapid progress or lengthy process? electronic personal health records in mental health. BMC Psychiatry 2011; 11:117Crossref, MedlineGoogle Scholar

13 Congral: Shared Care Plan. https://www.sharedcareplan.org/HomePage.aspx (Accessed January 4, 2013)Google Scholar

14 Srebnik DS, La Fond JQ: Advance directives for mental health treatment. Psychiatr Serv 1999; 50:919–925LinkGoogle Scholar

15 US Preventive Services Task Force: Recommendations for Adults. Rockville, Md, US Preventive Services Task Force, 2013Google Scholar

16 McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA: The quality of health care delivered to adults in the United States. N Engl J Med 2003; 348:2635–2645Crossref, MedlineGoogle Scholar

17 Brook R: The RAND/UCLA appropriateness method, in Clinical Practice Guideline Development: Methodology Perspectives. Edited by McCormick KMoore SSiegel R. Rockville, Md, Agency for Health Care Policy and Research, 1994, pp 59–70Google Scholar

18 Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE: The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med 1998; 338:1888–1895Crossref, MedlineGoogle Scholar

19 Hibbard JH, Stockard J, Mahoney ER, Tusler M: Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004; 39:1005–1026Crossref, MedlineGoogle Scholar

20 Ware JE, Kosinski M, Bayliss MS, McHorney CA, Rogers WH, Raczek A: Comparison of methods for the scoring and statistical analysis of SF-36 health profile and summary measures: summary of results from the Medical Outcomes Study. Med Care 1995; 33(suppl):AS264–AS279Crossref, MedlineGoogle Scholar

21 Lau AY, Sintchenko V, Crimmins J, Magrabi F, Gallego B, Coiera E: Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial. J Am Med Inform Assoc 2012; 19:719–727Crossref, MedlineGoogle Scholar

22 Wright A, Poon EG, Wald J, Feblowitz J, Pang JE, Schnipper JL, Grant RW, Gandhi TK, Volk LA, Bloom A, Williams DH, Gardner K, Epstein M, Nelson L, Businger A, Li Q, Bates DW, Middleton B: Randomized controlled trial of health maintenance reminders provided directly to patients through an electronic PHR. J Gen Intern Med 2012; 27:85–92Crossref, MedlineGoogle Scholar

23 Druss BG, Dimitropoulos L: Advancing the adoption, integration and testing of technological advancements within existing care systems. Gen Hosp Psychiatry 2013; 35:345–348Crossref, MedlineGoogle Scholar

24 Fontaine P, Ross SE, Zink T, Schilling LM: Systematic review of health information exchange in primary care practices. J Am Board Fam Med 2010; 23:655–670Crossref, MedlineGoogle Scholar

25 Osborn CY, Mayberry LS, Mulvaney SA, Hess R: Patient web portals to improve diabetes outcomes: a systematic review. Curr Diab Rep 2010; 10:422–435Crossref, MedlineGoogle Scholar

26 Smith A: Smartphone Adoption and Usage. Washington, DC, Pew Research Center, 2011Google Scholar

27 Boulos MNK, Wheeler S, Tavares C, Jones R: How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX. Biomed Eng Online 2011; 10:24Crossref, MedlineGoogle Scholar