Top Hospital Lists Miss the Mark

Jim Kling

May 17, 2011 — Rating systems such as the US News & World Report's "America's Best Hospitals" and HealthGrades' "Best Hospitals" generally fail to identify quality high-volume hospitals, with the exception of some oncologic surgeries. This finding was reported in an article published in the May issue of the Archives of Surgery.

The US News & World Report's "America's Best Hospitals" list identifies the "50 Best" hospitals among 16 medical and surgical specialties. It also names the 50 best cancer hospitals. HealthGrades is an Internet-based resource that assigns report cards to 5000 American hospitals. Both reports are highly publicized and are frequently used by hospitals to market themselves.

The reports are highly influential, but previous studies suggest that they are inconsistent in their ability to identify quality hospitals. Nicholas H. Osborne, MD, from the Michigan Surgical Collaborative for Outcomes Research and Evaluation, Department of Surgery, University of Michigan, Ann Arbor, and colleagues set out to determine whether the 2 reports identified the best hospitals for cancer surgery.

In a retrospective cross-sectional study, the authors used the 2005 to 2006 MedPar data set to determine 30-day mortality in all hospitals for 3 cancer surgery procedures. Included patients were those diagnosed with a malignancy who underwent pancreatectomy, esophagectomy, or colectomy (n = 82,724).

After adjusting for patient factors and surgical acuity, the researchers compared risk-adjusted mortality rates between the hospitals identified as the best by the US News & World Report and HealthGrades and all other hospitals. They also controlled for hospital volume.

US News & World Report best hospitals had lower risk-adjusted mortality rates for all 3 procedures: pancreatectomy (odds ratio [OR], 0.42; 95% confidence interval [CI], 0.30 - 0.58), esophagectomy (OR, 0.48; 95% CI, 0.37 - 0.62), and colectomy (OR, 0.69; 95% CI, 0.55 - 0.86). In HealthGrades' best hospitals, risk-adjusted mortality was lower for colectomy (OR, 0.79; 95% CI, 0.65 - 0.95).

When hospital volume was taken into account, the only statistically significant reduction of risk-adjusted mortality among US News & World Report's best hospitals was for colectomy (OR, 0.79; 95% CI, 0.62 - 0.99). There were no significant differences for HealthGrades' best hospitals.

Study limitations include only accounting for 30-day mortality as measurement of surgical quality; additional relevant factors not assessed include rates of complications and readmissions after surgery, as well as cost. Another limitation is the reliance on Medicare data, which restricts the study population to those older than 65 years.

The authors suggest that to better evaluate hospital quality, "more robust patient and hospital data" should be included.

Internet-Based Rankings Not Ideal, But Gaining Importance

In an accompanying editorial, David C. Linehan, MD, and David Jaques, MD, from the Department of Surgery, Section of Hepatic, Pancreatic, and Biliary Surgery, Washington University School of Medicine, St. Louis, Missouri, note that the 2 reports are inconsistent: few hospitals appear on both lists. The fact that the risk-adjusted mortality rate improvements mostly disappeared after adjustment for volume suggests that the "best hospital" designation in the 2 guides is simply a surrogate for high-volume hospitals.

Nevertheless, in the era of healthcare reform, outcome measures and rankings are likely to gain in importance. "The key to this process is developing and validating scientifically rigorous ways to measure quality and to adjust for risk based on pertinent patient-specific variables. It is clear that the popular, commercially available media-based and internet-based rankings fall far short of this ideal," the editorial authors write.

The authors and editorialists have disclosed no relevant financial relationships.

Arch Surg. 2011;5:600-604, 604-605. Article Abstract, Editorial Extract