"A major goal of residency training in the 21st century is to create expert workplace learners who will continue to learn throughout their careers," said Matthew McEvoy, M.D., vice chair for educational affairs and residency program director for the Vanderbilt University School of Medicine Department of Anesthesiology. "As all training programs move into the 'Milestones' era in the ACGME Next Accreditation System this type of near real-time, detailed feedback will be critical to help residents evaluate their practice and provide pathways for improving care."

In this project, researchers created an automated case evaluation tool for use by the anesthesiology program director, the Clinical Competency Committee, and anesthesiology residents. Data on 24,154 completed anesthetic cases from February 2010 to March 31, 2013, measured five metrics: antibiotic administration, glucose monitoring, temperature management, pain management and central line insertion. This tool allows the program director to assess a number of milestones easily and efficiently, and provides residents with near real-time performance feedback.

"Assessing and reporting resident performance on all milestones and core competencies is a daunting undertaking for faculty," said Warren Sandberg, M.D, Ph.D., chair of the Vanderbilt Department of Anesthesiology. "For example, there are 60 residents in our program. There are 25 milestones proposed for anesthesiology and five possible levels of performance that need to be evaluated for each milestone. Each resident is expected to be thoroughly assessed every six months, which means that there have to be 1,500 data points for our program during each evaluation cycle reported to ACGME, along with a report of personal performance for each resident. Finding a tool that moderates the administrative workload while still providing reliable data is a tremendous benefit."

The study found that the majority of the residents who responded to a survey regarding the automated case evaluation tool and evaluation program "agreed or strongly agreed" that they want frequent updates on their personal clinical performance, as well as a comparison of their performance to their fellow residents.

"The training of physicians is undergoing massive changes in the coming years," said Jesse Ehrenfeld, M.D., M.P.H., first author on the study. "This feedback system is one part of a larger system being designed that will ultimately track a resident throughout their training and allow them to graduate when they have reached an appropriate competency level, rather than simply having served as a resident for a prescribed period of time after medical school. Residents want to know the time spent in training is of value to them, because they defer significant income and make modest salaries during residency and fellowship training, while they are also paying for medical school."

Dr. McEvoy and the educational team in the Vanderbilt Department of Anesthesiology report that they have already added several other metrics to the performance dashboard in order to respond to resident requests, such as rate of post-operative nausea and vomiting, and new acute kidney injury in their patients during the perioperative period. Plans are underway to continue to expand the performance metrics to make it a scalable and robust system that provides ongoing, personalized data concerning the quality of care delivered by residents and the program as whole.

"In short," said Dr. McEvoy, "we are creating a performance assessment system that will serve as the backbone of an ongoing quality improvement process in our training programs such that education and delivery of patient care can be closely monitored and regularly improved over time."