Members of the DORA Advisory Group at UHN included, (clockwise from top left), Drs. Mathieu Albert, James CL. Chow, Ralph S. DaCosta, Anna Gagliardi, Michael Hoffman, Behrang Keshavarz, Pia Kontos, Jenny Liu, Mary Pat McAndrews and Stephanie Protze. (Image: UHN StRIDe Team)

Driven by UHN's mission for research excellence, Dr. Anna Gagliardi, Senior Scientist at Toronto General Hospital Research Institute (TGHRI), led an advisory group to redefine research assessment practices at UHN.

In January 2020, UHN signed the San Francisco Declaration on Research Assessment (DORA). The agreement outlines principles that promote better approaches to assessing research impact than the traditional overreliance on journal and publication metrics.

"This commitment marked an important milestone," says Dr. Brad Wouters, UHN Executive Vice President of Science and Research. "We are changing research assessment in ways that provide a more complete understanding of our researchers' work in order to drive continuous improvement in research quality and impact.

"This is done as part of our larger strategy to advance our vision of A Healthier World."

The DORA Advisory Group, which included researchers from across UHN, was created to put the principles outlined in DORA into action.

Two recent publications outline their approach and progress – one published in PLoS One and the other in Health Research Policy and Systems – and are the first publications featuring researchers from across all six UHN research institutes.

"Initially, we focused on identifying and prioritizing measures for assessing research performance according to the DORA principles," says Dr. Gagliardi. "These measures would help with hiring, annual review, promotion and tenure decisions."

Through a comprehensive review of literature, the group compiled an initial list of 50 ways of assessing research activity and grouped them into eight domains. These domains included but were not limited to research productivity, strategic value, funding, publications, innovations and impact.

Himani Boury, (L), a previous practicum student at the Toronto General Hospital Research Institute (TGHRI), was first author on the Health Research Policy and Systems publication, and Dr. Anna Gagliardi, Senior Scientist at TGHRI, led the advisory group to redefine research assessment practices at UHN. (Photos: UHN StRIDe Team)

"To consolidate and prioritize these measures, we conducted two rounds of surveys with a panel of diverse researchers," says Himani Boury, a previous practicum student at the Toronto General Hospital Research Institute and first author on the Health Research Policy and Systems publication.

The findings led to the identification of the following ten priority research assessment measures:

High ImportanceModerate Importance
Advancing existing knowledgeChallenges to research productivity
Innovative research plansPotential to improve health or health care
Independent funding supportTeam science
Relevant research outputsCollaboration
Publication content qualityRecognition by professional/academic groups

"Our next focus was exploring the potential barriers to implementing and adopting these measures on an organizational level," says Himani. "To do so, we conducted 18 interviews with researchers and research support staff from across UHN's six research institutes.

"This shed light on a several strengths and barriers to implementation."

Participants found the measures to be comprehensive, relevant across disciplines and rigorously developed. The reporting template for the measures was also deemed easy to understand and use. However, concerns were expressed over the measures' relevancy across research disciplines. Another concern raised was that the complexity of the measures would make them more time-consuming to use.

To overcome these barriers, various strategies were suggested, including high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting tools, guidance for evaluators, and sharing of approaches among research institutes.

"Addressing barriers and implementing supportive strategies is crucial for the adoption of prioritized measures. These findings urge research organizations worldwide to reassess evaluation practices, embrace DORA-compliant measures, and work collaboratively to foster a transformative shift towards prioritizing quality and impact in research assessment," concludes Dr. Gagliardi.

The work of the DORA Advisory Group will be continued by the Research Assessment Implementation Committee led by Drs. Michael Hoffman and Gagliardi.

This work was supported by UHN Foundation and The Princess Margaret Cancer Foundation.

Dr. Mathieu Albert is an Emerita/Emeritus Scientist at The Institute of Education Research (TIER) at UHN and an associate professor in the Department of Psychiatry and the Institute of Health Policy, Management and Evaluation at the University of Toronto (U of T). Dr. James CL. Chow is a clinician scientist at Princess Margaret Cancer Centre and an associate professor in the Department of Radiation Oncology at U of T. Dr. Ralph S. DaCosta is a Senior Scientist at Princess Margaret Cancer Centre and holds a Cancer Care Ontario Research Chair in Cancer Imaging. Dr. Anna Gagliardi is a Senior Scientist at the Toronto General Hospital Research Institute and a professor in the Department of Surgery, the Institute of Health Policy, Management and Evaluation and the Institute of Medical Science at U of T. Dr. Michael Hoffman is a Senior Scientist at Princess Margaret Cancer Centre and an associate professor in the Departments of Medical Biophysics and Computer Science at U of T. Dr. Behrang Keshavarz is a Senior Scientist at UHN's KITE Research Institute and an adjunct assistant professor at the Department of Psychology at the Toronto Metropolitan University. Dr. Pia Kontos is a Senior Scientist at UHN's KITE Research Institute and a Professor at the Dalla Lana School of Public Health at U of T. Dr. Jenny Liu was a postdoctoral research fellow at UHN's Krembil Research Institute. Dr. Mary Pat McAndrews is a Senior Scientist at UHN's Krembil Research Institute and a professor in the Department of Psychology at U of T. Dr. Stephanie Protze is a scientist at UHN's McEwen Stem Cell Institute and an assistant professor in the Department of Molecular Genetics at U of T.

Administrative support for the DORA Advisory Group was provided by the Research Analytics Department of UHN's Research Solutions and Services.


More than 23,000 individuals and organizations in 162 countries (coloured in yellow) across the world have signed the San Francisco Declaration on Research Assessment. (Source: sfdora.org)

Quicklinks
Back to Top