HSRAANZ Response to the ARC Engagement and Impact Assessment Paper

Author Professor Jon Karnon

Professor Jon Karnon

A key issue with the impact measurement of Medical and Health Sciences research through the analysis of citations in peer reviewed journals is that Public Health and Health Services research is generally mostly relevant to the jurisdiction in which it is conducted. This means that Public Health and Health Services research undertaken in Australia is less likely to be cited in other jurisdictions than laboratory-based or clinical Medical and Health Sciences research. The world standard for Public Health and Health Services research is driven up by research undertaken and cited by researchers in larger countries, such as the US and UK.

Analyses of citations in peer reviewed journals do not reflect impact on healthcare policy and practice and on patient outcomes and population health within Australia. Practitioners and policymakers do not tend to report in peer reviewed journals, but are more likely to cite research that influences policy and practice in non-peer reviewed government reports and other publications found in the grey literature. These citations are analysed in Google scholar, which consistently report significantly higher citation counts than reported by Scopus and other engines that only count citations in the peer-reviewed literature. The HSRAANZ supports the expansion of citation analyses to include the grey literature.

The HSRAANZ also supports the inclusion of case studies or exemplars of research that has influenced policy and practice in the assessment of research impact and engagement in the Medical and Health Sciences division.

As an example, the HSRAANZ awards an annual prize for the best policy impact health services and policy research project or program. The award is intended to identify and promote examples of outstanding research that have been successfully translated into health policy, management, or clinical practice. Applicants submit a 500-word description of the research to demonstrate an association/temporal relationship between the research and evidence of impact (changes in practice, policy, etc.).

In reviewing case studies or exemplars there is a distinction to be made between research that informs policy and practice and research to implement policy and practice. An example is the difference between researchers who win a government tender to develop guidelines for a particular clinical practice or policy process, and researchers who undertake research that is represented within the guidelines. Likewise, a research team might be contracted to manage the implementation of a proposed program, whilst an alternative research team developed and evaluated the proposed program and another group developed and tested the implementation strategy.
Developing a quantitative metric to assess the relative value of alternative impacts on policy or practice is not straightforward (noting that the seemingly intuitive approach to assessing citations with respect to world standards is subject to potential interdisciplinary bias).

For the HSRAANZ award, a panel review and mark each submission according to the defined criteria, resulting in a ranking of submissions. For a more general research assessment exercise, it might be more appropriate to count significant examples of research translation to active researchers. A grading of demonstrated impact and engagement might be established, for example, impact might be defined as high, moderate or low, whilst engagement may be direct or indirect.

Analogous to the analysis of average citation rates across research papers within an ANZSRC group, the average frequency and grade of policy or practice impact and engagement across eligible researchers assigned to each ANZSRC group could be estimated.

Assessments of research impact based on citation analyses and case studies or exemplars could be reported separately. A single, pooled measure of research impact would be more consensus-based and subjective than the numbers driven assessment of citations (noting that quantitative citation measures may provide false quality assurance by not explicitly accounting for differences in research applicability across jurisdictions).

Whilst the collection of case studies and exemplars is less open to automation than the extraction of references and citations, the process should not be overly onerous. The NHMRC commonly requests evidence of translation in grant applications and every person with an RGMS account must complete their CV, which also includes a section on translation.