Appendix B The National Health and Medical Research Council (NHMRC) Evidence Classification Scale 17
Appendix C META-EVALUATION ASSESSMENT TOOL 19
Appendix D Promising Programs: Case study analysis 25
Appendix E 37 EVALUATED Programs NOT INCLUDED IN THE META-analysis 179
The Aboriginal and Torres Strait Islander Suicide Evaluation Project is funded by the Australian Government through the Department of the Prime Minister and Cabinet. The opinions, comments and analysis expressed in this document are those of the author/s and individual participants and do not necessarily represent the views of the Government and cannot be taken in any way as expressions of Government policy.
A key objective of the Aboriginal and Torres Strait Islander Suicide Prevention Evaluation Project (ATSISPEP) is the development of an evidence base for what works in community-led Aboriginal and Torres Strait Islander suicide prevention.
This report contributes to that objective through:
a meta-evaluation of 16 program evaluations with findings and success factors identified;
cases studies of 19 promising practice programs with strong community leadership or engagement; and
providing an overview of 37 evaluated promising programs that were not within the ATSISPEP Terms of Reference
The project to identify and analyse promising practice involved six steps:
Identify in-scope suicide prevention activities – 88 were identified, of which 51 were selected for further analysis based on them being programmatic and/or previously evaluated. It should be noted that programs commenced or evaluated after December 2015 are not included in this meta-evaluation.
Categorise the 51 programs as ‘promising’ or ‘other’ based on their level of community leadership/ engagement with reference to the National Aboriginal and Torres Strait Islander Suicide Prevention Strategy. Through this, 14 were identified. (NB: The 37 programs that were assessed as ‘other’ are described in Appendix E to this report).
Consult and seek expert opinion on the 14 identified promising evaluated programs by means of a stakeholder survey, and seek advice as to whether there were additional community-led or shaped promising programs that should be added to the list – 5 additional programs were identified (19 in all).
Prepare case studies of the 19 selected programs.
Develop and apply a meta-evaluation analysis framework to:
assess the evaluations which have been done on the evaluated case study programs (11 of the 14 programs were included in this part of the process); and
attempt to assess the 5 unevaluated case study programs using the meta-analysis framework and include these evaluations in the meta-evaluation itself – 16 programs in all.
Distil findings and identify success factors based on the meta-evaluation.
A meta-evaluation analysis is an ‘evaluation of evaluations’. The results of the meta-evaluation analysis are not a critique of individual programs. The analysis simply seeks to compare the strengths and weaknesses of the evaluations previously applied to the programs as a first step to distilling findings and identifying success factors.
The project team identified three categories of information for examination in its meta-evaluation analysis framework:
Whether community leadership and engagement had been effectively evaluated;
Whether quality evaluation indicators were evident in the evaluation; and
Eight of the promising programs were identified as having been the subject of an effective evaluation using the meta-evaluation framework
Two of the promising programs were identified as having been subject to a partially effective evaluation – that is, that some elements of the program were subject to effective evaluation but others were not.
The meta-evaluation assessment enabled identification of broader issues that need exploration in future policy and program development of Aboriginal and Torres Strait Islander suicide prevention programs.
Finding 1: Evaluation should be built into suicide prevention program design. Communities, and service providers exploring new models of suicide prevention activity with communities, should routinely access evaluation expertise before commencing the implementation of a new initiative – otherwise they run the risk of being unable to demonstrate the ongoing value and scalability of their initiative.
Finding 2: Promising programs with strong community engagement and/or leadership are not evenly geographically distributed. The applicability of promising community led or shaped models for suicide prevention needs examination in a broader range of jurisdictions; there are large populations of Aboriginal and Torres Strait Islander people that seem less likely to be able to access promising programs in New South Wales, Victoria, South Australia and Tasmania.
Finding 3:Partnerships between Indigenous communities/providers and general population providers to develop new, or adapt existing suicide prevention programs for use in Indigenous communities should be encouraged. Governments should explore mechanisms for encouraging such partnerships.
Finding 4: Community leadership and engagement in Indigenous suicide prevention activity (universal and selective in the context of Indigenous communities) should be maintained, but strengthened in indicated and postvention services. Leadership groups in Aboriginal and Torres Strait Islander suicide prevention should assess whether there is a way to increase the breadth of new program development by program type beyond the current focus on prevention programs.
Finding 5: The systems approach to suicide prevention should be assessed for its suitability in diverse Indigenous community settings, and otherwise adapted under community leadership/ with community engagement to account for Indigenous cultural and experiential differences in those settings.
Success factors identified from the eight effectively evaluated programs
Success factors identified from the programs that have been effectively evaluated include: