management and self evaluation. This is based on what the projects have reported in their exit interviews, their project final reports and includes comments from the NET discussed throughout this report.
Most of the program management and evaluation practice established and implemented
throughout the sub-program was effective. The following are recommendations for improvements at the project and program level.
12.1.1 Project level recommendations
Application process: The submission timeframes are set outside the Christmas / New Year period to enable projects to develop a consultative and thorough document.
Application and/or contract requirements: For future programs the application and/or contract could be enhanced by including some other components as listed following.
Dissemination: A requirement to disseminate is included in the application guidelines or project contract. This could include each project completing a ‘dissemination or communication plan’, either as a stand alone document or perhaps as an activity in the project plan, and then keeping the dissemination log and report progress on the dissemination activities. This would ensure that all projects are aware of this requirement so that they can plan and budget for dissemination activities.
Risk management: Some DoHA programs, such as EBPRAC, include a section in the application form to list risks and management strategies. For the successful projects, a risk management plan is then required with the initial set of deliverables and then this is reported in the progress reports. This process helps projects to consider potential problems and have strategies at the ready to implement.
Final report: A final reporting template was included as a deliverable for all projects in this sub- program. Rather than a table format, perhaps the requirement is changed to ensure that a more comprehensive summary of the project is documented. This would assist with dissemination and generalisability of the projects. This needs to be clearly stated in the application guidelines or project contract to allow the projects to ensure that they plan for this deliverable in terms of time but also skills to be able to complete the final report to a certain standard (such as for CareSearch). Perhaps for future programs a clear document could be provided to guide projects by, for example, stating the potential time frames for final report writing (for instance, allow six weeks for final report writing), with a completed example report and/or the template modified to include standard responses to particular questions as a guide. Evaluation plan and activities: Evaluation section is included in the application form and
guidelines outlining the requirement for each project to conduct a project level evaluation, and requesting a budget line for evaluation activities in the application. Application guidelines are written to explain what evaluation is required to assist projects in this process. Project
Geographical boundaries: Careful review of geographical boundaries to be aware of, and if appropriate, reduce the number of projects with overlapping regions.
The exception may be when there are projects that are geographically isolated.
Panel membership: Select a broader representation of related DoHA and State / Territory programs, and a detailed examination of the purposes and methods of proposed projects to ensure these complement, rather than duplicate, previous projects and resources.
Evaluation: The organisation’s capacity to deliver not just a worthwhile project but also a high quality evaluation is considered in the selection process.
Project planning: The project plans needed to be written in enough detail so that they can be reviewed to ensure that the tasks planned are realistic and achievable in the time frames, with the resources provided.
Project officers (recruitment and retention): Projects are provided with guidelines to ensure the support of project officers, such as an orientation process, adequate office space and equipment, and ongoing support and/or mentorship. This may include information (or a manual) on the skills and possible resources required to manage a project (such as MOU templates), and lists options such as contracting in evaluation expertise and/or report writing assistance, and lists possible training programs (e.g. program management).
Organisational support / partnerships: Applications should include a demonstrated history of providing organisational support for initiatives, and evidence of their capacity to develop and maintain partnerships.
Project evaluation planning and implementation: An evaluation plan is included either as part of or attached to the project plan as a mandatory deliverable. Rather than adding evaluation tasks to a project plan at a later date, it would be preferable to build evaluation into the process from the start. This would mean including evaluation plans, strategies and tools in the final project plans. Rather than simply documenting whether project activities have been completed (i.e. process evaluation), the plan should include details of the kinds of impacts and outcomes expected and how these will be measured. Completion of project plans including evaluation activities and tools could be part of the developmental phase. It could be that the national evaluator reviews and supplies comments on the evaluation plans for DoHA to ‘sign off’, prior to project commencement. Project evaluation tools: Certain evaluation tools could be mandated for projects to use.
Although the projects are diverse, there are coherent themes within the program, and the consistent use of some tools would have allowed some aggregation of data across the program and facilitated comparisons among projects.
Project ethics process: A requirement of projects to obtain ethics approval could be included in the application guidelines and in the project contracts.
Project evaluation training: Information and education on evaluation requirements and
methodologies is provided by the national evaluator. This could be at the first workshop or at state workshops, depending of the needs.
Information and education on writing up evaluation results for final reports is provided by the national evaluator. This could be at the second or third workshops or state workshops. Project resources i.e. timing and funding: Projects incorporate recruitment and project establishment costs in terms of time and money, as well as report writing requirements, in their applications for funding and project planning.
Project activities:
Literature reviews: Depending on the program focus, a generic literature review could be undertaken as a first step, which could be used by all the projects.
Project reporting: One template is developed and distributed early on in the project that includes the program (i.e. contract) and the evaluation reporting requirements.
Project sustainability, generalisability and dissemination: The sub-program application focused on sustainability, generalisability and dissemination but this was not monitored or
reviewed by DoHA except through the national evaluation report. This report is based on what the projects said would happen. If this is a priority it might be useful to conduct a follow-up, such as an on-line survey (Scheirer et al., 2008), where they obtained details on what have been sustained and disseminated (after the funding has ended). If the project’s final reports and /or resources and tools are posted to Care Search, it may be useful to monitor if they are accessed and if possible track if they are used in other settings / services / sectors.
12.1.2 Program level recommendations
Linkages across the Palliative Care Programs: Future programs should include a requirement to link individual program across the Palliative Care program (PCOC, PEPA, NSAP, RPCP, etc) plus across the Commonwealth (EBPRAC) and across each jurisdiction (e.g. NSW Health Representatives at State Forums).
Program activities: There are a number of program activities that would be recommended for future programs.
Site visits: Site visits are continued in other programs. One site visit to all projects in the start up period. This needs to be flexible due to potential (and inevitable) recruitment delays. Plus the additional visits for the second and third years to allow follow-up visits for projects with issues such as new project officer, or data collection or report writing. First time project officers may also need a second visit, depending on their local support systems.
Regular telephone contact: The phone calls which followed each six-monthly progress reports would be a useful communication and support process for other programs.
Workshops: The annual workshops achieved a number of aims and would also be recommended for future programs.
State forums: The State Forums were a new initiative for the sub-program and would be recommended for future programs with the change that all projects are included in a forum. Consideration needs to be given to the commonality of the projects, and well as their
geographical location. If state based, then the inclusion of state / territory representatives would be recommended. Formal leadership, with more detailed implementation guidelines may improve the forums.
Communication within the program: DoHA provides information about other programs and their contribution to the national strategy to ensure all funded projects and their evaluators are aware of other current and recent developments.
Strategic management of the program: A significant number of projects have indicated their sense of isolation in a professional and geographic sense. This suggests that there are
opportunities for a more strategic approach to project selection and overall project management, by ensuring that projects are linked into existing service and policy contexts. For example, there appears to be a strategic approach to project selection and needs identification, as well as
program management (i.e. co-ordination) in Victoria, with strong involvement of the Department of Human Services and the lead role taken by Palliative Care Victoria (PCV). Such an approach is
prospects of sustainability.
Program dissemination: There may be opportunities for DoHA, as funders and administrators of the Care Planning Sub-Program, to take a more active role in engaging key stakeholders and informants around the objectives, opportunities and processes – as well as outcomes - of the sub- program. This would facilitate a receptive context to the work being undertaken at the local level by individual projects, and enhance the potential for project outcomes to be sustained beyond the life of the sub-program.
Care Planning Evaluation reports: The final reports should be made available to the sector, as these reports provide important lessons in terms of project management, evaluation,
dissemination, and sustainability, as well as facilitating a more receptive context to the application of project outcomes.
National evaluation role: If the national evaluator was contracted earlier, then they could be part of the project selection process – such as a technical advisor for evaluation and other relevant components. They would also be able to develop or obtain common evaluation tools that could be used across all the projects.