The Commission employed a consultative approach to the Progress Evaluation Audit, providing weekly updates for the Ambulance Victoria Workplace Reform team, conducting sense-checking sessions with the Staff Reference Group and leadership personnel, and delivering senior leadership briefings. Monthly updates from the Commissioner have regularly informed staff and the broader Victorian public about progress. The Commission aimed to provide a respectful, trauma-informed model of listening and consultation that will inform Ambulance Victoria’s continued reform efforts towards workplace equality.
The purpose of the Progress Evaluation Audit is to determine the extent to which Ambulance Victoria has moved towards compliance with the Equal Opportunity Act 2010 (Vic).
The objectives of the Progress Evaluation Audit were to:
assess the extent and adequacy of the steps taken to implement priority recommendations, including whether those steps align with the intent of the Commission’s recommendation
assess some early impacts arising from the implementation of relevant priority recommendations, including early impacts on the workforce and target audiences
identify the barriers and enablers of successful implementation, any areas for further improvement and related further guidance to support implementation.
Accordingly, 25 priority recommendations were selected for the Progress Evaluation Audit by agreement with Ambulance Victoria, based on the following criteria:
the impact of the reform is significant for the workforce or, for example, for improving oversight of the issues identified in Phase 1
where the Commission identified a reform as a priority through the wording of its recommendation (for example, as soon as practicable, by x months after etc.)
where Ambulance Victoria identified a reform as a flagship reform in the Your AV Roadmap
where implementation of a reform is needed as soon as practicable to address significant risks (for example, of harm to people)
where the reform is foundational and essential to the successful implementation of the broader package of reforms.
While the Commission considered and assessed Ambulance Victoria’s progress towards embedding reforms to achieve long-term change, it did not examine the implications of reform efforts on the long-term intended outcomes of the priority recommendations. Given the significant number of recommendations delivered to Ambulance Victoria in Phase 1 and the stage of implementation, the Commission concluded that it would be premature to undertake such an assessment.
Phase 3 of the Review did not reassess the extent of unlawful and harmful workplace behaviours which were considered during Phase 1. The Commission notes that it did not expect the rates of unlawful and harmful workplace behaviours to have shifted significantly since Phase 1 of the Review given the relatively short time that had passed between these phases. Re-examination of these issues was therefore unnecessary and inconsistent with the Commission’s trauma-informed approach.
To conduct the Phase 3 Progress Evaluation Audit, the Commission collected and analysed data from a range of sources.
The Commission requested documentation from Ambulance Victoria to support evidence of implementation across the 25 priority recommendations. In response to this request, Ambulance Victoria supplied the Commission with over 1,900 documents.
Other documentary evidence that informed the Commission’s analysis included Ambulance Victoria’s Safe Fair Inclusive: Your AV Roadmap 2022–27, the Your AV Roadmap – Outcomes Focussed Model, its online Recommendations Implementation Tracker, the Your AV Impact Report 2022-23, 2021–2023 Ambulance Victoria People Matter Survey results, advice provided by the Commission in response to requests during Phase 2 and relevant academic literature.
Ambulance Victoria has a workforce of over 7,000 people spread across Victoria. To access as many members of the workforce as possible, the Commission developed a 53-question survey that was distributed to all current employees and first responders (see the definitions used in the survey and report at Appendix A). The survey was open between 14 June and 7 July 2024. The survey was developed with the assistance of an external survey provider engaged by the Commission, ORIMA Research (ORIMA), who have extensive experience and expertise in quantitative research including survey design and analysis. The Commission also sought the advice of the Department of Premier and Cabinet’s Behavioural Insights Unit on the survey design, to facilitate uptake and ensure the quality of data collected.
The survey asked about people’s:
engagement and satisfaction with the workplace equality reforms made in response to the priority recommendations from the Commission’s Phase 1 Review
perspectives on the progress of implementation of the recommendations, both positive and negative
views relating to specific priority recommendations from the Commission’s original Review that assisted us to understand the early impacts and adequacy of implementation measures.
The survey also collected two types of demographic data: work and personal. The former asked questions about respondents’ work with Ambulance Victoria, such as tenure, work status and role, while the latter collected more general data including gender, background and age.
Prior to release of the survey, cognitive testing was undertaken by ORIMA with 10 members of the Ambulance Victoria workforce (representing both operational and non-operational roles), to ensure the language of the questions would make sense to the intended audience.
The online survey could be completed using an internet-enabled device with a web browser, including computers, tablets and smartphones. Each participant accessed a unique link to the survey, delivered automatically to their work email, to ensure they could complete the survey only once.
ORIMA administered the survey using a list of names and email addresses provided by Ambulance Victoria. Contact information was deleted by ORIMA following completion of the survey. Participation in the survey was voluntary and strictly confidential. Employees and first responders were not asked to provide their names when completing the survey and ORIMA only provided the Commission with aggregated, de-identified responses.
A copy of the survey questions can be found at Appendix B.
Group and individual interviews and focus groups were conducted as part of the data collection for the Progress Evaluation Audit. These occurred online or person and were conducted by two Commission team members. Interviews and focus groups were semi-structured, including questions predetermined by the Commission as well as flexibility for discussion of any other matters that participants wanted to share with the Commission.
Everything shared in interviews and focus groups was confidential. The Commission used the information discussed to identify the themes and issues that are detailed in this report. No identifying information has been published or shared outside of the Commission’s Progress Evaluation Audit team.
The Commission conducted group interviews with members of Ambulance Victoria’s Board and separately with its Executive. These discussions invited participants to share their reflections on implementation progress and the barriers and enablers to implementation of the Commission’s recommendations.
A further 5 group interviews were conducted to bring together internal subject matter experts and leaders with oversight of, and accountability for, priority programs of reform. These focused on central themes relating to the 25 priority recommendations:
workplace reform
complaints
flexibility
learning and development
safety and prevention.
The Commission gave all individuals who were invited to participate in a group interview the opportunity to participate in an individual confidential interview, as an alternative or in addition to the group interview.
Individual interviews were offered to several Ambulance Victoria senior leaders with significant roles in the implementation and direction of the reforms to share their perspectives in their own words.
The Commission and Ambulance Victoria worked together to identify these relevant internal personnel and the Commission then invited individuals to participate in an interview. Interviews were entirely voluntary and all individuals were informed by the Commission that they could decline to participate.
In total, the Commission conducted 4 individual interviews.
The Commission spoke with key stakeholder organisations including representatives from the Victorian Ambulance Union, Ambulance Employees Australia – Victoria, Ambulance Managers and Professionals Association, Victorian Department of Health, WorkSafe Victoria and the Commission’s dispute-resolution team.
The Commission conducted 4 focus groups with:
representatives of the current Staff Reference Group
representatives of former staff reference groups
regional directors
workforce members with roles or job functions related to diversity and inclusion at Ambulance Victoria (to assist the Commission in gaining an intersectional perspective on the reforms).
These voluntary sessions broadly explored people’s perceptions of the reforms to date, both positive and negative, and what they saw as enablers and barriers to reform.
Participants were also asked for their perceptions of the early impacts of reform areas including complaints, safety, flexibility and governance.
The session with regional directors also provided an opportunity for the Commission to hear about the implementation of the reforms from people with leadership and management responsibilities within Ambulance Victoria.
The Commission’s Progress Evaluation Audit team was invited by Ambulance Victoria to participate in visits to sites where paramedics were working, typically hospitals and branch locations. The Commission visited the Gippsland and Metro 2 regions, and had the opportunity to confidentially hear from operational and non-operational staff on matters related to the reforms that were important to them.
The Commission invited members of the current and former staff reference groups and (separately) regional directors to sessions where the Commission presented some of the emerging findings identified through the above-described data collection methods. The purpose of these 2 sessions was to sense-check whether the emerging findings around reform implementation progress accurately captured people’s perceptions and experiences as members of the workforce.
The Commission took both quantitative and qualitative approaches to analysing data from the above-described sources. Through this approach, the Commission formed an evidence-based and rigorous assessment of the implementation extent and adequacy, early impacts, and implementation barriers and enablers for each priority recommendation.
The Commission’s Progress Evaluation Audit team reviewed extensive documentation and data provided by Ambulance Victoria evidencing the extent of implementation of the 25 priority recommendations. This included new and amended policies, procedures and plans, progress planning documents, relevant data and reports.
The qualitative data analysis software NVivo was used to facilitate analysis of transcribed audio-recordings from each interview and focus group.
The Commission used both deductive and inductive content analysis approaches to code the transcripts. Content analysis is a method of analysing written, verbal or visual communication.[3] Deductive codes related to specific recommendations or groupings of recommendations (for example, complaints, safety, governance) and inductive codes were derived through finding common groupings of responses (for example, organisational culture, transparency, structural issues).
[3] Satu Elo and Helvi Kyngäs, ‘The qualitative content analysis process’ (2008) Journal of Advanced Nursing 62(1) 107.
Ambulance Victoria advised the Commission it has a current workforce of 7,293. The workforce survey obtained 1,389 responses, representing a response rate of 19%. By comparison, the survey of Ambulance Victoria employees and first responders undertaken by the Commission during Phase 1 in 2021 had a response rate of 25%, with 1,859 respondents out of 7,580 employees/first responders at the time.
The key work-based characteristics of workforce survey respondents in 2024 were as follows:
similar to the profile of the 2021 survey, 97% of respondents were current employees, with 3% being first responders/volunteers (compared to 92% and 8% respectively)
work-status and length-of-service profiles were also similar to those in 2021, with 85% being full-time employees (compared to 79% in the 2021 survey) and 55% having worked at Ambulance Victoria for more than 10 years (compared to 52% in 2021)
team composition was also similar, with 58% of respondents working in teams with roughly equal numbers of men and women (compared to 57% in 2021).
The demographic characteristics of the 2024 survey respondents were not significantly different to the population group of all Ambulance Victoria employees and the 2021 survey respondent profile, where comparable.
Ambulance Victoria provided the Commission with staff headcount numbers for each work location and across different roles to provide a comparison on the representativeness of survey respondents. Representation was very consistent across work location, with 61% of survey respondents in a metropolitan location (compared to 60% across all employees) and 7–9% across all other locations (compared to 7–8%).
There were some small differences in representation across work roles between the workforce and those who responded to the survey:
51% of survey respondents were Advanced Life Support (ALS) paramedics, compared to 53% of the workforce
18% were in a non-operational role, compared to 11% of the workforce
9% were in a Mobile Intensive Care Ambulance (MICA)/MICA flight paramedic[4] role, compared to 6% of the workforce.
While the last 2 roles above were slightly over-represented in the survey, their relative differences have had a minimal impact on results for the survey.
In this Progress Evaluation Audit report, where workforce survey responses are reported: ‘agree’ combines workforce survey responses for ‘agree’ and ‘strongly agree’; ‘disagree’ combines responses for ‘disagree’ and ‘strongly disagree’; ‘confident’ combines responses for ‘completely confident’ and ‘very confident’; ‘satisfied’ combines responses for ‘very satisfied’ and ‘satisfied’; and ‘dissatisfied’ combines responses for ‘dissatisfied’ and ‘very dissatisfied’.
Throughout this report, we provide aggregated response data from the survey (for example, ‘56% of survey respondents agreed that …’). These percentages are based on the number of valid responses to the survey question. Valid responses are those where the question was applicable to the respondent and the respondent expressed a view. That is, ‘don’t know’ and ‘not applicable’ responses are usually not counted unless doing so significantly aids in the interpretation of the results (we note specifically in the report where this is the case). To improve readability of the report, the Commission does not report the number of valid responses that each percentage is based on. Rather, it is reported as ‘n=x’ value by exception where the percentage is based on fewer than 50 valid responses.
[4] Note: the workforce survey grouped MICA/MICA flight paramedics under the one occupational category, in line with the occupational categories supplied to the Commission by Ambulance Victoria for the purposes of the workforce survey.
In September 2024 the Commission met with Ambulance Victoria organisational leaders who had accountabilities for implementing priority recommendations. At this workshop the Commission presented early findings including an assessment of the implementation progress of each recommendation. The Commission also gave an overview of 8 thematic barriers and enablers to reform identified during data collection and analysis. The purpose of this workshop was to ascertain what further information Ambulance Victoria would like in the report to assist it to understand the Commission’s findings.
As the workforce survey (both this survey and the 2021 survey) was conducted as a census, the results are not subject to sampling measurement error (and hence statistical precision margin-of-error calculations are not applicable). However, the results are subject to non-sampling measurement error. Unlike sampling error, non-sampling error is generally not mathematically measurable. The main non-sampling error risk with the survey conducted is the potential for non-response bias to affect results, considering that the response rate for the 2024 survey was lower (17% compared to 25% in 2021). Non-response bias arises if the people who respond to the survey differ systematically to non-respondents in terms of characteristics relevant to the survey, specifically, demographic characteristics, and experiential and psychographic characteristics.
While the Commission endeavoured to capture a diverse mix of perspectives on implementation via the Progress Evaluation Audit, it was not possible to speak in depth to all individuals who had something to share. All employees and first responders could complete the online survey and, while all enquiries were responded to, these have not been included in the Progress Evaluation Audit.
