Five-Year Departmental Evaluation Plan 2023-2024 to 2027-2028 (2024)

Co-development and Co-creation with Indigenous Partners

ISC Evaluation is increasingly emphasizing models of co-development and co-creation with Indigenous partners in all evaluation projects. In the short- to medium-term, this includes ways to integrate Indigenous evaluation expertise, knowledge, world views and/or Indigenous capacity development at key points in evaluations (planning, methodology, data collections, and development of findings and recommendations). In the long-term, the ultimate goal is to support the establishment of Indigenous evaluation functions outside of government.

In general, the evaluation innovation projects have the following objectives:

• To support Indigenous evaluators and experts, research institutes, think tanks, and/or organizations to advance the implementation of ISC program evaluations with a focus on service transfer.
• To support Indigenous organizations to strengthen their capacity to implement program evaluations, enhance and coordinate Indigenous policy and evaluation networks, and organize the Indigenous policy and evaluation field (e.g., conferences, a center of excellence, etc.)
• To partner with Indigenous evaluators, researchers, think tanks, and/or organizations to advance evaluations tools, guides, policies, strategies and other evaluative practices that integrate Indigenous knowledge and worldviews.

In this context, ISC Evaluation continues to support two multi-year evaluation innovation projects with Indigenous partners.

Clustering Evaluations

ISC Evaluation sees clustering evaluations as an effective method to determine how programs are working together and what the overall impact of multiple programs is within communities. The cluster approach means the implicated programs are evaluated as a system rather than at the detailed and programmatic level which has been past practice. Evaluating programs in a cluster provides a more holistic view of the interrelationships between and among programs and the broader impacts of the clustered programs on communities. Additionally, breaking down silos in evaluation work has been an ongoing and important theme with Indigenous partners. New and emerging Indigenous-led evaluation frameworks emphasize holistic evaluation approaches as an important way to uncover systemic barriers and to have greater impacts on improving overall community well-being.

The clustering approach also aligns more closely with the structure of the recently updated 2023–24 Departmental (ISC) Results Framework (DRF) that supports a single 'Core Responsibility' focusing "Indigenous Well-being and Self-determination". The new structure shifts the organization of the department's programs from service delivery types into those that support outcomes for Indigenous Peoples and communities—aligning with the department's mandate and vision to support and empower Indigenous Peoples to independently deliver services and address socio-economic conditions in their communities.

The implementation of the new Departmental Results Framework (DRF) will demonstrate interdependencies across Service Areas that influence socio-economic outcomes (e.g., the social determinants of health). It will provide better alignment for programs that share common objectives, outputs, and recipients (e.g., home and long-term care, infrastructure programs, etc.), improved performance measurement, and a coordinated approach to departmental engagement by Service Area. It will also simplify engagement with the goal of facilitating greater co-development and direction from partners. Service Areas are emphasized as the priority to ensure that service transfer can be achieved through the many forms it may take both at the community and regional, and at times, national level. For the 2023–2024 year, priorities are linked to six Service Areas: Health, Children and Families, Education, Infrastructure and Environments, Economic Development, and Governance.

In general, the conduct of cluster evaluations requires unique yet mixed sets of skills, including expertise in systems and complexity theory, evaluation theory, Indigenous cultural competencies, and to some extent, expertise in the program itself and in other administrative/ organizational system-level domains (e.g., HR, Financial, data, etc.). The challenge is that, in many cases, the department's programs were not designed or implemented in a systemic way. Due to this complexity, additional effort is required in the planning and co-development phase. While cluster approaches offer some economies such as common frameworks, a generic theory of change, and the development of common interview guides and data collection tools, this approach does require more time to do as well as substantial financial and human resources.

ISC Evaluation is confident that clustering evaluations is an important approach going forward as it is better aligned with Indigenous partners' feedback on evaluation approaches as well as the closer alignment with the new departmental DRF. It is recognized that integrating this new approach will take a couple of years as methodologies are refined and as the planning process evolves year over year, but clustering will be increasingly integrated into the ISC Evaluation practice.

Finally, while cluster evaluations focus on a system, not on individual programs, they still provide a "light touch" evaluation on each program in the cluster and provide high level details on individual program relevance and performance. It is important to also note that this approach does not preclude the conduct of individual program evaluations at the sub- and sub-sub levels as may be determined by the priorities and needs of the department and partners. It is expected that many sub-programs will continue to be individually evaluated.

Current examples:

The Cluster of Health Infrastructure Service for First Nations and Inuit
ISC Evaluation initiated a new method to evaluate six of ISC's health programs by clustering the programs into a single evaluation in order to take a holistic view of how the multiple programs work in concert to improve health infrastructure for First Nations and Inuit communities. The Cluster Evaluation of Health Infrastructure Support for First Nations and Inuit includes: Health Planning and Quality Management; Health Services Integration Fund; Quality Improvement and Accreditation Program; Health Facilities; e-Health Infostructure; and Health Human Resources. British Columbia Tripartite Relations have been used to inform best practices and lessons learned in the cluster evaluation. The evaluation took a broad strategic approach, and sought to assess whether the mix of "health system infrastructure programs" meets the current needs of Indigenous communities, supports the Canadian Patient Safety Framework, and aligns with the Department's mandate of transformation in the short term and the transfer of programs to Indigenous Peoples in the long term. The final report for this evaluation is anticipated by 2024.

Grouping of Infrastructure Evaluations
While not being identified as a formal clustering approach, ISC Evaluation applied an integrated approach in the conduct of three concurrent infrastructure-related evaluations, i.e., Evaluation of the On-Reserve Housing Program, Evaluation of the Education Facilities Program, and Evaluation of the Other Community Infrastructure and Activities Program. The design of the methodologies and development of the evaluation instruments took into consideration the similarities and linkages among the three programs. The data collection activities of the three evaluations were coordinated and integrated to the extent possible to reduce engagement fatigue among First Nations communities. As a result of the integrated approach, ISC Evaluation was able to identify crosscutting themes across the evaluations and provide a more holistic and systemic view of the needs, challenges and solutions in relation to the three infrastructure programs.

Crosscutting themes of enquiry in strategic/priority areas

The Government of Canada's commitment to advance reconciliation with Indigenous Peoples underpins all of ISC Evaluation's work and activities as reflected in the ISC Evaluation's vison and key objectives. However, there are a number of other lenses which are added to evaluation frameworks that are also important to inform and influence current and future service delivery and the gradual transfer of services to Indigenous partners. These other cross-cutting themes include: Service Transfer; Gender-Based Analysis (GBA) Plus; Indigenous Children and Families; Impacts of COVID–19; and Climate Change.

Service Transfer: One of the primary goals of ISC is to support Indigenous Peoples in assuming control of the delivery of services. ISC Evaluation has an important role to play in terms of informing program change and re-design that supports the transfer of control. As such, ISC evaluations are including standard lines of enquiry in all program evaluations that assess progress towards service transfer; barriers and opportunities to service transfer; and perspectives on service transfer readiness.

GBA Plus: GBA Plus provides a rigorous method for developing, implementing, monitoring and evaluating initiatives to provide equal outcomes, with an emphasis on inclusiveness and reaching those marginalized due to a combination of social norms and attitudes; institutional power structures; policies and practices that together can limit access to opportunities.

GBA Plus is an intersectional analysis – a framework for understanding how aspects of a person's identity combine to create different modes of discrimination and privilege. In doing GBA Plus, many factors should be considered. The needs and experiences of different groups of people are influenced by intersecting parts of their identity, the context they are in and their lived experiences. All federal government organizations are directed to apply GBA Plus as part of all decision-making processes.

ISC Evaluators are actively applying the GBA Plus approach in their work, weaving its processes into evaluations to understand how diverse groups of people may be affected differently by ISC policies and programs. To the extent possible, ISC evaluators are applying an intersectional, distinctions-based lens to the whole evaluation cycle, starting at the design phase, through data collection and analyses, and in the generation of findings and recommendations. For example, in the case of the Indigenous Entrepreneurship and Business Development program evaluation, a distinctions-based lens was applied to the designing of the evaluation Technical Advisory Committee to ensure that representation on the committee reflects the diverse knowledge and experience of partners involved in the program. Indigenous perspectives in research and evaluation typically involve an acknowledgment of intersectionality and differential impacts, and further communications between the Government of Canada and Indigenous partners and communities could illuminate new ways to work these perspectives into departmental GBA Plus processes.

Indigenous Children and Families: Most programming at ISC touch the well-being of Indigenous children and families either directly or indirectly. In the effort to emphasize and adopt more child- and family-centered evaluation approaches, lines of enquiry on the impacts of the programming on Indigenous children and families are systematically added to all ISC program evaluations.

Impacts of COVID–19: The consequences of the worldwide COVID–19 pandemic had significant impacts on the delivery of ISC programming as travel restrictions were put in place and many Indigenous communities declared a state of emergency, allowing only limited access for essential functions and remaining closed to outside visitors for several months. In addition, regional and community partners faced enormous workload challenges due to the pandemic and the multiple effects across Canada. These challenges persisted well into 2022. ISC Evaluation is looking at preliminary impacts in each evaluation as a discrete line of enquiry, to investigate lessons learned on programming activities.

Climate Change: It is well understood that Canada's changing climate exacerbates existing challenges and health stressors for Indigenous Peoples in Canada, including wildfires, permafrost thaw, changing wildlife patterns, diminishing access to traditional food sources, and flooding. ISC Evaluation seeks to better understand how Inuit, First Nations and Métis people are experiencing climate change by collecting preliminary and/or cursory information on climate change impacts in Indigenous communities as a cross-cutting theme in all of its evaluations.

Independent Third-Party Evaluation for ISC Reform and Child and Family Services Suite of Evaluations

Evaluation work related to child and family welfare has expanded substantially and has become increasingly elaborate in response to the Indigenous child and family services reform agenda, including the Canadian Human Rights Tribunal (CHRT) order of January 26, 2016, that found Canada had engaged in discriminatory conduct in relation to the First Nations Child and Family Services program. As a result, Canada was ordered to cease its discriminatory practices and reform the Program, and also to cease applying a narrow definition of Jordan's Principle and to take measures to immediately implement its full meaning and scope.

In December of 2021, Canada agreed with the other parties to the CHRT matter to an Agreement in Principle on long-term reform and later obtained a corresponding order, 2022CHRT8, on the consent of Canada and the Parties. Included in the 2022CHRT8 order was a requirement related to the reform of ISC that Canada work with the Parties to establish an Expert Advisory Committee (EAC) within 60 days of the order to develop and oversee the implementation of an evidence-informed work plan to prevent the recurrence of discrimination.

Following this order, Canada, the First Nations Child and Family Caring Society and the Assembly of First Nations worked together to identify experts from across the country to form the EAC. It is comprised of academics and specialists, with lived experience and expertise in First Nations child and family services; Indigenous health, mental health and addiction; Government of Canada expertise; Indigenous law; Indigenous culture and heritage; Indigenous history in Canada; participatory and culturally sensitive evaluation; and Indigenous research and ways of knowing. The Committee is advising on the design and implementation of an independent third-party evaluation to identify and provide recommendations to redress internal departmental processes, procedures and practices that contribute to the discrimination identified by the CHRT. These measures will be complemented by mandatory staff training and revisions in performance metrics for staff that affirm non-discrimination.

In addition, and over the short and medium term (next five years), ISC Evaluation will complete a suite of large and complex evaluation projects, some of which are in progress: a literature review and synthesis report informing the third-party evaluation for ISC Reform; an evaluation framework for the independent third-party evaluation for ISC Reform; a synthesis report of the substantive reports and studies conducted on the First Nations Child and Family Services Program since 2015; and, evaluations of the other Child and Family Services-related programs and initiatives.

Ongoing engagement and Indigenous evaluation capacity

Engagement approaches can be expected to vary by evaluation project as some programs have well-established engagement protocols with national Indigenous organizations, regional Indigenous organizations, Indigenous technical/manager working groups, and communities while others do not. ISC Evaluation works with individual programs to determine best approaches for engagement while at the same time is exploring opportunities with Indigenous experts in the evaluation community and/or the private, academic and not-for-profit sectors. ISC Evaluation is also exploring the adoption of distinctions-based evaluation approaches and discussions with national Indigenous organizations are ongoing.

In general, ISC Evaluation has a number of engagement practices that enable and advance Indigenous evaluation involvement:

• Establishing working groups or advisory committees that include internal and external subject matter experts and community members. The main purpose of these working groups is to ensure evaluations are undertaken in a culturally responsive and appropriate manner, and they produce meaningful results that are oriented on Indigenous worldviews and perspectives;
• Travelling to Indigenous communities for engagement and data collection rooted in ceremony, respect, and reciprocity, as defined by each participating community;
• Hiring Indigenous-led evaluation consulting firms;
• Working with partners to achieve high Indigenous/community representation in data collection activities (focus groups, surveys, informant interviews) while taking into consideration Indigenous principles of data sovereignty;
• Seeking guidance from Indigenous Elders and Knowledge Keepers throughout the evaluation journey, where possible; and
• Bringing Indigenous partners to co-present to Senior Management at critical stages of the evaluation.

As an example, an Education Facilities Evaluation Advisory Committee has been established, of which half of the participants are First Nations members and First Nations organizations external to the Department. The Committee meeting is a platform for advice from the perspectives of First Nations stakeholders at major evaluation milestones. The process continues to promote exposure of external stakeholders to the evaluation function and develop relationships amongst ISC Evaluation and various communities and organizations – even beyond the committee itself. While ongoing, the Committee is expected to greatly contribute to the success of recruiting participants for the evaluation, the cultural appropriateness of the findings, and the relevance of the products to diverse stakeholders. Equally, lessons learned from the complex process of member selection can be used to inform future advisory committees.

Corporate advice and management practice

ISC Evaluation provides ongoing advice to support the development of Treasury Board Submissions and Memoranda to Cabinet to ensure evaluation evidence is represented and used appropriately, that plans for future evaluations are properly referenced, and to provide high level commentary and advice on the Department's Performance Information Profiles. The Directorate also provides annual input into the Departmental Plan, the Departmental Results Report, as well as to the corporate risk and business planning processes. ISC Evaluation participates in the regular Neutral Assessment exercise and the annual TBS Capacity Survey Report, and also ensures the quality of published documents.

ISC Evaluation's Management Response and Action Plan (MRAP) process and tracking tools continue to evolve. The interactive database ISC Evaluation developed to store and structure detailed data for each MRAP continues to provide aggregate information on the status of MRAPs from a departmental, sector, or program level and for different aspects of the MRAP (such as recommendations, action items, and timelines). ISC Evaluation works closely with Programs to develop sound and robust MRAPs which are updated on a quarterly basis and tabled at the ISC Performance Measurement and Evaluation Committee twice annually. This database helps ISC Evaluation monitor progress made in addressing evaluation report recommendations and to ensure commitments are met and program improvements are made. However, the potential of the database goes beyond the tracking, monitoring and reporting of MRAPs. For example, from an analytical perspective, it can be mined by using "text" identification and coding methods to uncover hidden insights regarding common or systemic program performance issues and barriers from a cross-sectoral or horizontal perspective. The MRAP tracker and process has become a foundational tool for the practice management function which also supports the TBS Capacity Survey and recently, ISC Evaluation (Practice Management MRAP team) launched an interdepartmental outreach on best practices for monitoring the implementation and completion of MRAPs. Further, the Department's Results Delivery Unit (Risk and Management Practices) has found this tool excellent for its own tracking and reporting needs (tracking MAF, Risk mitigation action plans, Management Practices activities, etc.) within the Unit. With the other government evaluation functions indicating that it was a best practice across the federal evaluation community, this has led to the creation of an interdepartmental Evaluation MRAP working group that meets on an ad-hoc basis.

In terms of publishing evaluation reports, the Practice Management team of the Evaluation Directorate has improved the approval and posting process significantly in recent years and has dedicated resources to ensure timely and high-quality publications. For example, a style guide has been developed for ISC Evaluation to inform evaluators early in the writing process of the requirements for documents intended for publication on the departmental website. An evaluation report template was developed to create efficiencies prior to even the document approval stage.

Finally, the approval processes are quite efficient as the Evaluation function benefits from the role that the Senior Management Committee (SMC) plays as the Department's Performance Measurement and Evaluation Committee (PMEC). The SMC meets frequently and can approve evaluation reports as they are completed as opposed to waiting for periodic and dedicated PMEC meetings. ISC Evaluation participates at the SMC at least once a month.

Impact of COVID–19 Pandemic

The consequences of the worldwide COVID–19 pandemic continue to resonate for evaluation planning purposes. In early 2020–2021, evaluation activities were curtailed as travel restrictions were put in place and many Indigenous communities declared a state of emergency, allowing only limited access for essential functions and remaining closed to outside visitors for several months. In addition, regional and community partners faced enormous workload challenges due to the pandemic and the multiple effects it is having across Canada. These challenges persisted into 2021–2022 and further into 2022–2023. For example, the Directorate's ability to advance the evaluation of the Communicable Disease Control and Management program has faced significant challenges. However, with restrictions lifted, many evaluations are back on normal schedules. However, there have been substantial human resource effects post-pandemic, including external and internal staff turnover and burnout.

Human resources

The dedicated ISC Evaluation Directorate was established in July 2019 with an allocation of 16 full-time equivalents (FTEs) with, at the time, only half of these positions permanently staffed. Evaluation planning exercises have consistently noted the risks to delivering on the Plan because the FTE complement has not been sufficient given the complexity and volume of the work involved. Beyond the 53 planned evaluations, there are also significant requirements and pressures from a practice management perspective (such as tracking and monitoring of Management Response Action Plans, input into TBS Submissions and Memoranda to Cabinet, TBS planning, reporting, and publishing requirements, etc.). Furthermore, ISC Evaluation has a unique and strategic role relative to other evaluation functions across government given its objectives to support and assess 'Service Delivery Transfer' to Indigenous partners and to also advance Indigenous evaluation capacities and evaluation functions outside of government. More generally, the demand for evaluation skills across government presents an additional challenge related to recruitment and retention, most particularly lower and mid-level evaluators. Over the past year, the Directorate has focused on building its capacity and has increased its FTE allocation to 41 for 2023–2024, which can be expected to alleviate some workload pressures.

Continuous improvement of evaluation function. The Directorate is taking steps to support the continuous improvement of the function and to address some of the capacity and workload challenges it faces in order meet its three-part mandate. The focus is on how to establish a function that can act as an effective change catalyst and be a key player in driving the Department's agenda, particularly with respect to building the relationships necessary for co-development work, improving service delivery and supporting eventual service transfer. These steps are also expected to ensure departmental evaluators have opportunities to develop the knowledge and cultural competencies required for their role and also in alignment with the Policy on Results.

Performance information

The availability and quality of performance information remains the major challenge for evaluation activities. The Government of Canada introduced the Policy on Results in 2016 to instill a culture of performance measurement and evaluation, taking a results-focused approach that relies on gathering performance information. All government programs are required to develop Performance Information Profiles (PIPs) to serve as a guide to gather performance information. However, collecting data at the front-line delivery level remains a challenge for program managers given the resources required for setting up databases that can be used to gather data and then create reports. Much of the performance information resides with First Nations communities, with little information available at the regional level to inform evaluations. Accessing data can often depend on the level of community engagement that has occurred.

The Evaluation Team faced challenges collecting internal data, which was not always aggregated or standardized. This is in line with what has been flagged as an issue in ISC's Management Accountability Framework