Library and Archives Canada
Symbol of the Government of Canada

Institutional links

ARCHIVED - About Us

Archived Content

This archived Web page remains online for reference, research or recordkeeping purposes. This page will not be altered or updated. Web pages that are archived on the Internet are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats of this page on the Contact Us page.

Audits and Evaluations

Evaluation Framework for Modernization

Previous | Table of Contents

4.0 Monitoring and Evaluation Plan

4.1 Performance measurement plan

Strategy and Key issues

The table below identifies key outputs and outcomes along with their associated indicators, sources and measurement frequency. The performance indicators will enable management to assess progress and performance in achieving their goals. The indicators presented in this section of the report were developed through discussions with key informants. They are linked with specific outputs, immediate outcomes and intermediate outcomes of the logic model.

Table 3: Performance Indicators for Modernization Initiative
Output/
Outcome
Indicator Data Source &
Method
Responsible Frequency
1. Plans and strategies Number of policy suites developed (Acquisition, Collections Management, and Resource Discovery) - Administrative data Sub-Committee Leads Quarterly
2. Work processes redefined Extent to which work processes at LAC have incorporated the principles of Modernization - Document review
- Key informant interviews
DG Steering Committee Semi-Annually
3. Horizontal collaboration Extent to which policies and processes are in place to support internal and external collaboration - Document review
- Key informant interviews
DG Steering Committee Semi-Annually
4. Employee engagement Extent to which employees actively support Modernization activities - Employee survey DG Steering Committee Semi-Annually
5. Policy-driven organization Degree of coherence between policy decisions taken and their implementation - Document review
- Key informant interviews
DG Steering Committee Annually
6. Professional bureaucracy Degree to which on-going monitoring and lessons learned are incorporated into subsequent plans and actions - Document review
- Key informant interviews
DG Steering Committee Annually

The Head of Evaluation will present an update to the Evaluation Committee on the progress of the implementation of the framework every six months once the framework is approved.

Reporting Strategy

Progress on the Modernization Initiative will be reported through the following mechanisms/documents:

  • Regular updates to Management Board by the DG Steering Committee on Modernization
  • A Departmental Performance Report (DPR) to be presented by LAC each year to the Treasury Board Secretariat (TBS); and
  • Program Evaluation reports and updates to be presented to the Library and Archives Evaluation Committee for approval and subsequent release to the public.

In addition, on-going reporting of Modernization activities will be carried out using various mechanisms such as the LAC website, the LAC intranet and messages from the Librarian and Archivist of Canada.

4.2 Evaluation Plan

The five-year evaluation cycle for the Modernization Initiative spans the fiscal years 2010-2011 to 2015-2016. A formative evaluation will be conducted in 2012-2013 to assess the performance measurement strategy, identify any issues that may arise, and assess data availability for the summative evaluation to follow in 2015-2016.

The summative evaluation will be conducted to assess overall progress in reaching the goals and objectives for the initiative. This evaluation will fully incorporate findings from ongoing performance measurement and will follow the Evaluation Standards presented in the Treasury Board Evaluation Standards in the Government of Canada (TBS, 2004). There are three primary areas that will be considered, namely relevance, success and cost-effectiveness.

Key Evaluation Issues and Questions

Relevance - does the initiative continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need?

To what extent does the Modernization Initiative continue to align with the priorities of Library and Archives Canada and the Government of Canada?

Do activities being undertaken under the Modernization Initiative continue to be relevant to the public? Do these activities address the evolving needs of stakeholders and clients?

Success - is the initiative effective in meeting its objectives, within budget and without unwanted outcomes?

Is the Modernization Initiative being implemented as expected?

To what degree have the expected immediate and intermediate outcomes of the Modernization Initiative been achieved?

Is the performance measurement strategy being carried out as planned?

How is the performance data that is collected being used?

Cost-effectiveness - Are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?

Have more effective / efficient ways of achieving the expected outcomes of the Modernization Initiative become apparent?

Evaluation Timeframes

The timeframes set out below are normal for most program evaluation cycles. The formative evaluation of the Modernization Initiative would be conducted in the fiscal year 2012-2013; the summative evaluation would occur in the fiscal year 2015-2016. Table 4 below summarizes the time frame applicable to evaluation activities.

Table 4: Evaluation activities timeframes
2010-2011 2011-2012 2012-2013 2014-2015 2014-2015 2015-2016
Develop Framework          
    Formative evaluation      
          Summative evaluation

Formative and Summative Evaluations

Table 5 below contains the evaluation framework that will guide the formative and summative evaluations for the Modernization Initiative. The framework addresses the three primary areas of relevance, success and cost-effectiveness, and contains a total of seven questions that will help to guide and focus the evaluation. For each evaluation question, the framework describes the indicators and data sources that will be used to address the question. All of the issues/questions will be addressed in both the formative and summative evaluations, however, the focus of the formative evaluation will be to assess if progress is being made in each of the respective areas.

Table 5: Evaluation Framework
Issues/Questions Indicators Data Analysis Methods Evaluation Activity
Formative Summative
Relevance
1. To what extent does the Modernization Initiative continue to align with the priorities of Library and Archives Canada and the Government of Canada? - Degree of alignment between Modernization activities and priorities of LAC and the Government of Canada - Document review
- Key informant interviews with LAC staff and key stakeholders
X X
2. Do activities being undertaken under the Modernization Initiative continue to be relevant to the public? Do these activities address the evolving needs of stakeholders and clients? - Evidence of public and stakeholder understanding of and acceptance of the need for Modernization - Document review
- Key informant interviews with LAC staff and key stakeholders and clients
X X
Success
3. Is the Modernization Initiative being implemented as expected? - Extent to which Modernization activities are implemented as planned - Document review
- Key informant interviews with LAC staff and key stakeholders
X X
4. To what degree have the expected immediate and intermediate outcomes of the Modernization Initiative been achieved? - Extent to which employees actively support Modernization activities
- Extent to which work processes at LAC have incorporated the principles of Modernization
- Degree of coherence between policy decisions taken and their implementation
- Degree to which on-going monitoring and lessons learned are incorporated into subsequent plans and actions
- Employee survey
- Document review
- Key informant interviews with LAC staff and key stakeholders
X X
5. Is the performance measurement strategy being carried out as planned? - Degree of alignment of performance measurement plan with actual performance activities undertaken - Document review
- Key informant interviews with LAC staff
X X
6. How is the performance data that is collected being used? - Evidence of use of performance data in decision making - Document review
- Key informant interviews with LAC staff
X X
Cost-effectiveness
7. Have more effective / efficient ways of achieving the expected outcomes of the Modernization Initiative become apparent? - Evidence of more effective / efficient ways of achieving the expected outcomes are apparent to staff and key stakeholders - Document review
- Key informant interviews with LAC staff
X X

Evaluation costs

The estimated costs for the formative and summative evaluations are provided in the table below. These are normal costs for contracted services.

Table 6: Evaluation costs
Study Fiscal Year Total O&M costs
Formative Evaluation 2012-2013 $25,000
Summative Evaluation 2015-2016 $75,000
Total $100,000

Previous | Table of Contents