Library and Archives Canada
Symbol of the Government of Canada

Institutional links

ARCHIVED - About Us

Archived Content

This archived Web page remains online for reference, research or recordkeeping purposes. This page will not be altered or updated. Web pages that are archived on the Internet are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats of this page on the Contact Us page.

Audits and Evaluations

Preliminary Survey of
A System Under Development
Audit of The AMICAN Catalytic Initiative

Final version approved by the Audit And Evaluation Committee
June 2007

Table of Contents


Introduction

Background

Library and Archives Canada (LAC) comprises the formerly separate institutions of the National Library of Canada and the National Archives of Canada. The overarching focus of the new institution is on stewardship and promotion of the documentary heritage of Canada in all its mediums. A Transformation Team was created in 2003 with the mandate to oversee and guide the merger process of the former institutions. It was also given the responsibility to develop a new organizational structure and vision for LAC. A number of workgroups were created to assist the team in that task.

The need to develop a system that would bring together the information in the existing databases to create an integrated, seamless, customizable public interface, was expressed repeatedly by the various transformation working groups in the both spring and fall of 2003. As a result, a project was initiated to develop AMICAN, a system to support the intellectual and physical management of and access to LAC's holdings built upon the current and emerging IT architecture. This project will also act as a catalyst in the transformation and implementation of key business processes, the information architecture as well as the overall IT architecture and in overall service delivery to LAC clients.

The original estimated cost of the AMICAN project was approximately $4.5M with the project estimated to run from September 2004 to September 2007. The Digital Preservation Infrastructure (DPI) Project arose, as did AMICAN, out of the Transformation initiative. That project is now called the Trusted Digital Repository Project or TDR. Resulting from the intricate interrelationship between these two projects, a common governance structure has been adopted and a single submission is being developed for Treasury Board approval. In this submission the estimated costs for the recommended option 3 amount to $22.8M from external sources plus $6.9M from a LAC contribution, over four years ending in 2010. Thus, AMICAN / TDR is a major, multi-year investment requiring political astuteness, careful planning, solid funding and extensive staff involvement.

The AMICAN / TDR Initiative is a large scale IT project and as such is high risk. Many Office of the Auditor General of Canada (OAG), Treasury Board Secretariat (TBS) and other internal audits and studies have been done that show the success rate of such projects is very low. Success is defined as the project being completed on time, within budget and meeting the original user requirements. Because the AMICAN / TDR Project is at various stages of development and implementation, it was determined a preliminary survey should be done to determine current risks and probability of project success.

Objective

The objective of this audit was to carry out a Preliminary Survey of a System Under Development (SUD) Audit of the AMICAN Catalytic Initiative at Libraries and Archives Canada. Emphasis was placed on risks, possible mitigation strategies and probability of success of the overall AMICAN project.

Scope

The RFP for this audit assignment indicated that the scope would include "all aspects of the AMICAN Catalytic Initiative including the planning processes; the project organization; project control mechanisms including committees, risk management processes, reporting mechanisms; financial controls, security and reporting." For the purposes of the audit, the scope of the AMICAN Catalytic Initiative was to include a list of sub-projects that relate directly to what is now called the AMICAN project cluster.

With the growing interdependencies between the AMICAN and the TDR initiatives, it would be very difficult to completely extricate AMICAN as would seem to be the initial scope of this audit. Accordingly, the audit scope was amended to include an exploration of the TDR governance structure, basic project processes and risks.

Approach and Methodology

The methodology used in individual project audits addresses the COBIT processes, but allows for the normally unique nature of each project. The SUD risk methodology is modeled after the normal set of accountabilities involved in developing software and managing systems projects; the following five domains are involved: Governance Risk, Business Risk, Project Risk, Testing Risk, and Technology/Infrastructure Risk.

The Software Engineering Institute (Carnegie/Mellon University) has published a series of critical risk measurements for both technology projects and technology management organizations. The Capability Maturity Model (CMM) outlines the necessary or strongly desirable elements of IM/IT management required for achieving higher levels of acknowledged maturity. COBIT contains these measures. All these elements have been combined into this audit approach to IM/IT management and SUD success and risk measurement. The assessment structure is further described in Appendix B.

Finding and Conclusions

General Conclusions

Overall, the AMICAN / TDR Project now enjoys a solid governance structure, appropriate risk management and control frameworks, and a consistent approach to project management. Nonetheless, the project represents a very significant change in the business and as such also represents a significant risk. Given the nature of the business evolution and the integration of two previously separate institutions, non-delivery of project objectives is the primary concern.

The main source of risk on the AMICAN / TDR Project is the lack of a consistent source of funding; the results are delays in project delivery, higher costs than necessary, variations, and perhaps inadequacies, in project staffing, an over-reliance on contractors, an inadequate infrastructure and the lack of solid maintenance support. Though this may be stating the obvious, the funding issue is the most important issue that must be addressed.

A good business practice that would address a number of potential risks is the development of Service Level Agreements between the service provider (Information Technology Branch - ITB) and the service recipients (the business units). An SLA should be prepared specifying the capacity, reliability and availability and service recovery targets required for system modules about to go into production. At present, there is confusion about requirements for reliability and availability of the AMICAN modules when they go into production. Issues related to business continuity planning have not been addressed. Capacity, reliability and availability targets (especially for infrastructure) are unknown. Further, without the detail normally part of SLAs, it is difficult to determine how successful the new system has been after implementation. As a result, it would be difficult to determine how benefits are achieved.

The current state of contingency planning in LAC is not clearly understood by all concerned stakeholders. Whereas business unit management felt a disaster recovery site was planned, ITB management confirmed that no such site currently exists. At one time there was an insurance plan for infrastructure recovery in case of disaster. That insurance was cancelled by PWGSC some years ago since systems at LAC were deemed to be non-mission critical by Treasury Board. It is critical that management address risks related to lack of backup and recovery. Specifically LAC should develop recovery plans covering all critical operations and services, and ensure that all stakeholders are aware of the target time for restoration of services.

Summary of Findings

The model used to assess risk is further described in Appendix B to this report. As stated earlier, the work was organized into five areas: governance, business, project, testing, and technology. This report adheres to this structure. Governance and technology are shared by both clusters, whereas business, project and testing are unique to each.

  • AMICAN / TDR Governance Risks

    The overall AMICAN / TDR governance structure and process are well organized and in line with recommended best practices. Several committees are defined and operational, although some are recent additions. How the committees operate and affect the scope and direction of the AMICAN / TDR Project is not yet fully worked out. In addition, the interrelationship of these committees needs to be refined. Scope management is operating, but senior management are concerned that short-term decisions are made without taking long-term objectives into consideration.

    The major risk, funding, has not been adequately addressed. The sporadic nature of funding has been the single most frequent cause of problems in this project. The project needs stable funding and a strategic funding plan over multiple years, whether the TB submission comes through or not. Specifically, LAC should ensure stable funding (whether through a successful TB submission or internally) to allow consultants to be brought on board for the full duration of the time they are required.

    Full delivery of the AMICAN / TDR set of objectives is necessary to meet present and future business needs. Nonetheless, tracking of benefits achieved from such implementation is needed for management to determine how well the delivery addresses individual business requirements. There are no preparations in place to measure benefits achievement. There are no studies or reports that describe detailed pre-implementation costs by process for comparison with the post- implementation costs. Neither the PRC nor the SC has discussed the question of how the achievement of benefits will be measured, although this is apparently the mandate of the SC.

  • AMICAN Risks

    Overall the potential risks to the AMICAN cluster of sub-projects have been well managed. Business requirements are well known and the solution design well understood. The majority of risks relate to staffing and the impact of sporadic funding: insufficient resources, an over-reliance on contractors, and difficulties in procurement of contract resources. The project does not appear to be planning for adequate maintenance support. Although risks are managed, more discussion at the PRC level would further enhance overall risk management.

    The Quality Assurance environment, a component of the development and delivery of software and hardware products, has not been set up risk-free. There are issues with the lack of space to use a complete copy of the production database and the absence of Oracle RAC - a technical product that will be used in production. The Oracle RAC issue has been mitigated somewhat by the recent decision to implement AMICAN first in a production environment without Oracle RAC, and then implement it with Oracle RAC later on.

    There is confusion about targets for reliability and availability of the AMICAN modules when they go into production. Issues related to business continuity planning have not been addressed. Capacity, reliability and availability targets (especially for infrastructure) are unknown. There are no service level agreements and senior management's expectations have not been clarified.

  • TDR Risks

    Overall, the TDR cluster of sub-projects is younger and represents generally higher risk than AMICAN. The TDR Cluster is broken into 4 sub-projects, and the approach used to specify the business requirements was varied because of the nature of the individual sub-projects. Overall, TDR is still at the very early stage of determining what is needed and proposing solutions. Management is making a good assessment of the challenges. However in reality we see that there is a gap between the ongoing development and the business requirements readiness. There is a significant risk that the product will not adequately address a number of business requirements.

    A contract is in place for some of the Virtual Loading Dock component and is scheduled for delivery in September by the Integrator. Although the road map is showing a "phased in" approach, we have noticed that the development work is moving faster than the analysis and specification work because of the contractual arrangements and budget availability.

    Business is changing and technology must become capable of handling digital collections as well as paper ones. Senior management thinks that change in technology does not represent a significant risk and that people will adapt to these changes. However, Business Managers feel that the project is moving towards a target that is not well understood and that more research is needed.

    A project plan and approach was prepared jointly with the integrator. The plan is such at a high level (no detail deliverable or task list) that the integrator could legally deliver something that would not meet LAC expectations. This potentially allows unexpected or unsatisfactory deliverables. Although a good track record has been observed up to now, this creates a situation where LAC does not have control over how much functionality it is going to get and opens the door to potential risks.

    LAC is partnering with Deloitte and PWGSC for the delivery of TDR. Although this arrangement has definite benefits - such as the provision for a larger scale of expertise for system development - it also has potential risks. Overall the tools, the documentation and the practices deployed by the integrator look good. However, the delivery of an acceptable solution by the supplier could represent a risk of dispute as outlined above.

  • AMICAN / TDR Technology Risks

    The infrastructure that supports AMICAN is, by general consensus, the weakest link in the project - hardware platforms for AMICAN are aging, almost obsolete. In addition, new technology is required to meet TDR objectives. The weaknesses are in two areas - the hardware platforms, and the support personnel. The scalable architecture design means that additional capacity can be added relatively quickly and easily; the issue is not a technical one but a financial one. Infrastructure upgrade plans depend on obtaining funds via the upcoming TB submission. If TB funds are forthcoming by April 1 2007, adequate investment in infrastructure must be made as a priority. If TB funds are not forthcoming by April 1 then the department will need to adjust the scope of the AMICAN project to what can be funded internally. However management focus on infrastructure should remain a priority.

    Transition risk is quite often under-rated. At the present time, this risk is being well managed / planned by IT management. Nonetheless, more attention to planning is needed, mainly in the area of infrastructure. Production procedures are in place but perhaps are not well known to all the project leaders concerned. There is no configuration management (CM) for infrastructure; lack of CM when the new modules are in production is risky.

    The Informatics Infrastructure Management Directorate (IIM) in the Information Technology Branch has the role of managing the technology infrastructure for LAC. However, there are insufficient personnel resources in IIM to support the current infrastructure and operations, and to support the impact of projects on that infrastructure at the same time - insufficient depth in numbers and in expertise. Should the investment arrive, LAC will get more resources and be better able to plan all elements of the infrastructure replacement including long terms needs. However, even with the investment, insufficient staff to support the upgrade can have detrimental effects. There is a risk of ongoing system support failing during the effort to provide for the new AMICAN / TDR suite of products and tools.