ARCHIVED - About Us

Audit of The AMICAN, 2007

Appendix A

AMICAN / TDR Risk Matrix

Table of Contents

Introduction

• 1. Governance Risk
  • 1.1 Senior Management Control Framework
  • 1.2 Change / Scope Management
  • 1.3 Investment / Benefits Management
• 2. AMICAN Risk
  • 2.1 AMICAN Business Risk
  • 2.2 AMICAN Project Risk
  • 2.3 AMICAN Testing Risk
• 3. TDR Risk
  • 3.1 TDR Business Risk
  • 3.2 TDR Project Risk
• 4. Technology Risk
  • 4.1 Infrastructure
  • 4.2 Technology Transition

Introduction

The detailed results of the AMICAN / TDR Preliminary Survey are presented in the format of a risk matrix. These results are a snapshot as of March 30, 2007. The review took place from January 29 to March 16 and considers that the situation further described in the matrix will continue to evolve rapidly.

The purpose of the matrix is to identify factors that could affect the successful delivery of AMICAN / TDR business solutions, on time, within budget, and that meet user requirements. The Risk Framework used to conduct the review is composed of five risk categories: Governance Risk; Business Risk; Project Risk; Testing Risk and Technology Risk. Each category of risk has been organized into three columns describing: the related project activity; the review conclusions and assessment of risk, and either an indication that no recommendations are required, or suggestions and recommendations for those areas where improvement would reduce the risk to an acceptable level.

The risk assessment describes the nature of the threat, the likelihood of it occurring, and the consequences of that threat. The risk assessment is based on the experience of the reviewer. The difference between projects fostered by unique business problems for which a solution is being sought, different business and project organizational approaches, personal project management styles, and different project control and system development methodologies, works against the applicability of a statistically derived assessment methodology. In this review, the following definitions are used:

Likelihood of Risk Materializing

Low

Reduction of the inherent risks either through a combination of significant project management measures or environmental influences. For example, Project Risk is reduced through the combined application of a recognized methodology, a quality assurance regime, and an experienced team familiar with the business and technology. Unlikely to occur.

Medium

Given that inherent risk is present, the project is taking adequate precautions to manage that risk and no unusual environmental influences are present. Would occur under the right circumstances.

High

Compounding the inherent risk with either the absence of adequate precautions or the presence of unusual environmental influences. For example, Business Risk would be greater due to the absence of a documented comprehensive control framework, in the case of a mission critical financial system. Technical Risk would increase with the use of technology with which the organization is not familiar with, or dependence on unreleased versions of vendor software for a promised infrastructure. Is occurring or very likely to occur.

Impact if Risk Materializes

Low

General compliance with accepted control principles, but with issues that can be resolved internally.

Medium

Partial compliance with accepted control principles which could result in management weakness but unlikely to seriously damage project.

High

Non-compliance with accepted control principles which could result in damage to the project, such as non-delivery, late delivery, unacceptable cost increases, etc.

Within each risk category and sub-category, there are numerous individual measures that are recorded in the following matrix. As some measures are not unique to an individual category, some apparent duplication is intentionally included for readability.

---

1. Governance Risk

1.1 Senior Management Control Framework

Activity

The governance structure as described in the project charter consists of a Steering Committee (senior management representatives from across the organization), a Project Steering Committee (operational management representation and chaired by SC members), and a set of project and business leads for each sub-project.

• The Steering Committee role is to set the scope of the AMICAN/TDR project and to allocate the resources.
• The Committee shifted direction in fall 2006 by integrating the TDR and AMICAN projects. The TDR governance structure initially was not as smooth as AMICAN - did not have enough governance / oversight earlier. SC members interviewed now feel more comfortable with the new management structure in place.
• Currently the information received by the committee is sufficient and accurate enough to make decisions. Progress, risk and budget issues are clear, issues and crises are handled well and fixes are found rapidly. A risk register is in place and the SC is presented with the issues and related PRC recommendations.
• Members feel consulted and have adequate opportunity to provide input, any potential issues are communicated beforehand before the PRC, lobbying and consultation are done before each meeting and thus no one is blindsided with an issue they did not anticipate.

Conclusion

The AMICAN / TDR governance structure and process overall is well organized and is in line with recommended best practices.

AMICAN and TDR both arose out of the Transition initiatives bringing the Library and the Archives together. AMICAN was first to push ahead and has developed a sound governance model. Partly as a result, TDR was moved under its umbrella despite the fact that some feel it doesn't really fit with the AMICAN mandate.

• LAC management must be aware of change to scope and mandate, and be willing to adapt to new situations as they arise. This could include scoping back on some aspects of AMICAN/TDR to ensure the project can actually achieve goals.
• The risk that the governance of AMICAN has become unwieldy as a result of the amalgamation of projects is a risk that management must be sensitive to. Ensuring the governance can play the role specified will mitigate this risk.

Risk

Impact: M
Likely: M

Recommendation:

It is recommended that LAC review and confirm the roles of the various committees.

---

Activity

The committee structure is relatively new and the role of the SC versus that of the PRCs has only recently been fully defined. As a result, some committee members felt that the roles and responsibilities were not fully understood by all.

• A review of SC Minutes reveal that they are general and high level - with no detailed review of key project management issues (no systematic cost review, no project schedule and deliverables review, the risk log not fully discussed). Thus, actual project review seems weak.
• Specifically for TDR, risks are very well documented in logs but not discussed in committee as should be. Instead, cluster manager prepares a Summary of IT and business risks that is reviewed by a TDR Management Committee organized by the CTO (this committee is in addition to the committees stated in the Charter).
• In addition, we have observed that a SC committee member chairs the PRC. This could restrain middle managers from having candid discussions on potential issues.
• There is a difference of opinion on the best level of membership for the PRC. The level is being changed to include mainly Directors rather than DGs. The concern and opposing view is that the PRC deals with too many hands-on issues.

Conclusion

The role of those committees within the charter definition, and some outside the project charter as well, seem to create confusion in the minds of stakeholders.

• For example: the existence of a TDR project review committee, a TDR Management Committee as well as TDR project team cluster committees. This could create some overlap between committees.

How the PRC reports to the SC and the level of representation on the PRC should be reviewed and refined.

• The PRC should be a working level group with the ability to fully understand the problems facing the project and the ability to make sound recommendations.
• The audit position is this is exactly where the detailed issues should be addressed.

Risk

Impact: L
Likely: M

Suggestions

It is suggested that once the roles have been confirmed, SC / PRC committee members develop guidelines for the level and extent of discussion.

• Members should be formally encouraged to report issues and state their opinions regardless of who chairs the meeting.
• It must be clear that all representatives have the right to state unpopular positions and will be supported by the committee if proved correct.

---

Activity

Although the information flow is seen as adequate for committee members, there is a risk that the information is perhaps too voluminous or not well designed for the senior management and middle management members.

• Audit review of PRC and SC reports reveals detail yet inappropriate organization that makes the detail difficult to digest.

Conclusion

The information provided to the committee should be more structured and should include a high level view of schedule, cost and functionality achievements, followed by risks and issues.

Risk

Impact: L
Likely: M

Suggestions

It is suggested that committee reports be short bullet-type reports on milestones, outcomes, schedule, key challenges, and decisions required and when necessary zoom into specific issues.

---

1.2 Change / Scope Management

Activity

Any issues affecting scope are subject to the change request process. Change requests are formalized and tracked (the change request form is documented in Project Plan and the change approval process is documented in the Project Charter). A registry of change requests is maintained.

• Scope has been more fully defined in the recent past - this has been a more recent effort for TDR. The TDR project scope is a 3-year plan to build the ingestion, preservation and access components of the TDR and will require setup of an appropriate infrastructure.
• SC Members interviewed understood the overall road map and schedule and have caused the committee to discuss the modules and the timeline. This has helped them understand the inter-dependencies.
• Changes are usually thoroughly discussed at lower levels before being presented to the PRC and / or SC. The approval is not officially signed off on the change form but the decision is noted in PRC minutes.
• Recently a major change (regarding the container issue) was dealt with at the PRC. It appears that the change process worked as planned in this case with the exception of the fact that the change request form was completed after the first discussion at the PRC instead of beforehand.

Nonetheless, the chair of the Steering Committee feels that scope management is limited in its approach. Short-term decisions are made without looking at the big picture. When a decision goes forward with a set of functionality for a specific module, the downstream effects on proposed future modules is unsure; there is no big picture view.

Conclusion

The change management approach is industry standard and appears to work well, although it has not been used very much within AMICAN. In fact there have been extremely few changes made to the project since the original requirements definition work was completed.

• The project seems to benefit from a robust management structure and qualified team of individuals. Tools are in place to manage changes the scope.
• LAC relies on a team of consultants and in-house professionals with a fixed set of deliverables set in a contract for TDR. Senior management has made a commitment to buy into the proposed scoping document presented by Deloitte and has set a guideline for the PRC to stick to the plan.

While it is difficult to actually independently evaluate if scope management is limited in its approach, there is no doubt that funding uncertainties (see following section) have resulted in short-term decision-making. What is important here is that the SC Chair is uncertain - and that the project must make sure that certainty is fully achieved - better information would be part of the solution.

Risk

Impact: M
Likely: M

Recommendation:

It is recommended that Project management address the ADM's concerns regarding sufficient regard for the larger picture when making change decisions.

---

1.3 Investment / Benefits Management

1.3.1 Funding - Cost/Benefit Decisions

Activity

All interviewees agree that project funding in the past has been dangerously sporadic; funds arrive piecemeal and unexpectedly as the department reallocates funds during the year to support AMICAN.

• From the point of view of the PRC members, IT seems to come for funding requests on a crisis basis and more time is spent dealing with budget management than on IT issues. The results of this start and stop approach are that costs have been higher than they should have been.
• Most of the project's budget goes to consulting resources for application development. Unnecessary costs are incurred either to keep consultants with LAC expertise on during periods when there may not be sufficient work for them, or in training new consultants because of turnover.
• On the infrastructure side, funding arrives in unexpected amounts often towards the end of the year. Then there is a rush to get equipment in and delays to the project are incurred during the training, installation, setup and testing of the new hardware.

Senior management has promoted the view that the TB submission will be successful. No contingency plan has been prepared. As a consequence, the project team's expectations have been built up. There will be tremendous disappointment, loss of momentum and morale if it is not.

• Cost / Benefit - The Steering Committee is happy with the process and the product presented to the board. The business case that was presented was credible and the SC was reasonably comfortable with it.

Conclusion

The major risk issue, funding, has not been adequately addressed. If the TB submission is unsuccessful, the department will most probably continue to fund the project (as stated in the business case). The risk affects not only the funding aspect but to the human resources involved.

However a specific funding plan has not been prepared and as a result there will be delays in the project, loss of expert staff and of project momentum.

Risk

Impact: M
Likely: M

Recommendation:

The sporadic nature of funding has been the single most frequent cause of problems in this project. The project needs stable funding and a strategic funding plan over multiple years, whether the TB submission comes through or not.

Specifically, it is recommended that LAC ensure stable funding (whether through a successful TB submission or internally) to allow consultants to be brought on board for the full duration of the time they are required.

---

1.3.2 Costing and Cost Tracking

Activity

There is a concern about funding and how funds are used. The relationship between the funds invested and the results to date is unclear.

• A review of all discussion re funding in all the PRC minutes from 2005 to present was carried out. In most PRCs funding was discussed and a spending plan presented. Usually these discussions covered the allocation of funds either from MB or from the ITB budget, estimates for completing certain modules were given, and schedules were adjusted according to funds allocated. However there are no reports of expenditures as compared to plan.
• The Director of Finance has recently created a new Finance governance committee which he will chair and which will oversee the financial situation of the project more closely. The mandate of this committee is to make certain that the budget is respected and is tracked, to ensure that resource requirements are properly justified, to bring independence to setting priorities for funding and to ensure that this committee is not chaired by IT. This financial committee will report to the Steering Committee.

Conclusion

The risk of poor management of costs is high but should be more closely managed with the creation of the new committee.

Risk

Impact: M
Likely: M

Suggestion:

It is suggested that ITB develop standards for cost estimating that are understandable to the non-technical person, and describe the target functionality in detail for each proposal.

---

1.3.3 Benefits Achievement

Activity

It is certainly not easy to develop a purely financial case for the AMICAN side of this project vs. the TDR side. The main driver for this project is the need to fulfill LAC's mandate of preserving Canada's heritage, specifically its digital heritage.

• The business case is mainly based on cost and risk avoidance. There are differences of opinion among the senior management of LAC about its all-or-nothing approach i.e. it presents the options as monolithic, rather than as components.

Since AMICAN has become a model of good IT project management various other projects were moved under its umbrella, despite the fact that some interviewees felt they don't really fit with the AMICAN mandate.

• As a result, the AMICAN project scope and budget have changed so much that it is impossible to track progress against original goals.
• It is hard to separate the case for each portfolio since AMICAN and TDR will be sharing the same hardware infrastructure and some software application components (such as security, virus scans)

The question of how achievement of benefits will be measured has apparently not been talked about at the PRC. It is apparently the new Steering Committee's mandate but the SC has not yet discussed it.

• Senior management mostly mentioned the benefit as increased productivity, i.e. the ability to do more with fewer staff and thus achieve cost savings. Their proposed measures are quite simple: FTEs for objects processed e.g. number of acquisitions and staff time to do an acquisition, numbers of acquisitions per year and cost per acquisition.
• Quality of service to users: whether the system has improved from the user point of view is another important measure.
• However the examples cited seem to be informal and high level measurements. There are no preparations in place to capture detailed pre-implementation costs by process for comparison.

Conclusion

As a business case strictly for AMICAN, it is a little confusing in its financial aspects, though reasonably convincing for the other non-financial rationales. For example, there is no clear explanation of what the current maintenance costs are for AMICUS; the maintenance costs of the new system are not shown anywhere. So it is difficult to support the financial cost benefit if one looks strictly at the AMICAN side.

Some AMICAN project members now feel that scope and budget have changed so much that it is much more difficult to track progress against original goals. This raises the risk that benefits cannot be determined and measured. It also raises the risk that project schedule will eventually be affected.

Risk

Impact: M
Likely: M

Recommendation:

It is recommended that LAC develop a detailed plan for measuring benefits achievement for AMICAN / TDR and begin now to track the selected measures to capture information about the pre-implementation situation.

Suggestions:

As the business case is now finalized it is impossible to change it, but it is suggested that LAC prepare and provide O&M costs if requested by TB and for use in future budget planning.

It is suggested that LAC develop guidelines as to what does and does not belong in the AMICAN mandate and apply these guidelines to any new projects that may be proposed.

---

2. AMICAN Risk

2.1 AMICAN Business Risk

2.1.1 Business Requirements Management

Activity

An industry standard approach was taken for the development of business requirements. The process started with the requirements that are documented in hard copy under former (AMICUS) module names.

• For the larger sub-projects they generated functional specifications through interviews with business experts, and then further developed these specifications through prototyping, JAD sessions, and developed use case analyses with key user representatives to validate the requirements.
• The smaller sub-projects may have used RAD techniques such as prototyping. The specifications comprise approved systems requirements documents with ratings from essential to desirable, and are very detailed.

These business requirements were then packaged into the new set of 6 AMICAN modules, and into releases for each module. The development schedule was set by logical order, since each module involves prerequisites to others. Another factor affecting priority was the direction to provide the same level of automation to Archives as Library already had.

• In general the business staff interviewed are very confident that their requirements have been captured and that the new processes will work successfully. They feel their staff were satisfactorily involved and are happy with the level of detail in the description of the requirements.

This project has redefined LAC's business processes - some of which were complex to begin with - and also handled the challenge of integration of processes which were previously separate in Library and Archives.

Conclusion

The entire project represents a very complex business situation in terms of the number of components and the relationship of the components to each other, as well as in the challenge of migration of the many sources of data previously held in unrelated incompatible databases. In some cases the project is going into business areas not previously automated.

In terms of business requirements risk this is a medium to high risk project: the breadth of business change is considerable as the AMICAN project is driven by the Department's transformation initiatives. The level of complexity of the business rules being defined is high and the additional factor is the integration of different business rules used by the previously separate sides of the Department. However the risk is being effectively managed for the modules being implemented in spring 2007.

However there are personnel changes in progress that may affect successful specifications analysis for later modules.

Risk

Impact: M
Likely: L

Suggestion:

It is suggested that LAC ensure the new business lead(s) have sufficient and broad knowledge of the business and that other personnel involved have a "big picture" view, have experience in the business side of both Library and Archives and understand the relationships among the different catalogues.

---

2.1.2 Solution Design

Activity

After the business requirements were finalized a detailed design was created and reviewed with selected superusers. The next step was to solicit input from users using prototypes. A final review of requirements is now being done prior to implementation.

• All users interviewed were very satisfied with the degree of user input. There have been very few changes requested as a result.

Elements of internal control have been strengthened with the new architecture. All modules include access controls at the user and record level. The databases will produce audit trails to protect data integrity.

• 3 Threat Risk Assessments (TRAs) have been done: one on IADAS data, one on AMICAN as a whole, and for CIM, both a TRA and a PIA were done. The requirements of all TRAs have been met, and in the case of the PIA for CIM, the requirements have been partially met, and plans for future compliance are in place.

It is difficult to evaluate the relationship of investment in cost and time to functionality received as there are no cost per function point measures. However several IT project leaders have said that the cost has been higher than necessary because of the stop and start nature of the funding, leading to higher turnover of consultants or paying them during idle times to keep them on.

Conclusion

There is very little to criticize in the way the solution design has been done. Users are extremely satisfied and appropriate security measures have been implemented. A minor concern is that there is no measurement of cost vs. functionality.

The development of the business solution is directly and adversely affected by the stop and start project funding approach discussed earlier. Addressing the funding issue would have a positive effect here.

Risk

Impact: M
Likely: L

Suggestion:

It is suggested that LAC develop some means of measuring investment vs. functionality for future use.

---

2.1.3 Management of Organizational Change

Activity

This project clearly has a huge impact on the major business processes of LAC; these huge changes are driven by the Transformation objectives. Of the people interviewed, no one had any doubts about the ability of the organization to deal with change or the staff's willingness to accept it. Acceptance has been promoted in several ways:

• There has been considerable participation from the business side,
• Process decisions were made at PRC with representation of all business areas,
• Users have worked with demos of the system,
• A communications team has active since the start.

Conclusion

While the impact on business processes is major, the project has done more than adequate preparation of both its procedures and its users. There seems to be a low risk of resistance affecting adoption of the new system or benefits realization.

Risk

Impact: L
Likely: L

No recommendation required

---

2.2 AMICAN Project Risk

2.2.1 Project Organization and Management

The roles and responsibilities of the project leaders, managers, team leaders and members of the various committees and project meetings at all levels are documented, and are understood by the participants.

• The majority of interviewees feel that project management works extremely well.

There are concerns about insufficient project resources both in terms of numbers and expertise.

• There is insufficient depth throughout the project. Project team leaders are overstretched in the variety of their tasks and responsibilities.
• There are tensions between IT permanent staff and contractors. There are debates over outsourced development vs. cheaper in-house development causing delivery delays.

The unpredictable nature of project funding is the biggest cause of problems. There is a constant stop and start of funds. Project leaders are often asked to rework the project plan to match funding available.

• The AMICAN project relies very heavily on contractors; the development staff is 80% consultants while the project management staff are 80% permanent employees. Project success depends on keeping the very experienced consultants on board. Procurement for consulting resources is a lengthy and cumbersome process. The Supply Arrangement used for consultants is about to expire and a new one will have to be created.
• Delays or unnecessary expenditures occur when money becomes unavailable because contractors are either terminated then rehired, or else are kept on but idle.

The project does not appear to be planning for adequate maintenance support, neither for the applications nor for the infrastructure.

Conclusion

The roles and responsibilities of the project leaders, managers, team leaders and of the various committees and project meetings at all levels are thoroughly understood and work extremely well.

The majority of the concerns relate to staffing; these concerns break down into three areas: insufficient resources, an over-reliance on contractors, and difficulties in procurement of contract resources. The project does not appear to be planning for adequate maintenance support.

The stop and start nature of the funding has led to project delays and to costs which are higher than necessary. Continued staffing and funding of the project and preparations for support and maintenance post-implementation are high sources of risk.

Risk

Impact: L
Likely: L

Recommendations:

It is recommended that LAC address human resource issues by:

• Establishing a plan and procurement vehicles that permit hiring contractors on a long term basis,
• Ensuring transfer of skills to permanent staff,
• Making staff aware of the Director General's La Relève plan,
• Building a plan to provide maintenance resources for new systems in production.

---

2.2.2 Project Control

Activity

Meetings at all levels are held regularly, are well controlled. Issues and roadblocks are presented and inter-team dependencies are made known. Project leaders do weekly status reports and have biweekly project meetings with the two project managers.

Plans are done in detail in MS Project however estimates are required to be done with a bare minimum of resources and time and without sufficient buffer or contingency built in. Budget uncertainties make it difficult to plan.

Data, which had to be migrated into the new systems, comes from numerous small incompatible databases that are difficult to combine.

Risk management meets EMF standards and the AMICAN risk management plan. While risk management is being done conscientiously at the project module level, the PRC rarely discusses risk issues as shown by a check of the October 2005 to October 2006 PRC minutes.

• The PRC risk register did not change during that period and was mentioned in only a few meetings.

Conclusion

Project management is done as effectively as possible under the circumstances but would be more effective if the underlying funding pressures are resolved.

• The project estimating and solution design were affected to some extent by unforeseeable data incompatibility issues. Similar factors will affect the migration of preservation data in future. Unforeseeable difficulties with the data can be considered a medium risk to the project.
• While risk management is being done conscientiously at the project module level, the PRC rarely discusses risk issues. Most of the time the risk management strategy is simply to acknowledge the existence of the risk. Thus the risk management process may not be completely effective.
• Project leaders are generally satisfied with the way project management and communications occur. However there are issues in project management caused by the uncertain nature of funding. Staff do not have confidence in their estimates because they are pressured to produce plans with a bare minimum of resources.

Risk

Impact: L
Likely: L

Recommendation:

It is recommended that LAC strengthen the risk management procedures at the PRC level

Suggestion:

It is suggested that LAC build sufficient contingency into plans for future projects involving migration and conversion of numerous databases e.g. preservation data.

---

2.2.3 Development Process

Activity

The solution design is well integrated, but very modular because it has been planned to be implemented very gradually due to lack of funds. While there is no formal SDLC the construction process can be described as waterfall, and the steps generally are industry standard. The methodology and the tools to be used have been standardized. A WBS is done for each module and weekly meetings are held at the project team level to track progress. Both the IT and the business managers are constantly aware of all activities.

Conclusion

Definition and control of the development process is satisfactory.

Risk

Impact: L
Likely: L

No recommendation required

---

2.3 AMICAN Testing Risk

2.3.1 Test Strategy and Organization

Activity

Testing is done first by developers and then by the UAT (user acceptance team). ITB has gotten commitment from business staff to do the testing.

• In the case of the CoC module, planned for production in May 2007, the test team consists of approximately 40 business people. This team has been in place since the requirements definition stage of the project and they know both the system and the business very well. They train superusers who then train the testers; they also develop the use cases.

There were however some issues raised concerning past test experiences, specifically some unfavourable experiences with AMICUS, which apparently contained defects when it was released in 1995, and with FS/PAM where the prototype testing phase was cut short.

Conclusion

The overall approach to testing is comprehensive and follows generally accepted practices. Management fully supports the testing approach as evidenced by the fact that they have assigned a large team of business staff to the UAT. It appears that the project management has learned from the lessons of the testing of the previous modules.

Risk

Impact: L
Likely: L

No recommendation required

---

2.3.2 Test Plan and Results

Activity

The testing approach for CoC and CIM, which are about to go into production, is to have the development people do their testing first.

• Superusers have been involved and do testing all along the development process. They also train the other testers. Then testing users test the system using test cases originally written by users.
• The Documentation-training Team prepares communications to UAT team on when the testers will be needed to do the testing.

There is a bug/change tracking system used during UAT. Weekly reviews are held with users where they discuss severity and priority of bugs found. These are logged and placed in release packages.

The QA environment is strictly controlled by the Configuration Management group. In QA testing the differences that exist between the development and production environments is a risk:

1. Lack of a production-sized database. Performance may be an issue since testing has been done on a partial production database only.
2. ORACLE RAC is not available for testing on the development platform but will be used in production.

Conclusion

Testing plans are consistent with best practices. The level of user and management involvement is impressive. All reasonable efforts to avoid testing risk have been made.

The only potential area of risk occurs within the QA environment. There are issues with the lack of space to use a complete copy of the production database, and the absence of Oracle RAC that will be used in production. The Oracle RAC issue has been mitigated somewhat by the recent decision to implement first in a production environment without Oracle RAC then implement it with RAC later on.

Risk

Impact: M
Likely: L

Suggestions:

It is suggested that ITB ensure good support from Oracle and HP at the point when ORACLE RAC is tested and implemented in production.

It is suggested that ITB be prepared to monitor and diagnose performance problems, and to increase capacity quickly if necessary.

---

3. TDR RISK

3.1 TDR Business Risk

3.1.1 Business Requirements Management

Activity

TDR will provide many of the tools needed to meet service delivery. The TDR Cluster is broken into 4 sub-projects, and the approach used to specify the business requirements was varied because of the nature of the individual sub-projects. Overall, TDR is still at the very early stage of determining what is needed and proposing solutions.

• The TDR sub-projects represent a very complex business situation in terms of the number of components and the relationship of the components to each other. However, the business side has worked hand in hand with the IT representatives and provided a good link between functional and business requirements.

Although there is a sense that the ground has shifted and that things have been getting better in the last 6 months, several indicators of risk were evident.

• From the business manger's perspective, the groundwork is being carried out but is still not complete. Further cycles of research are needed to determine requirements for digital collection.
• Some TDR PRC members see a risk in not having sufficiently specified all the business requirements yet still going ahead with system development.
• TDR is a still young project, and the requirements were not as well defined as for AMICAN.

Conclusion

The four sub-projects are progressing relatively well although they have been managed loosely and separately in the past year. With the new PRC and the cluster structure implemented last fall, a more homogeneous approach for these initiatives should become evident.

• Overall there as been a lot of committee work put into the business requirements definition. There is evidence of a good management structure in place. However minutes are not always taken in committee meetings and could lead to issues falling into the cracks.
• Management is making a good assessment of the challenges. However in reality we see that there is a gap between the ongoing development and the business requirements readiness. There is a significant risk that the product will not address a number of business requirements for the upcoming September project deliverables.

Risk

Impact: H
Likely: M

Recommendation:

It is recommended that business requirements and contract deliverables for September be reviewed. And a fit-gap analysis carried out to clarify the present and future requirements to be addressed by LAC.

Suggestion:

It is suggested that the SC be conservative and enforce committee decisions to mitigate the risks.

---

3.1.2 Solution Design

Activity

ITB feels that Deloitte has produced a good functional requirement document for the business side of conceptual TD Repository.

• The solution requires an ingestion component (VLD) to load and store all the collections. There is an important need on the LAC side to build a user case and own the issues around completeness and types of objects; audit trails; OAIS standards; and, institutional responsibility of archiving.
• Staff interviewed did explain that in phase1 infrastructure modules were being delivered first, such as security, virus detection, and ingestion tools. The user will interact less with the end functionality. This will allow more time for LAC to complete the business requirements in parallel.

There is evidence of good scoping documentation for the TDR project including a conceptual architecture, hardware and software requirements, etc. A contract is in place for some of the VLD component and is scheduled for delivery in September by the integrator.

• SC Members raised the issue of data migration as a significant risk. This is one area where there is a need to factor in trying to get everything perfectly migrated versus moving along with a reasonable degree of comfort.
• Currently the TDR solution is composed mostly of new experimental technologies.
• There are some concerns about the storage capacity requirements and costs for e-pubs and web harvesting. Cost is high and success linked to acquisition of new infrastructure through TB funding. Otherwise LAC is at risk of not harvesting at all with current infrastructure.

Conclusion

Although the road map is showing a "phased in" approach, we have noticed that the development work is moving faster than the analysis and specification work because of the contractual arrangements and budget availability.

• The VLD portion represents the No 1 business risk. If this fails the whole digital repository idea is jeopardized. It is the key component of the digital collection.
• However, the VLD requirements definition work is still at the conceptual level while the VLD component is being developed for delivery in September 2007. So questions exist - how will the missing requirements be integrated later? At what cost? How is the testing going to be done - at the conceptual level?

The TDR project is either in a prototyping approach mode or in a good faith relationship with the integrator.

• A good faith relationship focuses on the confidence in the integrator's expertise to deliver the needed modules for future digital objects collections such as: security, virus detection, ingestion tools, etc, (for which there is no trace of detailed functional specifications). In this kind of relationship, LAC must complete the business requirements in parallel. The risk of this relationship has not been explored.

Risk

Impact: H
Likely: M

Recommendation:

It is recommended that project management focus on completing the VLD requirements definition work.

• A detailed task list of activities for each of the VLD solution deliverables should be developed.

Suggestion:

It is suggested that LAC develop a closer relationship with the integrator and be pro-active in determining clearly what each party is providing and how the integrator will technically conform with contract deliverables.

See also Development Process.

---

3.1.3 Management of Organizational Change

Activity

Interviewees felt that LAC has not been effective in managing system driven changes and the impact of such changes on the business process. For example, after doing pilots to determine the cost/benefits of digital storage, the SC is now looking at how they can accelerate the big picture to capitalize on these benefits. However, Business Managers feel that the project is moving towards a target that is not well understood. More research is needed.

• Management thinks that change in technology from paper collections to digital process does not represents a big risk to LAC's human resources; that people will adapt to the changes. Getting the tools designed properly is the biggest risk.
• However, there is a significant risk in abandoning the paper trace of documents and guaranteeing that the technology picked today to store digital collections will enable preservation in 50 years.
• The modularity of the new system does mitigate this risk somewhat. Managing the project in a modular approach means that change is also incremental.

Conclusion

It is too early in the TDR project to conclude how change management is being handled.

• A positive note is the presence of a very well structured risk register that will help log issues around changes, actions to take and follow up on issues.

The business representatives view the selection of technology as a risk. There is no option in looking back to paper methods to mitigate that risk because many sources of information today are only available on web sites and in digital format. So management will need to be conservative and exercise proper due diligence for the selection of future technologies.

Risk

Impact: H
Likely: M

Recommendation:

It is recommended that LAC management focus on mitigating the risks of technology changes by:

• Fully exploring the risks and discussing them at PRC / SC
• Having TDR project management and staff adhere closely to the international community focus groups.

---

3.2 TDR Project Risk

3.2.1 Project Organization and Management

Activity

TDR is an umbrella of sub-projects under the authority of the TDR Cluster Mangers (Business & IT) who report to the PRC.

• The Virtual Loading Dock (VLD) is the most crucial and vital component for the future of the Digital Repository. Currently 80% of the activity is around the VLD sub-project and thus represents the biggest risk.

Roles and responsibilities appear to be clear. However, the entire project represents a very complex situation in terms of coordinating the inter-dependencies and roles and responsibilities between the players.

• Overall the partnership between LAC and the external service provider, and between the business and ITB, is working well and on a good faith basis, but there are some resource issues.
• The IT Cluster Manager is operating with borrowed resources - no resources are shown as reporting to her position in the organizational chart. Often when resources are needed, they are not available or the manager must deal with staff that have other priorities.

Conclusion

There is a good cohesion between the ITB and the Business sector, and SC members are happy with the TDR project organization in place.

Although overall the project management is working well, the IT Cluster Manager is operating on borrowed staff from ITB and being sometimes relegated to second priority. Since the VLD project is the highest priority one wonders why the IT Cluster Manager has no employee working under her authority to ensure the success of this project.

• This seems like the weak area of this management structure and constitutes a significant risk especially when deliverables start arriving.

Risk

Impact: M
Likely: M

Recommendation:

It is recommended that ITB dedicate IT resources to the VLD Project and have them report to the Project Manager.

---

3.2.2 Project Control

Activity

A project plan and approach was prepared jointly with the integrator. The TDR project plan is at a very high level, and we have not seen evidence of detailed deliverables and an associated task list.

• ITB feels that the plan should include the deliverables of the business side
• The test cases are part of the integrator set of deliverables. However, it is not clear how they are to align with the high level functional specifications. A mitigation strategy has been discussed and would consist of internal staff doing detailed specifications breakdown.

Until recently, the TDR project had few project management controls in place. The TDR Cluster Manager plans to establish a control framework for the next fiscal year. Current controls include:

• Cluster Managers conduct weekly meetings, with all the Business and IT Team Leads.
• A risk log is maintained and is discussed at the TDR Management Committee.
• Time sheets and status reports are in place to assess the work in progress.
• Quality Control Assurance is an area where they feel that they have done well during the planning phase and will be able to reuse at QA time.
• Business subject matter experts were teamed with IT resources to produce control requirements and users were involved in design validation.

Conclusion

The plan is such at a high level (no detailed deliverable or task list) that the integrator could legally deliver something that would not meet LAC expectations. Although, the teams have been working in good faith since the beginning and have been very professional, this potentially allows unexpected and unsatisfactory deliverables.

• Project management has not evolved into what it should have. This now represents a risk by leaving a loophole for undefined detailed deliverables.
• Although a good track record has been observed up to now, the problem is that the design and the build phase were embedded into one contract with the same supplier. This creates a situation where LAC does not have control over how much functionality it is going to get and opens the door to a potential risks. The product might not meet the LAC expectations but could still conform to specifications.

Overall the controls and the expertise are in place to support the process. After reviewing the documentation it is fair to state that good project summary presentations have been made to inform management of the findings and the progress of activities. There is evidence of project plans and status reports for all the TDR sub-projects.

• However minutes are not always taken systematically. Minutes will play an important role in the project completion taken in consideration the number of players involved.

Risk

Impact: M
Likely: M

Recommendation:

It is recommended that project management address project control risks by:

• developing a task list associated to the project deliverables.
• having dedicated resources implement a plan to execute this task list over the course of the project.
• relying on the risk register to review the plan on an on-going basis.

It is recommended that project management enforce delivery of minutes and status reports to management.

---

3.2.3 Development Process

Activity

LAC is partnering with Deloitte and PWGSC for the delivery of TDR. This partnering provides for a larger scale of expertise for the development of the TDR.

• Design is based on a central repository and architecture and Deloitte is using proven technology (Interwoven) that reduces risk. A road map of deliverables exists and good documentation has been produced
• For the other projects, WH and JPEG 2000, LAC is doing good research efforts and preparing recommendations for decision-making process before going ahead

However, dealing with an integrator has its own risks that must be managed by LAC management. Deloitte promised big things to get the contract and now must deliver on their promises. ITB believes that LAC runs a significant risk by outsourcing the development work to an integrator, when project oversight and control cannot be maintained.

• Management has set timeframes that were aggressive. In this time period, the project has been delayed twice (largely from a lack of stable funding) from January to March and then to July and now September 2007.
• As a result, the integrator has to deliver in a short period of time and there is a concern that the integrator will not be able to deliver as planned
• Also scope definition is ambiguous so in consequence the results are likely going to be different than expected.

Conclusion

Overall, development by integrator seems adequately managed. There seems to be some mistrust between ITB and the integrator that needs management attention to sort out.

• This type of situation can pose a serious risk on the project if the energy is not directed in the right way.

Overall the tools, the documentation and the practices deployed by the integrator look good. However, the delivery of an acceptable solution by the supplier could represent a risk of dispute. Although a good track record is observed up to now, it is always a delicate situation when the specifications are produced at a level where things are not concrete.

• There is a risk that LAC does not have sufficient control over how much functionality it is going to get. The product might not meet the LAC expectation level but would conform to the specification delivered.

Risk

Impact: H
Likely: H

Recommendations:

It is recommended that LAC management address development risk by:

• developing a closer relationship with the integrator
• implementing controls that are realistic for both parties

---

4. Technology Risk

4.1 Infrastructure

Activity

There is insufficient capacity to run the current load or the projected load. Performance and uptime are unsatisfactory. The infrastructure seems to be unreliable at present; there are numerous instances of unplanned downtime. The infrastructure is under-funded though designed to be scalable thus will be able to grow quickly if funds are made available.

• ITB does not see a big risk in the setup and management of the TDR infrastructure, although problems with the infrastructure did occur when doing the second run of Web Harvesting. The new infrastructure is expected to resolve this problem

For AMICAN, HP carried out an Oracle sizing study to develop capacity requirements for the production environment. 2 of 3 servers needed are in place and are adequate for the CoC start in May 2007. However there will not be sufficient capacity for the whole AMICAN project; capacity plans call for one more server but there is no definite date for its purchase.

• For development and QA testing, the environments are on the same server; the server performance is slow and it lacks capacity for adequate volume testing. However a new development server will be in place soon.
• Public expectations of availability will likely increase in October when the CBC genealogy show airs. This is an issue that should be clarified with an SLA.

There was a good requirements definition document done for the first phase of TDR. But ITB concerns included the fact that the long-term infrastructure requirement is not well analyzed. For example, the network area storage requirement is a risk. This situation is considered a good example of doing things in separate phases that are funding driven.

• If ITB does not get the funding from TB it would still be able to purchase with LAC internal funds and the storage area network (SAN), the most important component, could still be put in place. They would still be able to support the TDR project for phase I. However, they would also run the risk that in the event of another crisis there would not be a lot of resources in reserve.

Conclusion

The infrastructure is by general consensus the weakest link in the project. The weaknesses are in two areas: the hardware platforms; and the support personnel.

• The hardware platforms are aging, almost obsolete. This aging equipment will need replacement but funding is the driving factor.
• The scalable architecture design means that additional capacity can be added relatively quickly and easily; the issue is not a technical one but a financial one. Infrastructure upgrade plans depend on obtaining funds via the upcoming TB submission.

There are plans to upgrade the infrastructure. In the interim, there are concerns about the capacity to handle large number of inquiries which will be triggered by the CBC show focusing on genealogy which will air in October.

Risk

Impact: H
Likely: M

Recommendation:

It is recommended that LAC develop a Service Level Agreement which would specify the capacity, reliability and availability targets required for the AMICAN modules about to go into production, with FS/PAM being a priority. These targets should then be used to determine whether the infrastructure needs additional capacity and redundant capability.

Suggestions:

It is suggested that if TB funds are forthcoming by April 1 2007, adequate investment in infrastructure must be made as a priority.

It is suggested that if TB funds are not forthcoming by April 1 then the department will need to adjust the scope of the AMICAN project to what can be funded internally. However investment in infrastructure should remain a priority.

---

Activity

There are related risks within the infrastructure environment in LAC.

• Verity (web content management software) is now at end of life - is to be replaced by an Autonomy product called IDOL 7, thus there is now no bug fixing or maintenance from the vendor. If left unsupported, LAC would have to retire Federated Search and go back to individual databases. There is only one staff member who has the expertise.
• Generally interviewees feel that the availability of the system, particularly the FS/PAM module, is unsatisfactory. While no specific availability targets have been stated (and no SLA is in place) the implication of direction from senior management is that public web search functions should be available 24/7 otherwise LAC's reputation will suffer.
• Currently, there are no hot and cold sites for contingency planning. There used to be an insurance plan supported by HP for an infrastructure recovery in case of disaster. That insurance was cancelled by PWGSC some years ago since no systems at LAC were confirmed to be mission critical by TBS.
• IIM complains about getting budgets late in the year and then having to scramble with few staff resources after the funding arrives, everybody wants their infrastructure setup quickly. This creates a lot of pressure on the staff and puts proper installation planning at risk.
  • Pressure was put on the IIM group to provide the infrastructure early for the first deadline to support the TDR initiative; after that the deadlines were set back. This is creating animosity and with reason. They were told to move fast and then told later we won't need it for another 3 months.
  • A specific problem exists in the infrastructure directorate; not only are they understaffed, but they are involved with many new projects and also spending 50% of their time supporting aging and unreliable hardware.

Conclusion

While concentration is focused on the delivery of AMICAN/TDR, residual systems and support are suffering from lack of investment.

The division of responsibilities and the communication between applications and infrastructure project resources are not working smoothly. The responsibilities of the IIM staff in planning infrastructure changes and their inadequate personnel resource situation are potential risks to the project.

A concern raised concerning TDR is that business people are reluctant to invite IIM to the table as IIM is not considered a project player but general service providers to all groups. So, IIM always finds out at the last minute about infrastructure requirements, and complains that they don't have the operating budget to maintain it, and delays are incurred before the infrastructure is put in place.

• The ITB perspective over this issue is; 1- Have them on the team early; 2- Plan IT infrastructure requirements in the design phase of the project; 3- This is where good PM methodology is missing and could help change things around.

Risk

Impact: M
Likely: H

Recommendations:

It is recommended that general infrastructure support be reviewed with the objective of highlighting and mitigating operational risks.

• Specifically, concerns about IIM should be assessed by reviewing the staffing situation in the infrastructure directorate

It is recommended that LAC address risks related to lack of backup and recovery. Specifically LAC should develop SLAs and recovery plans covering all critical operations and services and ensure that senior management is aware of the target time for restoration of services.

---

4.2 Technology Transition

Activity

The new architecture for AMICAN is based on load balanced scale-out HP servers, Oracle RAC and Linux. While some of the application project leaders and the project manager have some anxieties regarding the transition to Oracle RAC and to eDirectory as a basis for CIM, the infrastructure staff do not.

• The move to HP from Alpha servers is not an issue, the Federated Search module has already been running on an HP server for some time and infrastructure staff are familiar with it.
• There is some question re whether the fact that the QA environment is different from the target production environment constitutes a risk. Oracle RAC is the only new aspect of the production platforms; the Oracle client does not change, is simply directed to a different database, but developers will see no change.
• According to infrastructure staff, the move to Linux is minor risk as systems administrators are already trained on it and it is similar to UNIX which is already in place.
• While some applications staff are concerned about the fact that the Novell edirectory will be basis for CIM, infrastructure, staff consider that it is reliable as it has been in use at LAC for 10 years.

As far as the applications and business processes are concerned, changes are being approached in a modular fashion. Post-production operational procedures are not a concern to most people; existing change, problem and release management procedures will apply. However in the case of CoC the business lead is uncertain about how change management and release management will be handled in production.

• CoC implementation will be a straight cutover to the new system but the plan is to retain Trakker for a few months as a contingency. If it is necessary to fall back to Trakker all data entered in CoC will have to be re-entered in Trakker; it is not clear how this would be done or who would support Trakker.
• There is no mechanism to consider changes after production. Configuration management is done by the CM group for applications only. They manage a QA and a Production Hold environment with UNIX-Linux RCS (automated tool) for version control. Code must be checked in and checked out via this system with permission of CM group. However, there is no CM for infrastructure - this is being established now.

PWGSC currently manages the contract for TDR. They also will be managing the IT system infrastructure for LAC through an agreement to bring LAC on board in the second wave of a GOC shared services initiative. LAC expects to lower its risk with this collaboration and could delay until the third wave if the first wave does not work out smoothly. However, deployment has risks:

• Infrastructure is not yet in place for the TDR sub-projects going ahead in production in September. Still unclear on sizing and what to buy. Awaiting recommendations and approval.
• The TDR Project Manager feels the requirements in space are huge and very expensive (web harvesting, digitization, digital audit/visual). For example, there is a risk that web harvests may be limited to the size of the budget and harvesting limited sets of information or collections. ITB needs an overall approach to data storage planning.

Coclusion

Three elements - the technology, the systems and the business processes - are all changing at the same time. Changes are being implemented in a modular way, as far as the applications and business processes are concerned. The technology changes include three new elements (Oracle RAC, LINUX and the HP servers), but only Oracle RAC is new, the others have been in place for a while. However, the scope of the activity, the dependence of future operations of LAC on technology, and the degree of change involved means this is a risk

• Transition risk is quite often under-rated. At the present time, this risk is being well managed / planned by IT management. Nonetheless, more attention to planning is needed, mainly in the area of infrastructure
• Production procedures are in place but perhaps are not well known to all the project leaders concerned.
• Trakker will apparently be shut off after a few months of successful operation of CoC. However there is no plan to fall back to Trakker if CoC were not successful.
• There is no configuration management for infrastructure; lack of CM when the new modules are in production is risky.

Risk

Impact: M
Likely: H

Recommendations:

It is recommended that LAC address transition risks by:

• ensuring good support is available from Oracle at the time of transition;
• developing better communications between AMS and IIM staff to alleviate AMS staff's concerns;
• ensuring a fallback plan is in place which includes application support arrangements for Trakker and a method for updating data in Trakker if necessary;
• clarifying the change, release and problem management procedures that will apply when CoC goes into production;
• implementing formal Configuration Management at the same time as the new modules.

---

Activity

There are insufficient personnel resources to support the infrastructure and to support projects at the same time - insufficient depth in numbers and in expertise. In several cases there is only one person in the infrastructure division who is knowledgeable about a specific technology but this person has no backup.

• The infrastructure staff have only recently gone on training despite the fact that Oracle RAC was purchased by LAC about a year ago so there is a risk that they have inadequate experience with it
• It is felt that infrastructure staff have inadequate project management skills, and do not produce the required types of plans showing how environments will be changed and developed
• Applications maintenance will be carried out by permanent LAC employees. Although staffing for maintenance is adequate, it is not ideal - ITB staff are stretched with regard to number of applications and tools they have to work with. They are being trained now by being assigned to work on the developed modules.
• The applications project manager is in the process of preparing a plan showing the movement of databases across servers in a phased implementation. It has been proposed, and agreed, that Oracle RAC would not be used in production immediately, in order to minimize the risk.

Conclusion

ITB management and staff do not see technical transition as a big risk; staff to support the growing demand is the issue. Should the investment arrive, LAC will get more resources and be better able to plan all elements of the infrastructure replacement including long terms needs. However, even with the investment, insufficient staff to support the upgrade can have detrimental effects. Staffing for application maintenance is adequate but not ideal.

There is a risk of ongoing system support failing during the effort to provide for the new AMICAN / TDR suite of products and tools. Ensure continued support remains a priority.

Risk

Impact: M
Likely: H

Recommendation:

It is recommended that in cases where there is only one person in the infrastructure division who is knowledgeable about a specific technology, that each expert has at least one backup.

Suggestions:

It is suggested that for future projects, ITB clarify the responsibilities of the IIM staff in planning infrastructure changes, and provide project management training if necessary

It is suggested that LAC ensure some of next year's O&M funding is directed to adequate maintenance staff.