This archived Web page remains online for reference, research or recordkeeping purposes. This page will not be altered or updated. Web pages that are archived on the Internet are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats of this page on the Contact Us page.
The detailed results of the AMICAN / TDR Preliminary Survey are presented in the format of a risk matrix. These results are a snapshot as of March 30, 2007. The review took place from January 29 to March 16 and considers that the situation further described in the matrix will continue to evolve rapidly.
The purpose of the matrix is to identify factors that could affect the successful delivery of AMICAN / TDR business solutions, on time, within budget, and that meet user requirements. The Risk Framework used to conduct the review is composed of five risk categories: Governance Risk; Business Risk; Project Risk; Testing Risk and Technology Risk. Each category of risk has been organized into three columns describing: the related project activity; the review conclusions and assessment of risk, and either an indication that no recommendations are required, or suggestions and recommendations for those areas where improvement would reduce the risk to an acceptable level.
The risk assessment describes the nature of the threat, the likelihood of it occurring, and the consequences of that threat. The risk assessment is based on the experience of the reviewer. The difference between projects fostered by unique business problems for which a solution is being sought, different business and project organizational approaches, personal project management styles, and different project control and system development methodologies, works against the applicability of a statistically derived assessment methodology. In this review, the following definitions are used:
Likelihood of Risk Materializing
Impact if Risk Materializes
Within each risk category and sub-category, there are numerous individual measures that are recorded in the following matrix. As some measures are not unique to an individual category, some apparent duplication is intentionally included for readability.
The governance structure as described in the project charter consists of a Steering Committee (senior management representatives from across the organization), a Project Steering Committee (operational management representation and chaired by SC members), and a set of project and business leads for each sub-project.
The AMICAN / TDR governance structure and process overall is well organized and is in line with recommended best practices.
AMICAN and TDR both arose out of the Transition initiatives bringing the Library and the Archives together. AMICAN was first to push ahead and has developed a sound governance model. Partly as a result, TDR was moved under its umbrella despite the fact that some feel it doesn't really fit with the AMICAN mandate.
It is recommended that LAC review and confirm the roles of the various committees.
The committee structure is relatively new and the role of the SC versus that of the PRCs has only recently been fully defined. As a result, some committee members felt that the roles and responsibilities were not fully understood by all.
The role of those committees within the charter definition, and some outside the project charter as well, seem to create confusion in the minds of stakeholders.
How the PRC reports to the SC and the level of representation on the PRC should be reviewed and refined.
It is suggested that once the roles have been confirmed, SC / PRC committee members develop guidelines for the level and extent of discussion.
Although the information flow is seen as adequate for committee members, there is a risk that the information is perhaps too voluminous or not well designed for the senior management and middle management members.
The information provided to the committee should be more structured and should include a high level view of schedule, cost and functionality achievements, followed by risks and issues.
It is suggested that committee reports be short bullet-type reports on milestones, outcomes, schedule, key challenges, and decisions required and when necessary zoom into specific issues.
Any issues affecting scope are subject to the change request process. Change requests are formalized and tracked (the change request form is documented in Project Plan and the change approval process is documented in the Project Charter). A registry of change requests is maintained.
Nonetheless, the chair of the Steering Committee feels that scope management is limited in its approach. Short-term decisions are made without looking at the big picture. When a decision goes forward with a set of functionality for a specific module, the downstream effects on proposed future modules is unsure; there is no big picture view.
The change management approach is industry standard and appears to work well, although it has not been used very much within AMICAN. In fact there have been extremely few changes made to the project since the original requirements definition work was completed.
While it is difficult to actually independently evaluate if scope management is limited in its approach, there is no doubt that funding uncertainties (see following section) have resulted in short-term decision-making. What is important here is that the SC Chair is uncertain - and that the project must make sure that certainty is fully achieved - better information would be part of the solution.
It is recommended that Project management address the ADM's concerns regarding sufficient regard for the larger picture when making change decisions.
All interviewees agree that project funding in the past has been dangerously sporadic; funds arrive piecemeal and unexpectedly as the department reallocates funds during the year to support AMICAN.
Senior management has promoted the view that the TB submission will be successful. No contingency plan has been prepared. As a consequence, the project team's expectations have been built up. There will be tremendous disappointment, loss of momentum and morale if it is not.
The major risk issue, funding, has not been adequately addressed. If the TB submission is unsuccessful, the department will most probably continue to fund the project (as stated in the business case). The risk affects not only the funding aspect but to the human resources involved.
However a specific funding plan has not been prepared and as a result there will be delays in the project, loss of expert staff and of project momentum.
The sporadic nature of funding has been the single most frequent cause of problems in this project. The project needs stable funding and a strategic funding plan over multiple years, whether the TB submission comes through or not.
Specifically, it is recommended that LAC ensure stable funding (whether through a successful TB submission or internally) to allow consultants to be brought on board for the full duration of the time they are required.
There is a concern about funding and how funds are used. The relationship between the funds invested and the results to date is unclear.
The risk of poor management of costs is high but should be more closely managed with the creation of the new committee.
It is suggested that ITB develop standards for cost estimating that are understandable to the non-technical person, and describe the target functionality in detail for each proposal.
It is certainly not easy to develop a purely financial case for the AMICAN side of this project vs. the TDR side. The main driver for this project is the need to fulfill LAC's mandate of preserving Canada's heritage, specifically its digital heritage.
Since AMICAN has become a model of good IT project management various other projects were moved under its umbrella, despite the fact that some interviewees felt they don't really fit with the AMICAN mandate.
The question of how achievement of benefits will be measured has apparently not been talked about at the PRC. It is apparently the new Steering Committee's mandate but the SC has not yet discussed it.
As a business case strictly for AMICAN, it is a little confusing in its financial aspects, though reasonably convincing for the other non-financial rationales. For example, there is no clear explanation of what the current maintenance costs are for AMICUS; the maintenance costs of the new system are not shown anywhere. So it is difficult to support the financial cost benefit if one looks strictly at the AMICAN side.
Some AMICAN project members now feel that scope and budget have changed so much that it is much more difficult to track progress against original goals. This raises the risk that benefits cannot be determined and measured. It also raises the risk that project schedule will eventually be affected.
It is recommended that LAC develop a detailed plan for measuring benefits achievement for AMICAN / TDR and begin now to track the selected measures to capture information about the pre-implementation situation.
As the business case is now finalized it is impossible to change it, but it is suggested that LAC prepare and provide O&M costs if requested by TB and for use in future budget planning.
It is suggested that LAC develop guidelines as to what does and does not belong in the AMICAN mandate and apply these guidelines to any new projects that may be proposed.
An industry standard approach was taken for the development of business requirements. The process started with the requirements that are documented in hard copy under former (AMICUS) module names.
These business requirements were then packaged into the new set of 6 AMICAN modules, and into releases for each module. The development schedule was set by logical order, since each module involves prerequisites to others. Another factor affecting priority was the direction to provide the same level of automation to Archives as Library already had.
This project has redefined LAC's business processes - some of which were complex to begin with - and also handled the challenge of integration of processes which were previously separate in Library and Archives.
The entire project represents a very complex business situation in terms of the number of components and the relationship of the components to each other, as well as in the challenge of migration of the many sources of data previously held in unrelated incompatible databases. In some cases the project is going into business areas not previously automated.
In terms of business requirements risk this is a medium to high risk project: the breadth of business change is considerable as the AMICAN project is driven by the Department's transformation initiatives. The level of complexity of the business rules being defined is high and the additional factor is the integration of different business rules used by the previously separate sides of the Department. However the risk is being effectively managed for the modules being implemented in spring 2007.
However there are personnel changes in progress that may affect successful specifications analysis for later modules.
It is suggested that LAC ensure the new business lead(s) have sufficient and broad knowledge of the business and that other personnel involved have a "big picture" view, have experience in the business side of both Library and Archives and understand the relationships among the different catalogues.
After the business requirements were finalized a detailed design was created and reviewed with selected superusers. The next step was to solicit input from users using prototypes. A final review of requirements is now being done prior to implementation.
Elements of internal control have been strengthened with the new architecture. All modules include access controls at the user and record level. The databases will produce audit trails to protect data integrity.
It is difficult to evaluate the relationship of investment in cost and time to functionality received as there are no cost per function point measures. However several IT project leaders have said that the cost has been higher than necessary because of the stop and start nature of the funding, leading to higher turnover of consultants or paying them during idle times to keep them on.
There is very little to criticize in the way the solution design has been done. Users are extremely satisfied and appropriate security measures have been implemented. A minor concern is that there is no measurement of cost vs. functionality.
The development of the business solution is directly and adversely affected by the stop and start project funding approach discussed earlier. Addressing the funding issue would have a positive effect here.
It is suggested that LAC develop some means of measuring investment vs. functionality for future use.
This project clearly has a huge impact on the major business processes of LAC; these huge changes are driven by the Transformation objectives. Of the people interviewed, no one had any doubts about the ability of the organization to deal with change or the staff's willingness to accept it. Acceptance has been promoted in several ways:
While the impact on business processes is major, the project has done more than adequate preparation of both its procedures and its users. There seems to be a low risk of resistance affecting adoption of the new system or benefits realization.
The roles and responsibilities of the project leaders, managers, team leaders and members of the various committees and project meetings at all levels are documented, and are understood by the participants.
There are concerns about insufficient project resources both in terms of numbers and expertise.
The unpredictable nature of project funding is the biggest cause of problems. There is a constant stop and start of funds. Project leaders are often asked to rework the project plan to match funding available.
The project does not appear to be planning for adequate maintenance support, neither for the applications nor for the infrastructure.
The roles and responsibilities of the project leaders, managers, team leaders and of the various committees and project meetings at all levels are thoroughly understood and work extremely well.
The majority of the concerns relate to staffing; these concerns break down into three areas: insufficient resources, an over-reliance on contractors, and difficulties in procurement of contract resources. The project does not appear to be planning for adequate maintenance support.
The stop and start nature of the funding has led to project delays and to costs which are higher than necessary. Continued staffing and funding of the project and preparations for support and maintenance post-implementation are high sources of risk.
It is recommended that LAC address human resource issues by:
Meetings at all levels are held regularly, are well controlled. Issues and roadblocks are presented and inter-team dependencies are made known. Project leaders do weekly status reports and have biweekly project meetings with the two project managers.
Plans are done in detail in MS Project however estimates are required to be done with a bare minimum of resources and time and without sufficient buffer or contingency built in. Budget uncertainties make it difficult to plan.
Data, which had to be migrated into the new systems, comes from numerous small incompatible databases that are difficult to combine.
Risk management meets EMF standards and the AMICAN risk management plan. While risk management is being done conscientiously at the project module level, the PRC rarely discusses risk issues as shown by a check of the October 2005 to October 2006 PRC minutes.
Project management is done as effectively as possible under the circumstances but would be more effective if the underlying funding pressures are resolved.
It is recommended that LAC strengthen the risk management procedures at the PRC level
It is suggested that LAC build sufficient contingency into plans for future projects involving migration and conversion of numerous databases e.g. preservation data.
The solution design is well integrated, but very modular because it has been planned to be implemented very gradually due to lack of funds. While there is no formal SDLC the construction process can be described as waterfall, and the steps generally are industry standard. The methodology and the tools to be used have been standardized. A WBS is done for each module and weekly meetings are held at the project team level to track progress. Both the IT and the business managers are constantly aware of all activities.
Definition and control of the development process is satisfactory.
Testing is done first by developers and then by the UAT (user acceptance team). ITB has gotten commitment from business staff to do the testing.
There were however some issues raised concerning past test experiences, specifically some unfavourable experiences with AMICUS, which apparently contained defects when it was released in 1995, and with FS/PAM where the prototype testing phase was cut short.
The overall approach to testing is comprehensive and follows generally accepted practices. Management fully supports the testing approach as evidenced by the fact that they have assigned a large team of business staff to the UAT. It appears that the project management has learned from the lessons of the testing of the previous modules.
The testing approach for CoC and CIM, which are about to go into production, is to have the development people do their testing first.
There is a bug/change tracking system used during UAT. Weekly reviews are held with users where they discuss severity and priority of bugs found. These are logged and placed in release packages.
The QA environment is strictly controlled by the Configuration Management group. In QA testing the differences that exist between the development and production environments is a risk:
Testing plans are consistent with best practices. The level of user and management involvement is impressive. All reasonable efforts to avoid testing risk have been made.
The only potential area of risk occurs within the QA environment. There are issues with the lack of space to use a complete copy of the production database, and the absence of Oracle RAC that will be used in production. The Oracle RAC issue has been mitigated somewhat by the recent decision to implement first in a production environment without Oracle RAC then implement it with RAC later on.
It is suggested that ITB ensure good support from Oracle and HP at the point when ORACLE RAC is tested and implemented in production.
It is suggested that ITB be prepared to monitor and diagnose performance problems, and to increase capacity quickly if necessary.
TDR will provide many of the tools needed to meet service delivery. The TDR Cluster is broken into 4 sub-projects, and the approach used to specify the business requirements was varied because of the nature of the individual sub-projects. Overall, TDR is still at the very early stage of determining what is needed and proposing solutions.
Although there is a sense that the ground has shifted and that things have been getting better in the last 6 months, several indicators of risk were evident.
The four sub-projects are progressing relatively well although they have been managed loosely and separately in the past year. With the new PRC and the cluster structure implemented last fall, a more homogeneous approach for these initiatives should become evident.
It is recommended that business requirements and contract deliverables for September be reviewed. And a fit-gap analysis carried out to clarify the present and future requirements to be addressed by LAC.
It is suggested that the SC be conservative and enforce committee decisions to mitigate the risks.
ITB feels that Deloitte has produced a good functional requirement document for the business side of conceptual TD Repository.
There is evidence of good scoping documentation for the TDR project including a conceptual architecture, hardware and software requirements, etc. A contract is in place for some of the VLD component and is scheduled for delivery in September by the integrator.
Although the road map is showing a "phased in" approach, we have noticed that the development work is moving faster than the analysis and specification work because of the contractual arrangements and budget availability.
The TDR project is either in a prototyping approach mode or in a good faith relationship with the integrator.
It is recommended that project management focus on completing the VLD requirements definition work.
It is suggested that LAC develop a closer relationship with the integrator and be pro-active in determining clearly what each party is providing and how the integrator will technically conform with contract deliverables.
See also Development Process.
Interviewees felt that LAC has not been effective in managing system driven changes and the impact of such changes on the business process. For example, after doing pilots to determine the cost/benefits of digital storage, the SC is now looking at how they can accelerate the big picture to capitalize on these benefits. However, Business Managers feel that the project is moving towards a target that is not well understood. More research is needed.
It is too early in the TDR project to conclude how change management is being handled.
The business representatives view the selection of technology as a risk. There is no option in looking back to paper methods to mitigate that risk because many sources of information today are only available on web sites and in digital format. So management will need to be conservative and exercise proper due diligence for the selection of future technologies.
It is recommended that LAC management focus on mitigating the risks of technology changes by:
TDR is an umbrella of sub-projects under the authority of the TDR Cluster Mangers (Business & IT) who report to the PRC.
Roles and responsibilities appear to be clear. However, the entire project represents a very complex situation in terms of coordinating the inter-dependencies and roles and responsibilities between the players.
There is a good cohesion between the ITB and the Business sector, and SC members are happy with the TDR project organization in place.
Although overall the project management is working well, the IT Cluster Manager is operating on borrowed staff from ITB and being sometimes relegated to second priority. Since the VLD project is the highest priority one wonders why the IT Cluster Manager has no employee working under her authority to ensure the success of this project.
It is recommended that ITB dedicate IT resources to the VLD Project and have them report to the Project Manager.
A project plan and approach was prepared jointly with the integrator. The TDR project plan is at a very high level, and we have not seen evidence of detailed deliverables and an associated task list.
Until recently, the TDR project had few project management controls in place. The TDR Cluster Manager plans to establish a control framework for the next fiscal year. Current controls include:
The plan is such at a high level (no detailed deliverable or task list) that the integrator could legally deliver something that would not meet LAC expectations. Although, the teams have been working in good faith since the beginning and have been very professional, this potentially allows unexpected and unsatisfactory deliverables.
Overall the controls and the expertise are in place to support the process. After reviewing the documentation it is fair to state that good project summary presentations have been made to inform management of the findings and the progress of activities. There is evidence of project plans and status reports for all the TDR sub-projects.
It is recommended that project management address project control risks by:
It is recommended that project management enforce delivery of minutes and status reports to management.
LAC is partnering with Deloitte and PWGSC for the delivery of TDR. This partnering provides for a larger scale of expertise for the development of the TDR.
However, dealing with an integrator has its own risks that must be managed by LAC management. Deloitte promised big things to get the contract and now must deliver on their promises. ITB believes that LAC runs a significant risk by outsourcing the development work to an integrator, when project oversight and control cannot be maintained.
Overall, development by integrator seems adequately managed. There seems to be some mistrust between ITB and the integrator that needs management attention to sort out.
Overall the tools, the documentation and the practices deployed by the integrator look good. However, the delivery of an acceptable solution by the supplier could represent a risk of dispute. Although a good track record is observed up to now, it is always a delicate situation when the specifications are produced at a level where things are not concrete.
It is recommended that LAC management address development risk by:
There is insufficient capacity to run the current load or the projected load. Performance and uptime are unsatisfactory. The infrastructure seems to be unreliable at present; there are numerous instances of unplanned downtime. The infrastructure is under-funded though designed to be scalable thus will be able to grow quickly if funds are made available.
For AMICAN, HP carried out an Oracle sizing study to develop capacity requirements for the production environment. 2 of 3 servers needed are in place and are adequate for the CoC start in May 2007. However there will not be sufficient capacity for the whole AMICAN project; capacity plans call for one more server but there is no definite date for its purchase.
There was a good requirements definition document done for the first phase of TDR. But ITB concerns included the fact that the long-term infrastructure requirement is not well analyzed. For example, the network area storage requirement is a risk. This situation is considered a good example of doing things in separate phases that are funding driven.
The infrastructure is by general consensus the weakest link in the project. The weaknesses are in two areas: the hardware platforms; and the support personnel.
There are plans to upgrade the infrastructure. In the interim, there are concerns about the capacity to handle large number of inquiries which will be triggered by the CBC show focusing on genealogy which will air in October.
It is recommended that LAC develop a Service Level Agreement which would specify the capacity, reliability and availability targets required for the AMICAN modules about to go into production, with FS/PAM being a priority. These targets should then be used to determine whether the infrastructure needs additional capacity and redundant capability.
It is suggested that if TB funds are forthcoming by April 1 2007, adequate investment in infrastructure must be made as a priority.
It is suggested that if TB funds are not forthcoming by April 1 then the department will need to adjust the scope of the AMICAN project to what can be funded internally. However investment in infrastructure should remain a priority.
There are related risks within the infrastructure environment in LAC.
While concentration is focused on the delivery of AMICAN/TDR, residual systems and support are suffering from lack of investment.
The division of responsibilities and the communication between applications and infrastructure project resources are not working smoothly. The responsibilities of the IIM staff in planning infrastructure changes and their inadequate personnel resource situation are potential risks to the project.
A concern raised concerning TDR is that business people are reluctant to invite IIM to the table as IIM is not considered a project player but general service providers to all groups. So, IIM always finds out at the last minute about infrastructure requirements, and complains that they don't have the operating budget to maintain it, and delays are incurred before the infrastructure is put in place.
It is recommended that general infrastructure support be reviewed with the objective of highlighting and mitigating operational risks.
It is recommended that LAC address risks related to lack of backup and recovery. Specifically LAC should develop SLAs and recovery plans covering all critical operations and services and ensure that senior management is aware of the target time for restoration of services.
The new architecture for AMICAN is based on load balanced scale-out HP servers, Oracle RAC and Linux. While some of the application project leaders and the project manager have some anxieties regarding the transition to Oracle RAC and to eDirectory as a basis for CIM, the infrastructure staff do not.
As far as the applications and business processes are concerned, changes are being approached in a modular fashion. Post-production operational procedures are not a concern to most people; existing change, problem and release management procedures will apply. However in the case of CoC the business lead is uncertain about how change management and release management will be handled in production.
PWGSC currently manages the contract for TDR. They also will be managing the IT system infrastructure for LAC through an agreement to bring LAC on board in the second wave of a GOC shared services initiative. LAC expects to lower its risk with this collaboration and could delay until the third wave if the first wave does not work out smoothly. However, deployment has risks:
Three elements - the technology, the systems and the business processes - are all changing at the same time. Changes are being implemented in a modular way, as far as the applications and business processes are concerned. The technology changes include three new elements (Oracle RAC, LINUX and the HP servers), but only Oracle RAC is new, the others have been in place for a while. However, the scope of the activity, the dependence of future operations of LAC on technology, and the degree of change involved means this is a risk
It is recommended that LAC address transition risks by:
There are insufficient personnel resources to support the infrastructure and to support projects at the same time - insufficient depth in numbers and in expertise. In several cases there is only one person in the infrastructure division who is knowledgeable about a specific technology but this person has no backup.
ITB management and staff do not see technical transition as a big risk; staff to support the growing demand is the issue. Should the investment arrive, LAC will get more resources and be better able to plan all elements of the infrastructure replacement including long terms needs. However, even with the investment, insufficient staff to support the upgrade can have detrimental effects. Staffing for application maintenance is adequate but not ideal.
There is a risk of ongoing system support failing during the effort to provide for the new AMICAN / TDR suite of products and tools. Ensure continued support remains a priority.
It is recommended that in cases where there is only one person in the infrastructure division who is knowledgeable about a specific technology, that each expert has at least one backup.
It is suggested that for future projects, ITB clarify the responsibilities of the IIM staff in planning infrastructure changes, and provide project management training if necessary
It is suggested that LAC ensure some of next year's O&M funding is directed to adequate maintenance staff.