Government

Leaving Archival Electronic Records in Institutions

Policy and monitoring arrangements

November 1993

This policy paper addresses why the National Archives of Canada may wish to leave archival electronic records of the Government of Canada in government institutions rather than transfer them to the control of the National Archives of Canada (NA). Such case-by-case decisions are authorized by the National Archivist and will respect the spirit of the National Archives of Canada Act by ensuring that archival records left in institutions are preserved, conserved, described, and made accessible. (The related issue of government institutions collecting and retaining private archival records is a separate one, to be addressed elsewhere, as indeed is the issue of leaving other media of records in government institutions: photographs, maps, etc.)

This paper outlines several circumstances and offers examples where leaving government archival electronic records in institutions may be appropriate, and suggests the model monitoring clauses that the archivist should have signed by the institution in order to facilitate the proper preservation of archival electronic records left in government institutions. The clauses focus on electronic records stored on magnetic tape, which is the immediate problem and the principal, although not sole, emphasis of this paper. It is recognized that similar monitoring clauses will have to be developed for other electronic media, that fully respect their physical characteristics and conservation concerns.

In writing individual Archival Appraisal Reports, archivists will use these guidelines, once approved, for two purposes: to negotiate and craft appropriate Terms and Conditions for the formal Agreement which institutions are to sign indicating their intention to comply, and to make recommendations to the National Archivist who will make the final decision.

A. Why Leave Archival Records In Government Institutions?

There are occasions, especially for electronic records, where it is not feasible or the best use of NA resources to acquire the records identified as archival at the time the Archival Appraisal Report is approved. The National Archivist may in such cases decide to leave electronic records identified as archival in the creating institution. Such occasions or reasons might include:

1. Where the cost of transfer of the record or other technical considerations (software copyright, data complexity, software and hardware dependency, etc.) make it impossible for the National Archives to acquire the record at this time, and/or
2. Where the institution has a continuing and long-term operational need for the record, which includes the provision of elaborate and extensive reference services, and/or
3. Where because of the nature of the record reference services can best be provided by the institution rather than by the National Archives, and/or
4. Where there are statutory provisions that prevent transfer to the National Archives of Canada.

In all such cases, the archivist, in making a recommendation to the National Archivist for any of these reasons, must ensure that the electronic record is not at threat or known to be (as part of regular data-processing routines) subject to erasure, updating, or deletion. The Agreement (with its attached Terms and Conditions) will specify, on a case-by-case basis, that the institution must ensure preservation of the records in a useable format, permit their description in archival inventories, and grant access to the records for researchers within the framework of the ATIP legislation, and that the National Archives will monitor compliance with these Terms and Conditions according to a specified timetable. The Authority of the National Archivist is granted explicitly on condition that the Agreement is so respected.

Archival records left in institutions are not considered to be under the control of the National Archives of Canada in terms of the application of the Access to Information and Privacy Acts.

It must also be noted that some institutions refuse to transfer electronic (or other) records to the control of the National Archives of Canada, usually because of access or security or legal considerations. This often involves the institution insisting on a retention period much longer than the National Archives can support. In these cases, the records are also de facto left in government institutions and not transferred to the National Archives, even though this is not a desired situation by the National Archives. Nevertheless, in such situations, the National Archives must recognize reality and seek to protect such records until their eventual transfer.

B. Categories and Examples of Records To Be Left In Institutions

Unlike paper-based records for which only one copy usually exists, it is normal practice for the National Archives to acquire an archival copy or version of electronic records as soon as archival value has been established, and in advance of the expiry of the period of operational use of the record in the creating institution. This practice is to ensure the archival preservation of the information in fragile electronic media. However, it is not always practical or realistic to follow this practice.

The following are categories of electronic records where leaving them in government institutions may be necessary or prudent, with examples given of each case. These categories are not mutually exclusive, and are not to be interpreted by archivists in making their recommendations as blanket or ironclad rules. Rather, these are guidelines designed to help the archivist in the appraisal process, who should at least consider the possibility of not acquiring the records if any of these circumstances pertain. Archivists should, however, justify their recommendation either way (i.e., to acquire or to "leave out") for these categories of records in their Archival Appraisal Reports, so that the National Archivist has clear grounds for making a final decision. As noted above, the last category below is one which the NA does not support, but which it must sometimes de facto recognize.

1. Cumulative, longitudinal systems and records, where by definition no data deletion, erasure, or replacement occurs. Scientific and environmental systems often display this character, for example, wildlife and ocean resource tracking systems, environmental incidents systems, mineral resources inventories, etc. Some social data, such as housing loans or longitudinal labour force employment profiles, may also be cumulative, although less often. Researchers would always approach the creating institution for access to live and current and cumulative system information, rather than trying to recreate the functionality of such systems by mounting flat files and merging into new software the numerous off-loaded annual or other "slices" retained at the NA.
2. Bibliographic or cataloguing systems or records, where the first point of access would never be the National Archives, because, again, the NA would only hold a truncated and incomplete version. Examples include the Crystalline Formulations database at National Research Council, the Canadian Heritage Information Network system housed at Communications, the DOBIS catalogue entries of the National Library, or FEDDOCS at the NA itself.
3. Data where the creating institution has as an operational requirement the provision of extensive and elaborate reference service, and has the resources and the willingness to provide such services to Canadians which for now the National Archives cannot match. The issue here, unlike B.1 and B.2 above, is not the nature of the data or record per se, but rather the level of reference service that can best be provided to Canadians. The Immigration Statistics Division of Employment and Immigration Canada employs several people answering (and creating research data subsets for) some 200 inquiries a month, whereas the National Archives has one-third of one archivist responsible for all three archival functions (including reference) for immigration electronic and textual records. This would be true as well for many of Statistics Canada's holdings and reference activities. In these cases, it is essential that lengthy retention periods be established, in order that NA resources not be overwhelmed with such high levels of reference work, even while acknowledging that access for researchers for some such electronic records left in institutions may not always be easy or economical. The point is that such access to the same records if they were at the National Archives would either be impossible or even less easy or economical.
4. Records where it is not technically feasible or cost effective to acquire a version of the record for National Archives holdings. Certain complex computer systems and software will also fall into this category, although decreasingly so with data interchange and software independent platforms emerging "out there" that increasingly allow the National Archives' own processing system to recreate the original system's functionality in an archival setting. Mega datasystems for atmospheric and weather conditions, however, sometimes cost literally in the billions of dollars and will continue to remain prohibitive for archival transfer consideration. (This B.4 item is different from B.1, although a corollary to it: here in B.4 it is technically or financially impossible to acquire the record; in B.1, the record could be acquired, but because of its cumulative nature, should not be. Of course, there is overlap between the two for some data: weather data systems may be both cumulative in nature and too costly. Others, however, like wildlife monitoring systems, are cumulative (B.1), but neither costly nor technically complex (B.4).)
5. Data where institutions for whatever reasons (security, sensitivity) refuse to transfer the record to the National Archives, at least until the expiry of a lengthy retention period. The National Archives does not encourage or support this situation, unless one or more of the above four circumstances is also present. The PCO senior appointments database is one example. Because the data is threatened in such situations, a more stringent monitoring clause is suggested in the next section of this paper.

C. National Archives Monitoring Of Electronic Records Left In Government Institutions

Where it is decided that electronic records are to be left in the institution, which decision must be justified in the Archival Appraisal Report, the archivist inserts one of the following two clauses (or close adaptation of them) into the formal Terms and Conditions attached to the Agreement, which the institution's MGIH senior official must sign formally indicating the institution's concurrence. The archivist also ensures that the Data Capture Report for the GADFAT database (see Procedure Paper B-12) is completed to reflect that this material "brought forward" at regular, specified intervals to be monitored by the NA to ensure that the records continue not to be threatened and that appropriate preservation standards are enforced:

1. Standard technical clause for Terms and Conditions where the immediate transfer of a cumulative or longitudinal datafile or other record is not desirable (meeting one or more of Categories B.1 to B.4 above):
   
   Preserve the xxx Data System for possible future transfer. The National Archives of Canada shall be notified if there are major changes to the system, including (but not limited to) new hardware environment, new software, significant addition or subtraction in the number of data elements incorporated in the system, significant new internal uses for the system (including migration to other systems), as well as the possible abandonment of the system. If such changes occur, the National Archives of Canada may exercise its option to acquire a copy of the data at that time. The National Archives of Canada will monitor the operation of the system every two years, beginning on (day/month/year).
2. Standard technical clause for Terms and Conditions where the immediate transfer of a non-cumulative datafile or other records is refused by the institution (Categories B.5 above)
   
   The magnetic tapes containing this [xxx data, datafile, etc.] will be stored in a controlled, dust-free environment with a maximum relative humidity of 50%. The magnetic tape must be rewound yearly with a tension of 4 oz./square inch and recopied every five years onto new tape. Provision must be made to keep the data on the magnetic tapes technologically current; if the computer system that created the tapes be changed, the stored magnetic tape must be updated so that it is usable on the new system. Two backup copies should be preserved, one of which must be stored offsite in a Federal Records Centre tape vault or similar facility. The National Archives of Canada will monitor these conditions every two years, beginning on (day/month/year).

Prepared by:	Terry Cook, Records Disposition Division 7 November 1993
Approved by:	Jean-Pierre Wallot, National Archivist of Canada 25 November 1993