OSI CONFORMANCE TESTING FOR BIBLIOGRAPHIC APPLICATIONS

Gilbert Arbez and Leigh Swain

Pre-publication draft of an article published in A Special Issue on Open Systems Interconnection. Library Hi Tech. 8:4 (1990), 119-136.

1.0 INTRODUCTION

For the most part, libraries have not been required to play a major role in conformance testing. Most of the impetus for establishing the standards, software tools and services for conformance testing has come from the information technology industry and government, both at the national and supra-national level. All of these participants are keenly aware of the critical role played by conformance testing in ensuring the availability of commercial systems which successfully interoperate. In the U.S., the Corporation for Open Systems (COS), a not-for-profit organization supported by major developers of OSI products, is developing and administering conformance test tools and services for all layers of OSI specifications. ^[1] Similarly in Europe the CTS-WAN programme (conformance testing services in wide-area-network) has initiated OSI testing services on behalf of the Commission of the European Communities (CEC).^{[2] [3]}

UPDATE: By 1997, very little of this infrastructure in North America and in Europe is still in place. COS is no longer in existence, and the well-developed European testing centres have been scaled down since the time of publication of this article.

The emergence of these OSI test centres means that for "generic" OSI standards such as File Transfer, Access and Manipulation (FTAM), Message Handling Systems (MHS), etc, an infrastructure is already in place that provides conformance testing tools and services. Libraries, like other users, can purchase and use OSI products based on these standards with the assurance that system interoperability will be a reality.

However where libraries have had to develop unique, or "niche" standards for bibliographic applications (e.g. interlibrary loan), it also has been necessary to develop the complementary standards and tools for conformance testing.^[4] Because library-specific standards such as the ILL protocol do not have general applicability to other industry sectors, and are therefore not "generic" in this sense, libraries cannot expect the responsibility for testing to be assumed by the conformance test centres within the COS and CTS-WAN infrastructure. Since widespread implementation of OSI standards such as the ILL protocol, particularly by commercial software vendors, is inconceivable without a mechanism for conformance testing, libraries have been forced to develop a mechanism for evaluating adherence to library-related standards by themselves. Once committed to the development of a niche protocol, libraries are also committed to developing and providing the means whereby implementations of such a protocol can be tested for conformity to the protocol specification. The Interlibrary Loan (ILL) protocol^{[5] [6]}, currently in the final stages of the standardization process within the International Standards Organization (ISO), has demanded a high degree of autonomy by libraries in the area of conformance testing. This article will describe how the National Library of Canada has adapted the general concepts and components of OSI conformance testing standards, tools and services to the testing requirements of the ILL protocol.

UPDATE: This represents the status of the ILL protocol (ISO 10161-1) in 1990. The first edition was accepted by ISO in 1991 and finally published by ISO in 1993. The second edition is expected to be published by ISO in 1997.

1.1 RATIONALE FOR CONFORMANCE TESTING

The primary role of conformance testing is to increase the probability of successful interworking (compatibility) between implementations or real (physical) systems. As such, conformance testing benefits both software vendors and users. For the vendors, conformance testing facilities provide support during the implementation process to verify the correctness of a product at each stage of development. As explained in Section 2.2.2, this type of testing, per se, lies outside the domain of OSI conformance testing. However it is a very tangible benefit of OSI conformance testing for systems developers and vendors. For vendors perhaps the most important result of conformance testing is the ability to demonstrate to potential users and clients that their OSI products can interoperate with the products of different suppliers. In concrete terms this takes the form of OSI conformance certificates which are issued by testing laboratories recognized by the software industry.^[7]

For users, conformance testing provides a verification that an OSI product supports a specified standard and is therefore more likely to meet their requirements. Product certification is the visible indication of conformance to a standard and demonstrated ability to interoperate with similar products from other vendors.

To understand the need for such testing one must consider that, typically, systems which incorporate an OSI standard are developed independently of one another by software vendors. Testing in this context is therefore limited to the usual methods of software debugging. In such a scenario, there is a very real possibility that implementations will not achieve successful interoperability^[8] . The primary reason for this is the high degree of optionality that exists within most OSI standards. Consistency in selected options among implementations of a OSI standard is essential to ensure interoperability (see section 3.4).

1.2 THE NATURE OF CONFORMANCE

There are two aspects to requirements for conformance to OSI standards: static and dynamic conformance (Figure 1) ^[9] . Static conformance requirements comprise both broad level constraints on standards such as classification of functional units and options into classes, as well as detailed level constraints such as parameter and timer values. Static requirements define the mimimal capabilities of an implementation in support of interworking. Dynamic conformance requirements comprise permissable behaviour exhibited by an implementation during actual communication. Dynamic conformance defines the maximum capabilities of a implementation conforming to a specific standard.

An integral component of OSI standards is the conformance clauses which set out in an unambiguous manner the functional requirements which must be supported by an implementation which claims conformance to a standard. A conformance clause specifies for a particular standard both the static and dynamic requirements of a conforming implementation. An implementor of an OSI standard provides a written indication, called a protocol implementation conformance statement (PICS), of the capabilities and options that have been implemented or, conversely, omitted. Conformance testing determines whether an implementation satisfies both the static and dynamic requirements as specified in the conformance clause and the PICS.

2.0 CONFORMANCE TESTING CONCEPTS

Since conformance testing measures conformance of an implementation to specific OSI protocol standards, it serves as a bridge between the standards world and the world of implementations. Figure 2 illustrates the major conformance testing activities and standards related to them.

ISO has set out a conformance testing methodology in the five part standard "ISO 9646 - Conformance Testing Methodology and Framework".^[10] This standard has been under development during the past 6 years and has become an ISO international standard this year. Part 1 of the standard, "General Concepts", introduces the conformance testing concepts and the other parts of the standard. Parts 2 to 5 provide guidelines for the two major conformance testing activities: test suite development and conformance testing services.

UPDATE: The latest edition of ISO 9646 was published in 1994, and is now issued in 7 parts.

2.1 TEST SUITE DEVELOPMENT

Test suite development consists of designing and producing specifications for abstract test suites. These are collections of tests, referred to as test cases, designed to exercise all possible implementations of a protocol standard. While an abstract test suite is designed to verify all protocol options, an executable test suite is derived from the abstract test suite to exercise a specific implementation. Test suite development is a standards activity which complements an OSI standard for which conformance testing is required.

Abstract test suites are standardized in the same manner as OSI protocols. These standards facilitate harmonized international testing by ensuring that the conformance testing performed by different bodies are based on the same set of tests.

The creation of OSI test suites is by no means a trivial task considering that protocol standards are essentially partial specifications of products, are relatively complex and often contain extensive optionality to satisfy the requirements of an international user community. The first test suites were created manually by protocol experts, but the use of software tools is becoming more prevalent in an effort to reduce the manpower required to create and maintain test suites.

Part 2 of ISO 9646, "Abstract Test Suite Specification", provides detailed guidelines for developing conformance test suites. It proposes an approach for specifying the test suite structure. Test cases are defined for verifying the capabilities of the implementation and its behaviour under normal and abnormal conditions. Part 2 of ISO 9646 also defines a set of test methods for exercising implementations. Each test method places different requirements on the implementation under test (IUT) and provides different levels of control over the IUT.

ISO 9646 recommends that all standardized test suites be specified in tree and tabular combined notation (TTCN) as defined in ISO 9646 Part 3. Two forms for TTCN have been defined: a graphical form consisting of tables in a human readable form; and, the machine processable (MP) form which allows computer systems to manipulate the test suites. TTCN has been developed to facilitate the specification of test suites for OSI standards.

The National Library of Canada manually developed an abstract test suite for the Canadian Preliminary ILL protocol standard.^[11] Currently, a test suite for the ISO ILL standard is being developed and will be submitted to ISO for standardization. A tool for semi-automatic generation of test cases is being used to develop the test suite.

UPDATE: Although the test suite for the ISO ILL standard was submitted by the National Library of Canada to ISO in the early 1990's, it never progressed through the formal ISO standardization process to become an international standard.

2.1.1 Remote Test Method

ISO 9646 provides a number of test methods for developing abstract test suites. Each test method provides a test architecture which allows different levels of control over the IUT. A test suite developer determines which test method is appropriate for the protocol to be tested. More than one test method may be chosen, in which case, an abstract test suite is specified for each one.

The remote test method is most commonly used. Figure 3 illustrates the view of the ILL implementation during conformance testing by this method. The main objective of conformance testing is to test the ILL protocol functions contained within the IUT. The conformance test facility is the component of the conformance test system which exchanges application protocol data units (APDUs) to measure the conformance of the IUT to the ILL protocol standard. APDUs are the basic units of information exchange between application protocol entities.

Using the remote method, the IUT is tested for its correct operation under normal conditions and under unexpected stimuli (e.g. incorrectly formatted APDUs, or APDUs which are illegal within the context of the current state of the transaction). Conformance testing does not address IUT performance or robustness.

To conform to the ILL protocol standard, an IUT must meet the requirements stipulated in the ILL conformance clause which consists of:

1. static conformance requirements stating the capabilities which the implementation must support for interworking such as services supported, APDUs supported, APDU data types supported, etc.
2. dynamic conformance requirements stating the valid behaviour permitted by the protocol.

During conformance testing of the ILL protocol implementations, the IUT is controlled via two "windows" (see Figure 4) to the protocol entity. This entity represents the portion of the IUT which performs all protocol functions. The first window allows the flow of ILL APDUs with a peer protocol entity represented in the conformance test facility.

The second window involves the ILL service primitives invoked within the IUT. The service primitives represent local actions which invoke protocol functions or are the results of protocol functions for the service user which represents the portion of the IUT which requires the service provided by the protocol entity.

The windows are referred to as points of control and observation (PCO) in ISO 9646. PCOs provide the points at which data can be captured and evaluated during the execution of test cases. For the remote test method, only one PCO exists, that is, the window used to exchange APDUs between the conformance test facility and the IUT.

The ILL protocol specification ^[12] defines the proper APDU exchange procedure between the conformance test facility and the IUT. A finite state machine (FSM) is used to specify the behaviour of the protocol entity. FSMs define a finite set of conditions (states) for the protocol entity and the valid transitions which can occur between these states. The specification also specifies the APDU syntax which defines how data elements within an APDU are delimited such that they can be identified and extracted from the APDU. A data element is the smallest unit of data which can be manipulated within a protocol.

As in the case of the ILL protocol specification, the ILL service definition ^[13] defines the service primitives used to invoke protocol functions in the IUT. It does not specify how the service window is implemented and gives complete freedom to the software developer to specify how the window will function within the implementation.

For the remote test method, the conformance test facility is implemented at a single site remote from the IUT. The greatest advantage of this method is that no modifications are required to the IUT for testing. The disadvantage of this method is that only one PCO is available to the test facility, thereby reducing its control over the IUT. Other test methods provide access to a second PCO to the IUT.

Peer protocol entities communicate by means of an underlying communications service. This same communications service is used for the transfer of APDUs between the conformance test facility and the IUT. By maintaining the conformance test facility remote from the implementation being verified, behaviour of the implementation will more closely resemble that required under actual working conditions.

During the conformance testing process, a predetermined sequence of service primitives is invoked within the IUT. How these service primitives are invoked is a local implementation issue, and no restrictions are placed on this by the remote test method. Each service primitive can be individually invoked by an operator, or the invocation of an entire sequence of service primitives can be performed automatically with no operator intervention. In either case, the conformance test facility will be unaware of how the service primitives were invoked.

Exchange of APDU's are monitored by the conformance test facility which generates results based on this exchange. The test facility will view the IUT as a peer protocol entity and verify the conformance of the IUT by inspecting the APDUs received during testing.

2.1.2 Test Suite Structure

Having selected a test method, the next step in developing an abstract test suite consists of defining its structure. The suite contains the following elements:

Figure 5 illustrates the test suite structure for the ISO ILL protocol. The three major test groups in the suite consist of:

Capability test group:

This test group contains the necessary test groups and test cases for verifying the static conformance requirements of the IUT, that is the minimum capabilities of the implementation, in order to facilitate interworking.

Valid behaviour test group:

This test group will exercise the IUT under normal communications. It verifies the IUT dynamic conformance requirements, that is its observable behaviour permitted by the ILL protocol standard.

Invalid behaviour test group:

This test group consists of testing the IUT's reaction to abnormal behaviour. This reaction should also be in accordance to the standard dynamic conformance requirements.

Another test group not illustrated in the figure is the basic interconnection test group. This group consists of a subset of the capability test group and provides limited testing to establish that there is sufficient conformance for basic interconnection. It is an optional group and not used to measure conformance to the standard.

Each of the above test groups are further subdivided into three groups each representing the roles defined in the ILL protocol: the requester, the responder, and the intermediary. Each of these groups are then further subdivided into groups containing test cases to exercise the following aspects of the IUT:

1. tests focusing on APDUs sent to the IUT,
2. tests focusing on APDUs received from the IUT,
3. tests focusing on relationships between data of APDUs sent and APDUs received,
4. tests related to mandatory capability,
5. tests related to optional capability,
6. variations in values of individual parameters.

2.1.3 Test Generation Software - TTCN Test Generator (TTG)

The generation of abstract test suites can be a time-consuming, tedious, and error-prone process. To facilitate the creation and maintenance of the ISO ILL test suite, the National Library of Canada has developed a software tool, illustrated in Figure 6, for creating TTCN test cases.

Test cases are created with the tool, called the TTCN test generator (TTG), which uses an executable protocol specification and instructions defined by the designer of the test suite. The executable specification refers to software which simulates the behaviour of an ILL protocol implementation. The protocol knowledge base, which consists of a set of PROLOG statements, represents this executable specification.

The TTG provides a set of query functions to allow the test suite designer the possibility of querying the knowledge base. For example, all integer types defined in the ILL APDUS can be listed. Traces through the Finite State Machine (FSM) defined in the ILL protocol can also be verified with the query functions.

The test generation functions provides the means of creating test cases using an English language instruction set. This portion of the TTG can create single test cases or multiple cases with one instruction. For example a set of cases for performing the transition tour of the FSM (each transition in the FSM is executed at least once) is performed automatically by the TTG. The output of these functions are translated to TTCN test cases by the TTCN formatter.

Executable specifications are produced using the logic programming language PROLOG. Since logic programming is declarative rather than procedural, it makes translation of specifications to an executable form a natural and relatively simple process.

The TTG provides the following advantages over the manual process previously used:

1. Since the software tool (TTG) will create the TTCN test cases using a protocol specification, the test suite designer need only be concerned with the purpose of test cases and instruct the TTG to create them. The tool will make the test suite development much less error-prone and time-consuming for the designer.
2. Translation of the protocol standard to an executable form is considerably easier and less time-consuming than creating a test suite for the protocol. For example, translating the 150 page ISO ILL protocol specification into PROLOG and generating the test suite with the TTG would be easier than manually creating the test suite containing an estimated 1000 pages. The TTG will make test suite development a more efficient process.
3. In the future, when a protocol is changed or extended, the test suite designer needs only to change the specification (and possibly instructions) to regenerate the test suite. The TTG will make updating the test suite much more efficient.

This approach relieves the test suite designer of the tedious process of having to specify all units of data in the test suite. The software tool itself will generate the necessary test data using the executable specifications. This approach should result in a considerable reduction of effort in developing and maintaining test suites.

The technique of specifying protocols with logic programming languages has been used successfully in other conformance testing development activities, for example:

1. The Corporation for Open Systems (COS) has used it for developing test cases for the FTAM (File Transfer Access and Management) and Session Layer protocols. COS has used logic programming as a powerful way to produce an executable form of the protocol specification such as the FTAM protocol. In this case, the technique was used to produce over 4000 test cases.
2. IBM European Networking Center has proposed a protocol engineering environment, based on the executable specification of protocols, for use in all aspects of protocol development including conformance testing.^[14] They have experimented with executable specifications and are currently implementing their proposed protocol engineering environment.
3. The University of Ottawa (Canada) developed the prototype Interactive Test Sequence Generator (ITSG) used in 1985/86 with the ILL protocol for the creation of test cases. The University of Ottawa has applied the technique to the transport protocol ^[15] and is currently developing executable specifications of ISDN protocols through a similar technique.

2.2 CONFORMANCE TESTING SERVICES

Standardized test suites are used in conformance testing services to provide the basis for evaluating the conformance of protocol standard implementations. Conformance testing of implementations takes place in the world of vendors and software implementors. Typically, the objective of a conformance testing service is to provide some official indication (certification) that an implementation conforms to an OSI standard. Certification will help the user community evaluate products and provide motivation to vendors to undergo conformance testing.^[16]

UPDATE: In 1997, the authors recommend that, because of problems of liability, there be no official certification process nor a conformance testing service for the ILL protocol implementations developed under the IPIG profile. Rather, the current recommendation to IPIG is to do interoperability testing based on a standard test suite. See Section 4.1 for details on interoperability testing.

The basic tool used for conformance testing is the conformance test system. This software tool executes the abstract test suite and creates test results, thereby exercising an implementation of the protocol standard. The necessary tests for evaluating a specific implementation (with selected options) are derived from the abstract test suite which contains a superset of standard tests.

Most conformance testing services include a formal testing process required to meet certification requirements and an informal debugging activity (not specified by ISO 9646) which permits an implementor to make use of the conformance testing system during the product development cycle.^[17] This process helps reduce the time that would otherwise be needed for formal testing.

Part 4 of ISO 9646, "Test Realization”, provides the requirements for the development of conformance test systems, to ensure that test cases executed reflect the behaviour specified in the abstract test suite. The standard addresses requirements related to abstract testing functions and the proper use of the abstract test suite standard. It considers implementation, acceptance and installation of test systems beyond the scope of the standard.

ISO 9646 Part 5, "Requirements on Test Laboratories and Clients for the Conformance Assessment Process", addresses the roles of both the test laboratory and the client during the conformance assessment process, the need to reach mutual agreement between the parties, and the requirements on each of them. It serves as the basis for developing conformance testing service procedures.

The National Library of Canada is currently preparing a conformance testing service for the ISO ILL protocol. This service is based on a similar service developed for the Canadian ILL preliminary standard. ^[18] This service will be used in Canada to test Canadian implementations of the ILL protocol. As part of this development, the conformance testing system has been ported to a 386 PC running UNIX System V - a platform which renders the software more portable. Current work on the test system will provide X.400 messaging support, EDIFACT and ASN.1 encoding, and other functionality for testing the ISO ILL protocol.

UPDATE: This testing software was successfully used by several Canadian implementors to test conformance of their ILL systems to the protocol. The last series of tests were run in 1994, after enhancements had been made to one of the implementations. However, at present (1997), the National Library of Canada is no longer offering a conformance testing service for the ISO ILL protocol. It is willing to make the testing software freely available for the use of implementors. IPIG is considering using the testing software as a debugging tool and to support interoperability testing.

2.2.1 Conformance Test System

This section focuses on the conformance test system - how it is configured and how it stores information about tests and test data. Figure 7 illustrates the major components of this system which consists of the PETS Generator and the conformance test facility.^[19]

During the test setup, a process generates the parameterized executable test suite (PETS) from the abstract test suite (ATS) using information supplied by the implementor in the PICS and PIXIT.

A PIXIT (protocol implementation extra information for testing) document contains all additional information not defined in the PICS and required for testing such as the library symbols and the address of the electronic mailbox of the implementation being tested. The PIXIT is a proforma which the implementor must complete. All the information contained in the PICS and PIXIT statements is used by the PETS generator to configure the abstract test suite and form the parameterized executable test suite (PETS).

Incoming APDU's from the implementation undergoing testing are compared with those specified in the PETS. The conformance test facility generates APDU's for transmission and verifies incoming APDU's using test data specified in the PETS. The test facility will tag a test case successful if all the test events within the test case were executed successfully. When an APDU is received the following checks are made:

1. verification of APDU syntax.
2. verification of APDU structure according to its ASN.1 specification (although the syntax for a APDU may be correct, there are conditions defined by the ASN.1 specification which can restrict the presence of certain data elements in APDUs).
3. comparison of APDU to expected content (actual data or attribute agreement).

The conformance test facility has also been designed to give the implementor remote control of the Facility for invoking tests and obtaining results. Special messages received over the underlying communications service are interpreted as system commands. This feature is used during the debugging activity of the conformance testing service.

2.2.2 Conformance Assessment Process

The ILL conformance testing service consists of both a formal testing activity and an informal debugging activity. Formal testing is an activity which determines how a specific implementation follows the rules of the protocol. ^[20] The debugging activity permits the implementor to use the conformance test system for evaluating the implementation before the more expensive formal testing is undertaken.

UPDATE: Use of the conformance tests for the purposes of debugging implementations of the ILL protocol is of value whether or not it is followed by formal conformance testing.

Figure 8 illustrates how these two major activities take place in three steps:

1. test setup
2. debugging activity
3. capability/behavior testing and result analysis.

The first and last steps constitute the formal testing activity of the service.

Step 1. Test Setup

The first part of the test setup consists of the conformance requirements review. The test center reviews how options allowed in the protocol standard were integrated in the implementation. This exercise is a paper analysis of the protocol implementation conformance statement, provided by the implementor. With this analysis complete, a test plan is defined. This plan will list the tests to be executed, the manner in which they are executed, and the conditions under which formal testing will occur.

The final part of the test setup is the creation of the PETS and the verification of communications between the test system and the IUT with basic interconnection testing. This latter task also provides an early warning in instances of severe non-conformance to the protocol.

Step 2. Debugging Activity

During the debugging activity, formal testing is temporarily suspended to provide the implementor an opportunity for debugging. The implementor is given access to the test system and allowed to run as many tests as desired. In effect, the implementor is given the freedom to administer the conformance testing in advance and rectify any problems which may arise.

The ISO conformance testing methodology standard does not specify this activity as part of the conformance testing assessment. It has been included in the ILL conformance testing service to provide the following opportunities to an implementor:

1. Execute in advance all conformance tests so that conformance of the implementation to the protocol can be verified before formal testing begins,
2. Reduce the testing effort of the implementor in the development of the implementation.

The implementor need not develop a set of tests for the ILL protocol functions as they are already provided by the conformance testing system. This process should also reduce the formal testing effort. The risk of discovering non-conformance during formal testing will be reduced since the tests will already have been executed.

Step 3. Capability/Behavior Testing and Results Analysis

Formal testing resumes in step 3 under the supervision of the test center. The IUT is put through a series of capability tests, which verify the capability of the IUT as stipulated in the PICS. Then follows a series of behaviour tests designed to examine the behavior of the implementation. The results of this testing is then analyzed to produce two reports: the protocol conformance test report (PCTR) which provides a summary of the results for each test case executed and the system conformance test report (SCTR). The SCTR provides an official statement of conformance and contains a list of standards against which the tests were carried out and their dates of publication. Any instances of non-conformance during the tests or other areas of concern are noted.

3.0 EVOLUTION OF CONFORMANCE TESTING

Section 2.0 provides an introduction to the current state of conformance testing methodology. It is the results of 6 years of standards work in OSI based on world wide efforts in generating test suites, test tools, and testing services.

Although much progress has been made in conformance testing, it is still very much in an evolutionary state. There are additional issues to be considered. Three major addendum items to ISO 9646 are under consideration:

1. protocol profile testing methodology,^[21]
2. multi-party test methods,^[22]
3. formal methods in conformance testing.^[23]

This addenda material is of interest for the testing of bibliographic protocols, particularly the profile testing methodology since profiles for the ILL protocol are being developed. The multi- party test methods are also relevant since the ILL protocol defines multi-party communications between a requester, a responder, and an intermediary.

Other issues related to conformance testing are interoperability testing, arbitration testing, and harmonization of testing services. It is important to define how these types of testing services will complement one another to support the use and proper implementation of OSI protocols.

3.1 Protocol Profile Testing Methodology

A protocol profile contains as a set one or more base standards and the idenfication of the chosen classes, subsets, options and parameters of those base standards. Profiles may be standardized in which case becomes an international standardized profile (ISP).

UPDATE: For more details, see Fay Turner's article "The Interlibrary Loan Protocol: An OSI Solution to ILL Messaging."

Just as conformance testing to a base standard seeks to increase the probability that two implementations of the base standard will interconnect, ISP implementation conformance testing is concerned with increasing the probability that two implementations of the same ISP will be able to interconnect.

ISO has produced a second working draft to define a methodology for profile conformance testing. The current ISO 9646 standard is being extended to include testing profiles. For example, the PICS discussed in section 2.0 becomes the ISPICS (ISP Implementation Conformance Statement). The ISPICS plays the same role as the PICS, that is, provides a statement on how an implementation conforms to an international standard profile.

3.2 Multi-Party Testing

Many OSI protocols such as ISDN, Transaction Processing (TP), Routing, and OSI Management will require testing with multiple parties, that is communication between more than two end to end peers. ISO is progressing a working draft to define the main requirements concerning multi- party test methods. The document also defines a multi-party test architectural model onto which can be mapped ISO 9646 test methods. The development of abstract test suites and test systems can then be based on these configurations.

3.3 Formal Methods in Conformance Testing

ISO has standardized three formal description techniques which can be used for specifying distributed and concurrent systems that use OSI: Estelle (Extended State Transition Language), LOTOS (Language of Temporal Ordering Specification), and SDL (specification and description language). A project has been created in ISO to define a standard for specifying a general methodology on how to perform conformance testing of a protocol implementation given a formal specification of a protocol standard.

4.0 BEYOND CONFORMANCE TESTING

4.1 Interoperability Testing

Conformance testing, no matter how rigorous, does not provide fail-safe assurance that a given protocol implementation will interwork with other implementations, despite their both having successfully undergone testing. Conformance testing primarily determines the conformance of an implementation to a standard.

In contrast, interoperability testing determines the ability of implementations to interwork in an operational environment. Given the complexity of OSI protocol standards and the inherent limits on exhaustive conformance testing, some form of interoperability testing is seen to be a practical requirement. In particular, conformance testing cannot detect incompatible selections of optional protocol parameters. Only interoperability testing can uncover such incompatibilities even when both implementations conform to the same standard. Interoperability testing seeks to increase the probability that real system interworking or interoperability will be attained in operational use.

While conformance testing evaluates an implementation in terms of its adherence to a specific OSI standard, interoperability testing compares an implementation with other products; in effect, software vendors test against one another. The disadvantage of interoperability testing is that each software product must be tested against every other product with which it will interact. The relative priority and advantages of conformance testing and interoperability testing is a controversial issue. ^[24] For some vendors, conformance testing takes a backseat to interoperability testing because the market, particularly the end user market, places a higher value on demonstrable interoperability than on conformance. Conformance to the standard is of less importance to this portion of the OSI market. However there appears to be a consensus emerging that sees both types of testing as complementary activites: conformance testing establishes the fundamental requirement of conformance to an OSI standard and in so doing simplifies the subsequent interoperability, or "inter-product" testing required between vendors' products

The National Library of Canada is currently evaluating, in conjunction with developers implementing the ILL protocol in Canada, the relative requirements of both conformance and interoperability testing. It appears that interoperability testing for ILL will be a feasible and useful complement to conformance testing.

UPDATE: During the testing of the ISO ILL protocol by Canadian developers, interoperability testing for ILL proved to be feasible and useful. Although it may replace the formal conformance testing as a means to test interoperability between systems, coordinators of the Canadian testing service recommend that it be done after systems have been debugged using the conformance test suite and have resolved major problems of conformance against a neutral site.

4.2 Arbitration Testing

Although two implementations have passed conformance testing and interoperability testing, there remains a possibility that the implementations may encounter problems communicating in the field. To resolve this situation, arbitration testing is proposed to identify the incompatibility. In this testing, a test tool which acts as a observer, is placed between the two implementations to monitor their communication without disturbing their exchange. The tool can then analyse the exchange to determine the cause of the communications problem.^[25]

4.3 International Harmonization

Organizations such as the Corporation for Open Systems (COS) in the U.S., the National Computing Center (NCC) in the U.K., the CTS-WAN program of the Commission of European Communities and the Canadian Interest Group on Open Systems (CIGOS) are working to develop or coordinate testing programs and establish test centers or laboratories.^[26] Requirements for international conformance testing "harmonization" must also be taken into account. Harmonized testing is necessary to ensure that products tested in one country will operate in other countries. Harmonization should also eliminate the need for duplicate test centers.

UPDATE: As was mentioned above, these organizations which had been founded in the late 1980's to develop and coordinate testing programs and establish test centres for OSI applications, are no longer operational in 1997.

Any movement toward harmonization of test suites, test procedures and software and test center operations is a significant contribution to the evolution of open systems testing. Ultimately, this will assist in the realization of the goal of open systems and networks for libraries.

5.0 CONCLUSION

Conformance testing is a critical factor in the adoption of OSI standards. For library-specific standards such as the ISO ILL protocol, the burden of developing the test suites, tools and services has fallen to libraries themselves. In the case of the ILL protocol, the National Library of Canada has developed the corresponding test suite, software tools and is providing a conformance test service for Canadian implementors. As part of its commitment to facilitating the migration to OSI standards and technologies by libraries, the National Library will make the test tools and related work available to other test centres. Coordination at an international level is required to achieve harmonization of conformance test activities for the ILL protocol and other library- specific OSI standards. In the case of the ILL protocol it is hoped that the various international projects now underway will lead to a coordinated approach to this challenge.

BIBLIOGRAPHY

Arbez, Gilbert, "Interlibrary Loan (ILL) Conformance Test System." in Canadian Interest Group on Open Systems (CIGOS), Conference Proceedings. (Ottawa, 1988)

Bucciarelli, Paola. "The European Community Conformance Testing Programme (CTS)", in The 5th International Conference on the Application of Standards for Open Systems Interconnection, March 7-9, 1989 (Tokyo, Japan), 173-186.

"Conformance Testing: Here to Stay or Gone Tomorrow". in OPEN: OSI Product and Equipment News. 3:10 (10 May 1990): 7-9.

"CTS-WAN" Iesnews Issue # 17 (August 1988): 21-23.

Davidson, Ian C. and Aminoff, Jan. "Towards a COS Mark Licensing Program." in International Open Systems 88 (Middlesex, UK: Online Publications, UK, 1988)

ISO DIS 9646-1 OSI Conformance Testing Methodology and Framework Part 1: General Concepts, July, 1988

ISO DIS 9646-2 OSI Conformance Testing Methodology and Framework Part Abstract Test Suite Specification, July, 1988

ISO 9646-3 OSI Conformance Testing Methodology and Framework Part 3: The Tree and Tabular Combined Notation

ISO 9646-4 OSI Conformance Testing Methodology and Framework Part 4: Test Realisation, June, 1988

ISO 9646-5 OSI Conformance Testing Methodology and Framework Part 5: Requirements on Test Laboratories and Clients for the Conformance Assessment Process, June, 1988

ISO 10160:1993 Information and Documentation -- Open Systems Interconnection -- Interlibrary Loan Application Service Definition

ISO 10161-1:1993 Information and Documentation -- Open Systems Interconnection -- Interlibrary Loan Application Protocol Specification -- Part 1:Protocol Specification

ISO/IEC JTC 1/SC 21/WG 1: "Formal Methods in Conformance Testing." (Seoul, May 1990)

ISO/IEC JTC 1/SC 21/WG 1: "Revised Working Draft for Multi-Party Test Methods." (Seoul, May, 1990)

ISO/IEC JTC 1/SC 21/WG 1: "Second Working Draft on Protocol Profile Testing Methodology." (Seoul, May 1990)

Jacobsen, Tom. "The European Community is developing Conformance Testing Services". Computer Networks and ISDN Systems 13(1987): 175-179.

MacKinnon, Dennis, William McCrum and Donald Sheppard, An Introduction to Open Systems Communication. New York: W.H. Freeman, 1990.

Neumeier-Mackert, Iris B., Lothar Mackert, and Albert Fleischmann, "A Knowledge-based Protocol Engineering Environment," in Proceedings of the Computer Networking Symposium, Washington, D.C. April 11-13, 1988, 183-191.

Rayner, David, "Harmonising OSI test laboratory operations," in International Open Systems 88. (Middlesex, UK: Online Publications, 1988), 199-210.

Rayner, David, "OSI Conformance Testing", in Forward with Harmonized Conformance Testing Services, (Amsterdam: Commission of European Communities, IOS, 1988), 23-52.

Turner, Fay, "Interlibrary Loan Protocol: Development and Status," in Canadian Library Journal, 45:2 (April 1988): 91-95.

Ural, Hasan and Russell Short, "An Interactive Test Sequence Generator," in Proceedings of ACM SIGCOMM 1986 Symposium. (Stowe, Vermont. August 1986), 241-250.

Voge, Christopher, "Conformance Testing Services for Wide Area Networks," in Forward with Harmonized Conformance Testing Services, (Amsterdam: Commission of European Communities, IOS, 1988), 91-135.

Wiechers, W.K., "IT Certification in Europe," in Forward with Harmonized Conformance Testing Services: Proceedings of the CTS Technical Days, Brussels, October 1-2, 1987. (Amsterdam, IOS, 1988)

GLOSSARY

ABSTRACT TEST SUITE (ATS)

A general test suite which contains all possible tests for verifying all possible options available in a specific protocol.

APPLICATION PROTOCOL DATA UNIT (APDU)

A unit of data which conforms to a protocol standard and which is exchanged between application layer protocol entities.

BASIC INTERCONNECTION TEST GROUP

A group of limited tests to determine whether or not there is sufficient conformance to the relevant protocol for interconnection to be possible, without trying to perform thorough testing.

CAPABILITY TEST GROUP

A group containing the test cases used to determine the capabilities of an IUT.

CERTIFICATION

Certification is the process which results in a legally recognized statement that an implementation has successfully completed conformance testing to OSI standard(s) and therefore can be said to be conformant.

CONFORMANCE TEST FACILITY

The component of the Conformance Test System which executes the executable test cases.

CONFORMANCE TEST SYSTEM

The software which executes the tests defined in the Test Suite and generates test results.

DEBUGGING ACTIVITY

An interval between Basic Interconnection Testing and Capability Testing, during which the implementor is given access to the Conformance Test System and allowed to run as many tests as desired.

DYNAMIC CONFORMANCE REQUIREMENTS

All those requirements (and options) which determine what observable behaviour is permitted by the relevant OSI standard in instances of communication.

FORMAL TESTING ACTIVITY

All activity encompassed within the Static Requirements Review, Basic Interconnection Testing, Capability Testing, Behavior Testing,and the Final Conformance Review. The objective of this activity is to carry out Conformance Testing of an implementation and produce a statement of an IUT's conformance to a protocol standard.

ILL PROTOCOL ENTITY

The portion of application software which performs all ILL protocol functions.

IMPLEMENTATION UNDER TEST (IUT)

The implementation undergoing conformance testing.

INTERLIBRARY LOAN (ILL)

Transactions in library materials are made available by one library to another. It includes the provision of copies as substitute for loans of original materials.

INTERNATIONAL STANDARDIZED PROFILE (ISP)

Documents which define combinations of base standards to provide specific functionalities; and selecting specific options and classes.

INTERWORKING

The ability to exchange meaningful information in support of a distributed processing task.

INVALID BEHAVIOUR TEST GROUP

A test group containing the test cases used to determine the extent to which the dynamic conformance requirements are met by the IUT under abnormal conditions.

PARAMETERIZED EXECUTABLE TEST SUITE (PETS)

The PETS is generated from an abstract test suite (ATS) at the start of the conformance testing of a specific implementation. Because of the options in any protocol standard, an ATS must be modified to include all options supported and to exclude all options not supported by an implementation.

PETS GENERATOR

A software tool which creates a Parameterized Executable Test Suite (PETS) from an Abstract Test Suite (ATS).

POINT OF CONTROL AND OBSERVATION (PCO)

A point defined for a test method, at which the occurrence of test events is controlled and observed, as specified in test cases for that test method.

PROTOCOL CONFORMANCE TEST REPORT (PCTR)

A document written at the end of the conformance assessment process, giving the details of the testing carried out for a particular protocol. It includes the identification of the abstract test cases for whcih corresponding executable test cases were run. It also includes the test purpos(s) and verdict for each test case.

PROTOCOL IMPLEMENTATION CONFORMANCE STATEMENT (PICS)

A statement, provided by an implementor, which specifies the capabilities and options that have been implemented for a specific protocol standard.It provides the basis for determining the tests that apply to a given implementation.

PROTOCOL IMPLEMENTATION EXTRA INFORMATION FOR TESTING (PIXIT)

A statement, provided by an implementor, which specifies additional information about an implementation under test (IUT).

REMOTE TESTING METHOD

A test architecture in which the IUT is remote from the Conformance Test System. The IUT operates under normal conditions without any modification.

RESULT ANALYSIS

The results of Capability and Behavior Testing are analyzed to produce two reports: one which summarizes the results of the testing on a case-by-case basis (PCTR), and the second which provides an official NLC conformance statement of the IUT (SCTR).

SERVICE PRIMITIVE

A local action which invokes a protocol function or results from the invocation of a protocol function.

SERVICE USER

The part of an implementation which requires the services of a protocol entity, e.g. an end user and/or end user application.

STATIC CONFORMANCE REQUIREMENTS

Constraints which are specified in OSI standards to facilitate interworking by defining the requirements for the capabilities of an implementation.

SYSTEM CONFORMANCE TEST REPORT (SCTR)

A document written at the end of the conformance assessment process, giving the overall summary of the conformance of the system to the set of protocols for whcih conformance testing was carried out.

TEST CASES

A specification of the actions required to achieve a specific test purpose.

VALID BEHAVIOUR TEST GROUP

A test group containing the test cases used to determine the extent to which the dynamic conformance requirements are met by the IUT under normal conditions.