Consolidation of postings to the IPIG Mailing List between 1 May to 22 May 1998 under the headings: The Real Question? Bag the Protocol?

Buried at the end of a posting on Fri, 1 May 1998, Barbara Shuh wrote:

Mary Jackson raised the first question on Barb's posting on 10 May 1998:

Art Zeman picked up the issue on 11 May 1998 and the discussion continued for the next two weeks: Thanks for raising this issue! I am also very glad that Mary reposted it. I had skimmed your original message and missed this point back on May 1. You asked for initial reactions as to whether your idea is "REQUIRED, DESIRED, or NEITHER."

Pigasus has done quite a bit of work in this arena and believes that a new interlibrary loan and document delivery protocol which is not backward compatible with ISO 10160/10161 is REQUIRED.

We believe that it would be much more difficult to "fix" ISO 10160/10161 than to revise and simplify the service definition, using all of the lessons learned from years of working with the existing ISO 10160/10161. We base this opinion on the constant stream of topics brought up in IPIG meetings, all of which lead to workarounds, IPIG extensions to the protocol, and arguments about ambiguous semantics in the written specifications and state tables.

When IPIG chooses to delve into this issue, we will gladly share all of the work we have done toward designing a service definition which could, in Barb's words, "rectify the real problem."

Barb answered Mary on 11 May 1998: I think that developers such as Art (who has responded to this message after your highlighting of it) know better than both you or I what the real problems are.

In the discussion that Joe and I were having about DF5, we felt that there were still some grey areas created by the changes proposed in DF (and picked up, in part, by Mike Wheatley's probing questions) - but we were not sure how to resolve them without causing other wrinkles that in turn would cause other problems. In this case, the "real problem" seemed to be that the SHIPPED message was optional for the Requester. That meant that the Processing Phase has to end for the Requester with the RECEIVED state rather with the SHIPPED state, because, in some cases, the Requester may never receive a SHIPPED APDU. Because of this, we are forced to build some work-arounds.

And so my question was: Have the advancements in technology, the changes in the costs of communications obviated the work-arounds and complexity which were built into the protocol? If so, can the ILL service definition and protocol be simplified? What would the costs be to both developers and users? Is it worth it to try? (It would mean another edition of the standard, probably without backward compatibility, and probably in the "post-millenial" timeframe.)

Art said, Yes, it is worth it to try.

Mark Wilson added his support for change: Having implemented most of the thing thrice [don't often get to use that word], I can say that the protocol:

and I would love to be involved.

Randy Menakes contributed a word of caution: I find it profoundly disheartening that we're discussing the obsolete nature of the protocol this week. How have we gotten this far in its development if it won't do the job? I'm sure that any protocol, once codified, is immediately inferior to what we might imagine tomorrow, but to jump ship before the protocol's value has been demonstrated seems to be sending all the wrong messages to the community that Mary Jackson and others have spent the last several years trying to nurture.

The last signal we ought to be sending to members of the ILL community is "This was a respectable first attempt, but we're going to do it right next time". It may be interesting to many to begin thinking what the next ILL protocol might look like. But the IPIG ought not be shutting down the effort and the momentum that seems to be finally coming into its own. I don't think our quarterly meetings can support both efforts - especially ones that are predetermined to be incompatible with each other.

Mark Wilson responded to Randy's concerns: Your points are well taken and I think we all agree with you, except for being disheartened. I personally am much enthused about the work I see going on around me. However, all work is a work in progress.

Although I really would like to see the protocol change substantially to deal with today's workflow and technology, I have no intention of scrapping what I have done so far. What I don't want is for this version to get more and more clumsy and kludgy until it becomes impossible to change or self-destructs.

I do not think that IPIG is the forum for re-writing the standard -- it needs to be a very different kind of working group. What is needed is a careful rework that preserves the thrust of the Canidian original but that also discards the environment that constrained the original. Let's face it, the change from Version 1 to Version 2 was not that major and did not deal with most of the issues that trouble us today in a broader and much changed technical environment.

As a modest proposal, may I suggest that it might be valuable to form a group, official or otherwise, that is well informed as to the lacunae and inclusions that make the current protocol less than it might be and who could at least write a critique that would inform the next version. Now, as we are bringing out real and diverse applications and have strong memories of problems and shortcomings, and in the near future as we deal with inter-operation, is the best time to do such a critique.

Linda Driver gave support to Randy's concerns: I agree with Randy. Six months ago, I might have felt differently. But working with the protocol has made me appreciate some of the features that I once viewed as overly complicated and burdensome. The protocol could be improved--hindsight is always 20/20. But let's work to improve it, not abandon it.

Barb Shuh expresponded to Randy: I must admit that I share some of your concerns and am pleased that you have voiced them. I had a similar discussion with one of my collegues here at NLC only yesterday.

I expect that the re-evaluation of the protocol will be a separate activity from the IPIG work. But I do anticipate that some of the same players will participate in both activities. And like the predictions that Mark Wilson made in one of our early discussions on ILL Directories, it could take another 10 years before it reaches an official status as an approved standard...

I do have the concern that we would be starting looking something new while many implementors are just starting work on the current generation of implementations. But I don't think that we should sit with blinders on and preserve the protocol as it was defined in the 1980's. There are certain complexities in the protocol which are there because of constraints in the computing and telecommunications environment at that time. There have been gigantic advances in these environments - can and should we revise the application standards to reflect this evolution of technology?

From what I heard from Art and MarkW, there is ample room for simplification of the protocol. What I'm not sure what the implications are if we start making wholesale changes. Therefore, I was thinking of possibly having Joe and I work (as the ILL ASMA) through various scenarios (some possible scenarios that come to mind are What would happen if the SHIPPED message were mandatory? What would be the impact of having a confirmed service? What impact will such changes have on the complexity of the ILL-ANSWER service? What else can we work on to simplify the ILL-ANSWER?) to try to gauge the impact of the various changes on the current implementations.

I will freely admit that it frightens me to think of it. But I don't expect that we would be looking at such a revision being approved much before 2005 (by which time the early IPIG implementors will be ready to revamp their products and I will be ready for early retirement!) It's a long time off. But looking at past history, both in the length of time it took to move the first edition of the standard through the ISO approval process and the time that it took me to get a single defect report ready for review, I don't think it unrealistic to start "thinking" about such a change now.

And after we get some basic advise from IPIG, functioning as the ILL ASMA Advisory Group, and from our ISO masters, TC46/SC4/WG4, we'll probably back off and work on a separate, parallel stream, and let the IPIG implementations work their way through implementations that will be good for another 8 years - if we do find that such changes would be warranted.

Just took time to read comments by MarkW and Linda. Mark, we appear to be looking in the same direction. And Linda, I'm interested in your comments about appreciating the features which 6 months ago appeared to you as overly complex and burdensome. 6 months ago, I would was arguing from the other side. But I decided that we should at least ask the question...

Art Zeman then tossed a couple more questions into the ring:

Art's Q1: What would happen if the state machine was not specified as part of the protocol? In other words, what if the protocol simply defined the vocabulary used to communicate ILL & DocDec requests and replies?

Art's Q2: What would happen if the protocol was not BER encoded? (Hey, you knew that I would ask that one, didn't you?)

Art's Q3: How can we accommodate Unicode or other non-Roman characters?

Art's Q4: What changes are necessary to allow communication of a request such as, "I want to borrow the item in 10 days for $5 or buy a photocopy in three days for $12 or send me an estimate of what you could provide?"

Art: If you and Joe can use help in preparing the scenarios, let me know.

Mike Wheatley summarizes his reaction: I have been following this discussion with interest and some concern.

My concern being that others new to reading the list may get the impression that the ILL protocol has severe problems. I don't believe it has. I think it is well fitted for its purpose.

But one must be clear that the protocol is not intended to support all of the functions required in the totality of providing ILL services. It has a set scope - the developers set deliberate boundaries to what it should do.

They deliberately left specific aspects to be covered by other standards then developing (eg information retrieval, directory services, accounting & payment methods, electronic document delivery).

However I do agree with Ruth that the protocol was developed at a time when there was (usually) expected to be a human being involved in the ILL processing. In practice there still usually will be. But it is clear that technology now has the capability of undertaking some of the decision making automatically. Though this requires more complexity within the ILL protocol to enable this to be done.

The scenario raised by Art of a complex set of optional service conditions is one example that requires increased complexity within the protocol for automated action.

The original developers of the standard also left some parameters in unstructured form (but human readable) because it was not going to be possible to reach consensus within a reasonable time.

One example is volume-issue in Item-id. This is simply defined as an ILL-string whereas structuring would assist machine processing. However even the bibliographic world has failed to achieve a consensus on a consistent (widely accepted and used) format for this!

Similarly a parameter such as Search-Type appears to have insufficient capability to tightly define the richness of service offerings from the many different document providers. Enabling it to do so will require a lot of discussion - and will complicate the protocol.

So I agree with others that the ILL Protocol is not broke. It may need improvements.

So let's not " investigate building a new protocol". But we should work to improve the existing ILL protocol as we identify the opportunities. And - yes - that should be for version 3... not making version 2 more clumsy and kludgy (to quote MarkW)

However - remember - that will result in increased complexity not less!

I look forward to seeing suggestions as to how the protocol could be simplified - as I reckon that it will be very difficult to do so without loss of capability, flexibilty and resilience to errors!

Randy thanks Mike Wheatley and others who have encouraged the IPIG to remain focused on the work at hand: I think there comes a crucial time in the development of a major standard (like ISO 10160/10161) when efforts begin to pay off:

This is a time of enormous potential, and I believe the ISO ILL protocol is just entering that period. I personally find that a lot more exciting than imagining what a brand new protocol might look like. I understand and appreciate the developer's drive that motivates Art to say,"the possibilities are interesting enough, and the potential rewards high enough, to investigate building a new protocol". But I don't think it belongs here, as it distracts us from from more serious and more purposeful work.

And hey, Art, there are many areas still begging for your creative energies - just witness next month's agenda. I'm sorry you can't make it this time.

Georgia Brown (OCLC) has the last word (on this go-round..): I have been reading some, but not all of the comments related to the ILL protocol. The comments that concern me are those that relate to bagging the protocol. As others have noted, the definition of the protocol is just the beginning to implementation. In the process of any system implementation, developers find they need to make changes in the intended design as it is impossible to foresee all possible circumstances when doing initial definitions. And developers roll with the changes and rework the designs as needed.

As we implement the protocol we will find changes that are needed. I hope we can work through the implementation changes and continue to use the protocol as a foundation. We have no other standard to fall back on and building and agreeing upon a new standard will take way too long given the market need.

We need to find ways for all our systems to work together and none of us can afford to build separate interfaces to each system. I hope we only resort to a new standard after we've given this one our best shot.