TISDAG meshes: Mesh of Multiple DAG servers - Results from TISDAG

Network Working Group                                          L. Daigle
Request for Comments: 2968                                      T. Eklof
Category: Informational                                     October 2000


           Mesh of Multiple DAG servers - Results from TISDAG

Status of this Memo

   This memo provides information for the Internet community.  It does
   not specify an Internet standard of any kind.  Distribution of this
   memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (2000).  All Rights Reserved.

Abstract

   The Common Indexing Protocol ([CIP1]) is designed to facilitate the
   creation not only of query referral indexes, but also of meshes of
   (loosely) affiliated referral indexes.  The purpose of such a mesh of
   servers is to implement some kind of distributed sharing of indexing
   and/or searching tasks across different servers.  So far, the TISDAG
   (Technical Infrastructure for Swedish Directory Access Gateways)
   project ([TISDAG], [DAGEXP]) has focused on creating a single
   referral index; the obvious next step is to integrate that into a
   larger set of interoperating services.

1. Introduction

1.1 Overview of mesh possibilities

   Two different possibilities are possible for extending the TISDAG
   service to a mesh model (or some combination of both).  First, it
   should be possible to create a mesh of DAG-based services.  Or, it
   might be interesting to use the mesh architecture to incorporate
   access to other types of services (e.g., the Norwegian Directory of
   Directories).  In either case, the basic principle for establishing a
   mesh is that interoperating services should exchange index objects,
   according to the architecture of the mesh (e.g., hierarchical, or
   graph-like, preferably without loops!).

   As is outlined in the CIP documentation ([CIP1]), many possibilities
   exist for mechanisms for creating indexes over multiple referral
   servers -- for example, WDSP index objects could be passed along
   untouched, or a referral index server's contents could be aggregated
   into a new index object, generating referrals back to that server.

   The proposal is that the mesh should be constructed using index
   objects aggregated over participating services' servers.  That is,
   referrals will be generated to other recognized services, not their
   individual participants.  This can be done as a hierarchy or a level
   mesh one-layer deep, but the important reason for not simply passing
   forward index objects (unaggregated) is that individual services may
   support different ranges of access protocols, have particular
   security requirements, etc.  Referrals should be directed to a CAP or
   CAPs -- either the standard ones used by the DAG system, or new ones
   established to support particular semantics of remote systems (e.g.,
   other query types, etc).  Within a given DAG system,  referrals to
   these remote servers will look just like any other referral, although
   a particular SAP or SAPs may be established to provide query
   fulfillment (again, to enable translations between variations of
   service, to allow secure access if the relationship between the
   services is restricted, etc).

   In the following scenarios of mesh traversal, the assumption is that
   the primary service in discussion (Country A in Scenario 1, Country B
   in Scenario 2) is a DAG-based service.  The scenarios are presented
   in the light of interoperating DAG services, but in most cases it
   would be equally applicable if the remote service was provided by
   some other service architecture.  Again, the key element for
   establishing a mesh of any sort is the exchange of the CIP index
   object, not internal system architecture.

1.1.1  Scenario 1:  Top Down

   Suppose 2 countries tie their services together.  A user makes a
   query in Country A.  A certain number of hits are made against the
   index objects of A's WDSPs.  There is also a hit in the aggregate
   index of Country B.  There are 3 possible cases under which this must
   be handled:

   Case 1:

   Country A and Country B are running services that are essentially the
   same -- in terms of protocols, queries, and schema that are
   supported.  In this case, one referral should be generated per
   protocol supported by Country B's service.  The referral can be
   passed back as far as the client, if its protocol supports referrals.
   Alternatively, the CAP may chain the referral through an appropriate
   SAP, in the usual fashion.  In other words, the CAPs of Country B's
   service act as WDSPs to Country A's service.

   Consider the following illustration (only relevant CAPs, SAPs, etc,
   are shown; others suppressed for lack of room):

             +-----------------+
        (1)  |-----+ Country A |     +-------+
      ------>|Prot1|   DAG     |     |A-WSDP1|
      <------| CAP |     +-----|     | Prot1 |
        (2)  |-----+     |Prot1|     +-------+
             |           | SAP |
      ----+  |           +-----|     +-------+
       (3)|  |    +-------+    |     |A-WDSP2|
          |  |    | RI-A  |    |     | Prot1 |
          |  +-----------------+     +-------+
          |
          |                          +-------+
          |                          |A-WDSP3|
          |                          | Prot2 |
          +----------------+         +-------+
                           |          [...]
                           |
                           |         +-----------------+
                           |         |-----+ Country B |     +-------+
                           +-------->|Prot1|   DAG     |     |B-WSDP1|
                                     | CAP |     +-----|     | Prot2 |
                                     |-----+     |Prot1|     +-------+
                                     |           | SAP |
                                     |           +-----|     +-------+
                                     |    +-------+    |     |B-WDSP2|
                                     |    | RI-B  |    |     | Prot1 |
                                     +-----------------+     +-------+
                                                              [...]

   where
      Prot[i] is some particular query protocol
      RI-A has an index over all A-WDSP[i] and RI-B
      RI-B has an index over all B-WDSP[i]
      (1) is the query to the Country A DAG system, which
          yields a referral based on the index object from RI-B
      (2) is that referral
      (3) is the resolution of that referral, which the client takes
          to the Country B DAG system directly (to find out which, if
          any, B-WDSP[i] have relevant information)

   Case 2:

   Country A and Country B are running services that address the same
   service type (e.g., whitepages), but are not using an identical
   collection of protocols, allowed queries, or schema.  The index
   object that Country B sent to Country A's DAG service must be
   constructed in terms of Country A's service, in order for appropriate
   hits to be generated against the index object (i.e. for referrals to
   Country B's service).  However, to resolve the referral, it will be
   necessary to do some further protocol/schema/query mapping.  This can
   be done by a special SAP established within Country A's service, that
   maps Country A's service into the published service of Country B.
   Country A may then elect to support only one of Country B's access
   protocols, and the designated SAP will always contact one type of CAP
   at Country B.

   Alternatively, Country B can establish a particular CAP that does the
   mapping from Country A's service into something that is most
   appropriate against the internal structure of its service.  In this
   case, Country A's referral will be to a special CAP in Country B's
   service (which, again, will look like a WDSP to the Country A
   service); in fact, the referral may be handled directly by the client
   software.  The difference between the two possible approaches lies in
   the responsibility of managing the relationship between the 2 service
   types.  On the one hand, Country A could handle it if it knows its
   service as well as the published access to Country B. On the other,
   Country B could be responsible for establishing a CAP for every
   country that may want to connect to it.  The latter can, in some
   cases, be justified by the amount of internal optimization that can
   be done, and because it reduces the overhead for Country A's service
   (can pass the referral directly back to the client software).

   Consider the following illustration (only relevant CAPs, SAPs, etc,
   are shown; others suppressed for lack of room):

             +-----------------+
        (1)  |-----+ Country A |     +-------+
      ------>|Prot1|   DAG     |     |A-WSDP1|
      <------| CAP |     +-----|     | Prot1 |
        (2)  |-----+     |Prot1|     +-------+
             |           | SAP |
      ----+  |           +-----|     +-------+
       (3)|  |    +-------+    |     |A-WDSP2|
          |  |    | RI-A  |    |     | Prot1 |
          |  +-----------------+     +-------+
          |
          |                          +-------+
          |                          |A-WDSP3|
          |                          | Prot2 |
          +----------------+         +-------+
                           |          [...]
                           |
                           |         +-----------------+
                           |         |-----+ Country B |     +-------+
                           |         |Prot3|   DAG     |     |B-WSDP1|
                           |         | CAP |     +-----|     | Prot3 |
                           |         |-----+     |Prot3|     +-------+
                           |         |---------+ | SAP |
                           |         |Country A| +-----|
                           +-------->|CAP:Prot1|       |
                                     |---------+       |     +-------+
                                     |    +-------+    |     |B-WDSP2|
                                     |    | RI-B  |    |     | Prot3 |
                                     +-----------------+     +-------+
                                                              [...]

   where
      Prot[i] is some particular query protocol
      RI-A has an index over all A-WDSP[i] and RI-B
      RI-B has an index over all B-WDSP[i]
      (1) is the query to the Country A DAG system, which
          yields a referral based on the index object from RI-B
      (2) is that referral
      (3) is the resolution of that referral, which the client takes
          to the Country B DAG system directly, but to a CAP that
          is specifically designed to accommodate protocols from
          Country A's service, and map it (and schema) into Country
          B's service.  Likely, all Country B referrals will be
          chained for the Country A client

   Case 3:

   The third possibility is, in fact, a refinement of the first.  If
   Country A and Country B are running services that are every way
   identical except for the data (WDSPs covered), then it may make sense
   to NOT aggregate Country B's WDSP index objects, but to copy them to
   Country A's server.  Then, Country A's CAPs might be given access to
   the SAPs of Country B in order to carry out chaining directly at the
   remote service (instead of implicating Country A's SAPs and Country
   B's CAPs, as in the first example above).  The answer does not come
   from technology -- it depends entirely on the nature of the
   relationship that can be established between Country A and Country
   B's services.

1.1.2  Scenario 2:  Working Up

   The above scenario implicitly assumes that Country A's server had
   received index objects from Country B's server.  This will be the
   case if Country A's server is higher in the levels of a hierarchy of
   services (established by agreements between the service operators),
   or if the network is comprised of servers that share their index
   objects with all others, for example.  In the latter case, searching
   at any one of the servers in the service yields the full range of
   results -- referrals will be made to any other server that might have
   data that fulfills the user's query.  The sharing of the index
   objects is a mechanism to allow each server to manage local data,
   while enabling distributed load-sharing on the basic query handling.

   However, if a hierarchical, or at least not-completely-connected
   model is used for the server network, queries carried out at a level
   other than the top of the hierarchy, or in one particular branch of
   the hierarchy, will not actually be matched against all index
   objects.  Therefore, there may be other servers to which the query
   should be directed if the full space needs to be searched. Suppose,
   for example, that in the above example Country B is in fact lower in
   the hierarchy than Country A.  A user sending a query to Country B's
   service may be content to limit the scope of the query to that
   country's information (this is true in enough real-life situations
   that this hierarchical relationship becomes an effective mechanism
   for scoping queries and avoiding having to flood the entire network
   with every single query or keep full copies of all data in every
   server).

   Still in theoretical stages, the DAG/IP provides control constructs
   to allow DAG components to act according to the topology of the mesh.
   A CAP might use the "polled-by" system command to establish what
   other servers in the mesh exist in higher levels (and therefore would
   be worth contacting if the scope of the search is to be increased).

   In the example above, a CAP in Country B's system could determine
   that Country A's service was polling Country B, and therefore make it
   a logical target for expanding the scope of the query.  More
   experience (primarily with server mesh topologies) is necessary
   before it will be clear how to best make use of these capabilities:

       .  should the CAP always broaden the scope? only if there are no
          local referrals? under user direction?
       .  should the CAP use a local SAP to contact the remote service's
          CAP?
       .  is it better to completely connect the mesh of servers, or
          produce some kind of hierarchy?
       .  etc

2. Other considerations

   Depending on the context in which a mesh is established (e.g.,
   between national white pages services, or different units of a
   corporate organization, etc), it may be useful to allow individual
   WDSPs to indicate whether they are willing to have their data
   included in a DAG system's aggregated index object (i.e., allowing
   the DAG system to receive referrals from other systems in the mesh).

3. Security Considerations

   This document describes different configurations for sharing
   information between information services.  It introduces no security
   considerations beyond those attendant in (and addressed by)
   particular directory service access protocols.

4. Acknowledgements

   The work described in this document was carried out as part of an on-
   going project of Ericsson.  For further information regarding that
   project, contact:

      Bjorn Larsson
      bjorn.x.larsson@era.ericsson.se

5. Authors' Addresses

   Leslie L. Daigle
   Thinking Cat Enterprises

   EMail:  leslie@thinkingcat.com


   Thommy Eklof
   Hotsip AB

   EMail: thommy.eklof@hotsip.com

6. References

   Request For Comments (RFC) and Internet Draft documents are available
   from numerous mirror sites.

   [CIP1]   Allen, J. and M. Mealling, "The Architecture of the Common
            Indexing Protocol (CIP)", RFC 2651, August 1999.

   [TISDAG] Daigle, L. and R. Hedberg "Technical Infrastructure for
            Swedish Directory Access Gateways (TISDAG)," RFC 2967,
            October 2000.

   [DAGEXP] Eklof, T. and L. Daigle, "Wide Area Directory Deployment
            Experiences", RFC 2969, October 2000.

   [NDD]    Hedberg, R. and H. Alvestrand, "Technical Specification, The
            Norwegian Directory of Directories (NDD)", Work in Progress.

7. Full Copyright Statement

   Copyright (C) The Internet Society (2000).  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.

   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement

   Funding for the RFC Editor function is currently provided by the
   Internet Society.