4. Client Protocol Operation

This chapter describes the operation of the client portion of VMTP in
terms of the procedures for handling VMTP user events, packet reception
events, management operations and timeout events.  Note that the client
portion of VMTP is separable from the server portion.  It is feasible to
have a node that only implements the client end of VMTP.

To simplify the description, we define a client state record (CSR) plus
some standard utility routines.


4.1. Client State Record Fields

In the following protocol description, there is one client state record
(CSR) per (client,transaction) outstanding message transaction.  Here is
a suggested set of fields.

Link            Link to next CSR when queued in one of the transmission,
                timeout or message queues.

QueuePtr        Pointer to queue head in which this CSR is contained or
                NULL if none.  Queue could be one of transmission queue,
                timeout queue, server queue or response queue.

ProcessIdentification
                The process identification and address space.

Priority        Priority for processing, network service, etc.

State           One of the client states described below.

FinishupFunc    Procedure to be executed on the CSR when it is completes
                its processing in transmission or timeout queues.

TimeoutCount    Time to remain in timeout queue.

TimeoutLimit    User-specified time after which the message transaction
                is aborted. The timeout is infinite if set to zero.

RetransCount    Number of retransmissions since last hearing from the
                Server.

LastTransmitTime
                The time at which the last packet was sent.  This field
                is used to calculate roundtrip times, using the
                RetransmitCount to match the responding packet to a

                particular transmission.  I.e. Response or management
                NotifyVmtpClient operation to Request and a management
                NotifyVmtpServer operation to a Response.

TimetoLive      Time to live to be used on transmission of IP packets.

TransmissionMask
                Bit mask indicating the portions of the segment to
                transmit.  Set before entering the transmission queue
                and cleared incrementally as the 512-byte segment blocks
                of the segment are transmitted.

LocalClientLink Link to next CSR hashing to same hash index in the
                ClientMap.

LocalClient     Entity identifier for client when this CSR is used to
                send a Request packet.

LocalTransaction
                Transaction identifier for current message transaction
                the local client has outstanding.

LocalPrincipal  Account identification, possibly including key and key
                timeout.

LocalDelivery   Bit mask of segment blocks that have not been
                acknowledged in the Request or have been received in the
                Response, depending on the state.

ResponseQueue   Queue of CSR's representing the queued Responses for
                this entity.

VMTP Header     Prototype VMTP header, used to generate and store the
                header portion of a Request for transmission and
                retransmission on timeout.

SegmentDesc     Description of the segment data associated with the CSR,
                either the area storing the original Request data, the
                area for receiving Request data, or the area storing the
                Response data that is returned.

HostAddr        The network or internetwork host address to which the
                Client last transmitted.  This field also indicates the
                type of the address, e.g. IP, Ethernet, etc.

Note: the CSR can be combined with a light-weight process descriptor
with considerable benefit if the process is designed to block when it

issues a message transaction.  In particular, by combining the two
descriptors, the implementation saves time because it only needs to
locate and queue one descriptor with various operations (rather than
having to locate two descriptors).  It also saves space, given that the
VMTP header prototype provides space such as the user data field which
may serve to store processor state for when the process is preempted.
Non-preemptive blocking can use the process stack to store the processor
state so only a program counter and stack pointer may be required in the
process descriptor beyond what we have described.  (This is the approach
used in the V kernel.)


4.2. Client Protocol States

A Client State Record records the state of message transaction generated
by this host, identified by the (Client, Transaction) values in the CSR.
As a client originating a transaction, it is in one of the following
states.

AwaitingResponse
                Waiting for a Response packet group to arrive with the
                same (Client,Transaction) identification.

ReceivingResponse
                Waiting for additional packets in the Response packet
                group it is currently receiving.

"Other"         Not waiting for a response, which can be Processing or
                some other operating system state, or one of the Server
                states if it also acts as a server.

This covers all the states for a client.


4.3. State Transition Diagrams

The client state transitions are illustrated in Figure 4-1.  The client
goes into the state AwaitingResponse on sending a request unless it is a
datagram request.  In the AwaitingResponse state, it can timeout and
retry and eventually give up and return to the processing state unless
it receives a Response.  (A NotifyVmtpClient operation resets the
timeout but does not change the state.)  On receipt of a single packet
response, it returns to the processing state.  Otherwise, it goes to
ReceivingResponse state.  After timeout or final response packet is
received, the client returns to the processing state.  The processing
state also includes any other state besides those associated with
issuing a message transaction.

   +------------+
   | Processing |<--------------------|
   |            |<-------------|      |
   |            |<---|         |      |
   +|------^--^-+  Single    Last     |
 Transmit  |  |    Packet    Response |
    |      |  |    Response  Packet   |
    |      |  |      |         |      |
    +-DGM->+ Timeout |         |   Final timeout
    |         |      |         |      |
   +V-----------+    |       +-----------+
   |  Awaiting  |----+       | Receiving |->Response-+
   |  Response  |->Response->| Response  |           |
   |            |  (multi-   |           |<----------+
   +-|--------^-+   packet)  +----------^+
     V        |                |        |
     +-Timeout+                +>Timeout+

                 Figure 4-1:   Client State Transitions


4.4. User Interface

The RPC or user interface to VMTP is implementation-dependent and may
use systems calls, functions or some other mechanism.  The list of
requests that follow is intended to suggest the basic functionality that
should be available.

Send( mcb, timeout, segptr, segsize )
                Initiate a message transaction to the server and request
                message specified by mcb and return a response in mcb,
                if it is received within the specified timeout period
                (or else return USER_TIMEOUT in the Code field).  The
                segptr parameter specifies the location from which the
                segment data is sent and the location into which the
                response data is to be delivered.  The segsize field
                indicates the maximum length of this area.

GetResponse( responsemcb, timeout, segptr, segsize )
                Get the next response sent to this client as part of the
                current message transaction, returning the segment data,
                if any, into the memory specified by segptr and segsize.

This interface assumes that there is a client entity associated with the
invoking process that is to be used with these operations.  Otherwise,
the client entity must be specified as an additional parameter.

4.5. Event Processing

The following events may occur in the VMTP client:

   - User Requests

        * Send

        * GetResponse

   - Packet Arrival

        * Response Packet

        * Request

     The minimal Client implementation handles Request packets for
     its VMTP management (server) module and sends NotifyVmtpClient
     requests in response to others, indicating the specified
     server does not exist.

   - Management Operation - NotifyVmtpClient

   - Timeouts

        * Client Retransmission Timeout

The handling of these events is described in detail in the following
subsections.

We first describe some conventions and procedures used in the
description.  A field of the received packet is indicated as (for
example) p.Transaction, for the Transaction field.  Optional portions of
the code, such as the streaming handling code are prefixed with a "|" in
the first column.

MapClient( client )
                Return pointer to CSR for client with the specified
                clientId, else NULL.

SendPacketGroup( csr )
                Send the packet group (Request, Response) according to
                that specified by the CSR.

NotifyClient( csr, p, code )
                Invoke the NotifyVmtpClient operation with the
                parameters csr.RemoteClient, p.control,

                csr.ReceiveSeqNumber, csr.RemoteTransaction and
                csr.RemoteDelivery, and code.  If csr is NULL, use
                p.Client, p.Transaction and p.PacketDelivery instead and
                the global ReceiveSequenceNumber, if supported.  This
                function simplifies the description over calling
                NotifyVmtpClient directly in the procedural
                specification below.  (See Appendix III.)

NotifyServer( csr, p, code )
                Invoke the NotifyVmtpServer operation with the
                parameters p.Server, csr.LocalClient,
                csr.LocalTransaction, csr.LocalDelivery and code.  Use
                p.Client, P.Transaction and 0 for the clientId, transact
                and delivery parameters if csr is NULL.  This function
                simplifies the description over calling NotifyVmtpServer
                directly in the procedural specification below.  (See
                Appendix III.)

DGMset(p)       True if DGM bit set in packet (or csr) else False.
                (Similar functions are used for other bits.)

Timeout( csr, timeperiod, func )
                Set or reset timer on csr record for timeperiod later
                and invoke func if the timeout expires.


4.6. Client User-invoked Events

A user event occurs when a VMTP user application invokes one of the VMTP
interface procedures.


4.6.1. Send

Send( mcb, timeout, segptr, segsize )
    map to main CSR for this client.
    increment csr.LocalTransaction
    Init csr and check parameters and segment if any.
    Set SDA if sending appended data.
    Flush queued replies from previous transaction, if any.
    if local non-group server then
        deliver locally
        await response
        return
    if GroupId(server) then
        Check for and deliver to local members.
        if CRE request and non-group local CR entity then

           await response
           return
        endif
        set MDG if member of this group.
    endif
    clear csr.RetransCount
    set csr.TransmissionMask
    set csr.TimeLimit to timeout
    set csr.HostAddr for csr.Server
    SendPacketGroup( csr )
    if DGMset(csr) then
       return
    endif
    set csr.State to AwaitingResponse
    Timeout( rootcsr, TC1(csr.Server), LocalClientTimeout )
    return
end Send

Notes:

   1. Normally, the HostAddr is extracted from the ServerHost
      cache, which maps server entity identifiers to host
      addresses.  However, on cache miss, the client first queries
      the network using the ProbeEntity operation, as specified in
      Appendix III, determining the host address from the Response.
      The ProbeEntity operation is handled as a separate message
      transaction by the Client.

The stream interface incorporates a parameter to pass a responseHandler
procedure that is invoked when the message transaction completes.

StreamSend( mcb, timeout, segptr, segsize, responseHandler )
    map to main CSR for this client.
|   Allocate a new csr if root in use.
|   lastcsr := First csr for last request.
|   if STIset(lastcsr)
|       csr.LocalTransaction := lastcsr.LocalTransaction + 256
|   else
|       csr.LocalTransaction := lastcsr.LocalTransaction + 1
    Init csr and check parameters and segment if any.
    . . . ( rest is the same as for the normal Send)

Notes:

   1. Each outstanding message transaction is represented by a CSR
      queued on the root CSR for this client entity.  The root CSR
      is used to handle timeouts, etc.  On timeout, the last packet

      from the last packet group is retransmitted (with or without
      the segment data).


4.6.2. GetResponse

GetResponse( req, timeout, segptr, segsize )
    csr := CurrentCSR;
    if responses queued then return next response
      (in req, segptr to max of segsize )
    if timeout is zero then return KERNEL_TIMEOUT error
    set state to AWAITING_RESPONSE
    Timeout( csr, timeout, ReturnKernelTimeout );
end GetResponse

Notes:

   1. GetResponse is only used with multicast Requests, which is
      the only case in which multiple (different) Responses should
      be received.

   2. A response must remain queued until the next message
      transaction is invoked to filter out duplicates of this
      response.

   3. If the response is incomplete (only relevant if a
      multi-packet response), then the client may wait for the
      response to be fully received, including issuing requests for
      retransmission (using NotifyVmtpServer operations) before
      returning the response.

   4. As an optimization, a response may be stored in the CSR of
      the client.  In this case, the response must be transferred
      to a separate buffer (for duplicate suppression) before
      waiting for another response.  Using this optimization, a
      response buffer is not allocated in the common case of the
      client receiving only one response.


4.7. Packet Arrival

In general, on packet reception, a packet is mapped to the client state
record, decrypted if necessary using the key in the CSR.  It then has
its checksum verified and then is transformed to the right byte order.
The packet is then processed fully relative to its packet function code.
It is discarded immediately if it is addressed to a different domain
than the domain(s) in which the receiving host participates.

For each of the 2 packet types, we assume a procedure called with a
pointer p to the VMTP packet and psize, the size of the packet in
octets.  Thus, generic packet reception is:

if not LocalDomain(p.Domain) then return;

csr := MapClient( p.Client )

if csr is NULL then
    HandleNoCsr( p, psize )
    return

if Secure(p) then
    if SecureVMTP not supported then
        { Assume a Request. }
        if not Multicast(p) then
            NotifyClient(NULL, p, SECURITY_NOT_SUPPORTED )
        return
    endif
|   Decrypt( csr.Key, p, psize )

if p.Checksum not null then
    if not VerifyChecksum(p, psize) then return;
if OppositeByteOrder(p) then ByteSwap( p, psize )
if psize not equal sizeof(VmtpHeader) + 4*p.Length then
    NotifyClient(NULL, p, VMTP_ERROR )
    return
Invoke Procedure[p.FuncCode]( csr, p, psize )
Discard packet and return

Notes:

   1. The Procedure[p.FuncCode] refers to one of the 2 procedures
      corresponding to the two different packet types of VMTP,
      Requests and Responses.

   2. In all the following descriptions, a packet is discarded on
      "return" unless otherwise stated.

   3. The procedure HandleNoCSR is a management routine that
      allocates and initializes a CSR and processes the packet or
      else sends an error indication to the sender of the packet.
      This procedure is described in greater detail in Section
      4.8.1.

4.7.1. Response

This procedure handles incoming Response packets.

HandleResponse( csr, p, psize )
    if not LocalClient( csr ) then
        if Multicast then return
|       if Migrated( p.Client ) then
|           NotifyServer(csr, p ENTITY_MIGRATED )
|       else
            NotifyServer(csr, p, ENTITY_NOT_HERE )
        return
    endif

    if NSRset(p) then
        if Streaming not supported then
            NotifyServer(csr, p, STREAMING_NOT_SUPPORTED )
            return STREAMED_RESPONSE
|       Find csr corresponding to p.Transaction
|       if none found then
|           NotifyServer(csr, p, BAD_TRANSACTION_ID )
|           return
     else
      if csr.LocalTransaction not equal p.Transaction then
        NotifyServer(csr, p, BAD_TRANSACTION_ID )
        return
    endif
    Locate reply buffer rb for this p.Server
    if found then
        if rb.State is not ReceivingResponse then
          { Duplicate }
            if APGset(p) or NERset(p) then
                { Send Response to stop response packets. }
                NotifyServer(csr, p, RESPONSE_DISCARDED )
            endif
            return
         endif
         { rb.State is ReceivingRequest}
         if new segment data then retain in CSR segment area.
         if packetgroup not complete then
             Timeout( rb, TC3(p.Server), LocalClientTimeout )
             return;
          endif
          goto EndPacketGroup
    endif
    { Otherwise, a new response message. }

    if (NSRset(p) or NERset(p)) and NoStreaming then
        NotifyServer(csr, p, VMTP_ERROR )
        return
|    if NSRset(p) then
|      { Check consecutive with previous packet group }
|       Find last packet group CSR from p.Server.
|       if p.Transaction not
|             lastcsr.RemoteTransaction+1 mod 2**32 then
|         { Out of order packet group }
|            NotifyServer(csr, p, BAD_TRANSACTION_ID)
|           return
|       endif
|       if lastcsr not completed then
|           NotifyServer(lastcsr, p, RETRY )
|       endif
|       if CMG(lastcsr) then
|           Add segment data to lastcsr Response
|           Notify lastcsr with new packet group.
|           Clear lastcsr.VerifyInterval
|       else
|           if lastcsr available then
|                 use it for this packet group
|           else allocate and initialize new CSR
|           Save message and segment data in new CSR area.
|       endif
|   else { First packet group }
        Allocate and init reply buffer rb for this response.
        if allocation fails then
            NotifyServer(csr, p, BUSY )
            return
        Set rb.State to ReceivingResponse
        Copy message and segment data to rb's segment area
         and set rb.PacketDelivery to that delivered.
        Save p.Server host address in ServerHost cache.
    endif
    if packetgroup not complete then
        Timeout( rb, TS1(p.Client), LocalClientTimeout )
        return;
    endif
endPacketGroup:
    { We have received last packet in packet group. }
    if APGset(p) then NotifyServer(csr, p, OK )
|   if NERset(p) and CMGset(p) then
|       Queue waiting for continuation packet group.
|       Timeout( rb, TC2(rb.Server), LocalClientTimeout )
|       return
|   endif

    { Deliver response message. }
    Deliver response to Client, or queue as appropriate.
end HandleResponse

Notes:

   1. The mechanism for handling streaming is optional and can be
      replaced with the tests for use of streaming.  Note that the
      server should never stream at the Client unless the Client
      has streamed at the Server or has used the STI control bit.
      Otherwise, streamed Responses are a protocol error.

   2. As an optimization, a Response can be stored into the CSR for
      the Client rather than allocating a separate CSR for a
      response buffer.  However, if multiple responses are handled,
      the code must be careful to perform duplicate detection on
      the Response stored there as well as those queued.  In
      addition, GetResponse must create a queued version of this
      Response before allowing it to be overwritten.

   3. The handling of Group Responses has been omitted for brevity.
      Basically, a Response is accepted if there has been a Request
      received locally from the same Client and same Transaction
      that has not been responded to.  In this case, the Response
      is delivered to the Server or queued.

4.8. Management Operations

VMTP uses management operations (invoked as remote procedure calls) to
effectively acknowledge packet groups and request retransmissions.  The
following routine is invoked by the Client's management module on
request from the Server.

NotifyVmtpClient( clientId,ctrl,receiveSeqNumber,transact,delivery,code)
    Get csr for clientId
    if none then return
    if RemoteClient( csr ) and not NotifyVmtpRemoteClient then
       return
|   else (for streaming)
|      Find csr with same LocalTransaction as transact
|      if csr is NULL then return
    if csr.State not AwaitingResponse then return
    if ctrl.PGcount then ack previous packet groups.
    select on code
      case OK:
        Notify ack'ed segment blocks from delivery
        Clear csr.RetransCount;
        Timeout( csr, TC1(csr.Server), LocalClientTimeout )
        return
      case RETRY:
        Set csr.TransmissionMask to missing segment blocks,
            as specified by delivery
        SendPacketGroup( csr )
        Timeout( csr, TC1(csr.Server), LocalClientTimeout )
      case RETRY_ALL
        Set csr.TransmissionMask to retransmit all blocks.
        SendPacketGroup( csr )
        Timeout( csr, TC1(csr.Server), LocalClientTimeout )
|       if streaming then
|          Restart transmission of packet groups,
|                starting from transact+1
         return
      case BUSY:
         if csr.TimeLimit exceeded then
             Set csr.Code to USER_TIMEOUT
             return Response to application
             return;
        Set csr.TransmissionMask for full retransmission
        Clear csr.RetransCount
        Timeout( csr, TC1(csr.Server), LocalClientTimeout )
        return
      case ENTITY_MIGRATED:
        Get new host address for entity

        Set csr.TransmissionMask for full retransmission
        Clear csr.RetransCount
        SendPacketGroup( csr )
        Timeout( csr, TC1(csr.Server), LocalClientTimeout )
        return

      case STREAMING_NOT_SUPPORTED:
        Record that server does not support streaming
        if CMG(csr) then forget this packet group
        else resend Request as separate packet group.
        return
      default:
         Set csr.Code to code
         return Response to application
         return;
    endselect
end NotifyVmtpClient

Notes:

   1. The delivery parameter indicates the segment blocks received
      by the Server.  That is, a 1 bit in the i-th position
      indicates that the i-th segment block in the segment data of
      the Request was received.  All subsequent NotifyVmtpClient
      operations for this transaction should be set to acknowledge
      a superset of the segment blocks in this packet.  In
      particular, the Client need not be prepared to retransmit the
      segment data once it has been acknowledged by a Notify
      operation.


4.8.1. HandleNoCSR

HandleNoCSR is called when a packet arrives for which there is no CSR
matching the client field of the packet.

HandleNoCSR( p, psize )
    if Secure(p) then
        if SecureVMTP not supported then
            { Assume a Request }
            if not Multicast(p) then
                NotifyClient(NULL,p,SECURITY_NOT_SUPPORTED)
            return
        endif
        HandleRequestNoCSR( p, psize )
        return
    endif

    if p.Checksum not null then
        if not VerifyChecksum(p, psize) then return;
    if OppositeByteOrder(p) then ByteSwap( p, psize )
    if psize not equal sizeof(VmtpHeader) + 4*p.Length then
        NotifyClient(NULL, p, VMTP_ERROR )
        return

    if p.FuncCode is Response then
|        if Migrated( p.Client ) then
|           NotifyServer(csr, p ENTITY_MIGRATED )
|       else
            NotifyServer(csr, p, NONEXISTENT_ENTITY )
        return
    endif

    if p.FuncCode is Request then
       HandleRequestNoCSR( p, psize )
    return
end HandleNoCSR

Notes:

   1. The node need only check to see if the client entity has
      migrated if in fact it supports migration of entities.

   2. The procedure HandleRequestNoCSR is specified in Section
      5.8.1.  In the minimal client version, it need only handle
      Probe requests and can do so directly without allocating a
      new CSR.

4.9. Timeouts

A client with a message transaction in progress has a single timer
corresponding to the first unacknowledged request message.  (In the
absence of streaming, this request is also the last request sent.)  This
timeout is handled as follows:

LocalClientTimeout( csr )
  select on csr.State
    case AwaitingResponse:
      if csr.RetransCount > MaxRetrans(csr.Server) then
             terminate Client's message transactions up to
             and including the current message transaction.
             set return code to KERNEL_TIMEOUT
          return
      increment csr.RetransCount
      Resend current packet group with APG set.
      Timeout( csr, TC2(csr.Server), LocalClientTimeout )
      return
    case ReceivingResponse:
      if DGMset(csr) or csr.RetransCount > Max then
         if MDMset(csr) then
            Set MCB.MsgDeliveryMask to blocks received.
         else
            Set csr.Code to BAD_REPLY_SEGMENT
         return to user Client
      endif
      increment csr.RetransCount
      NotifyServer with RETRY
      Timeout( csr, TC3(csr.Server), LocalClientTimeout )
      return
  end select
end LocalClientTimeout

Notes:

   1. A Client can only request retransmission of a Response if the
      Response is not idempotent.  If idempotent, it must
      retransmit the Request.  The Server should generally support
      the MsgDeliveryMask for Requests that it treats as idempotent
      and that require multi-packet Responses.  Otherwise, there is
      no selective retransmission for idempotent message
      transactions.

   2. The current packet group is the last one transmitted.  Thus,
      with streaming, there may be several packet groups
      outstanding that precede the current packet group.

   3. The Request packet group should be retransmitted without the
      segment data, resulting in a single short packet in the
      retransmission.  The Server must then send a
      NotifyVmtpClient with a RETRY or RETRY_ALL code to get the
      segment data transmitted as needed.  This strategy minimizes
      the overhead on the network and the server(s) for
      retransmissions.

5. Server Protocol Operation

This section describes the operation of the server portion of the
protocol in terms of the procedures for handling VMTP user events,
packet reception events and timeout events.  Each server is assumed to
implement the client procedures described in the previous chapter.
(This is not strictly necessary but it simplifies the exposition.)


5.1. Remote Client State Record Fields

The CSR for a server is extended with the following fields, in addition
to the ones listed for the client version.

RemoteClient    Identifier for remote client that sent the Request that
                this CSR is handling.

RemoteClientLink
                Link to next CSR hashing to same hash index in the
                ClientMap.

RemoteTransaction
                Transaction identifier for Request from remote client.

RemoteDelivery  The segment blocks received so far as part of a Request
                or yet to be acknowledged as part of a Response.

VerifyInterval  Time interval since there was confirmation that the
                remote Client was still valid.

RemotePrincipal Account identification, possibly including key and key
                timeout for secure communication.


5.2. Remote Client Protocol States

A CSR in the server end is in one of the following states.

AwaitingRequest Waiting for a Request packet group.  It may be marked as
                waiting on a specific Client, or on any Client.

ReceivingRequest
                Waiting to receive additional Request packets in a
                multi-packet group Request.

Responded       The Response has been sent and the CSR is timing out,
                providing duplicate suppression and retransmission (if

                the Response was not idempotent).

ResponseDiscarded
                Response has been acknowledged or has timed out so
                cannot be retransmitted.  However, duplicates are still
                filtered and CSR can be reused for new message
                transaction.

Processing      Executing on behalf of the Client.

Forwarded       The message transaction has been forwarded to another
                Server that is to respond directly to the Client.


5.3. State Transition Diagrams

The CSR state transitions in the server are illustrated in Figure 5-1.
The CSR generally starts in the AwaitingRequest state.  On receipt of a
Request, the Server either has an up-to-date CSR for the Client or else
it sends a Probe request (as a separate VMTP message transaction) to the
VMTP management module associated with the Client.  In the latter case,
the processing of the Request is delayed until a Response to the Probe
request is received.  At that time, the CSR information is brought up to
date and the Request is processed.  If the Request is a single-packet
request, the CSR is then set in the Processing state to handle the
request.  Otherwise (a multi-packet Request), the CSR is put into the
ReceivingResponse state, waiting to receive subsequent Request packets
that constitute the Request message.  It exits the ReceivingRequest
state on timeout or on receiving the last Request packet.  In the former
case, the request is delivered with an indication of the portion
received, using the MsgDelivery field if MDM is set.  After request
processing is complete, either the Response is sent and the CSR enters
the Responded state or the message transaction is forwarded and the CSR
enters the Forwarded state.

In the Responded state, if the Response is not marked as idempotent, the
Response is retransmitted on receipt of a retransmission of the
corresponding Request, on receipt of a NotifyVmtpServer operation
requesting retransmission or on timeout at which time APG is set,
requesting an acknowledgment from the Client.  The Response is
retransmitted some maximum number of times at which time the Response is
discarded and the CSR is marked accordingly.  If a Request or a
NotifyVmtpServer operation is received expecting retransmission of the
Response after the CSR has entered the ResponseDiscarded state, a
NotifyVmtpClient operation is sent back (or invoked in the Client
management module) indicating that the response was discarded unless the
Request was multicast, in which case no action is taken.  After a

     (Retransmit Forwarded Request and NotifyVmtpClient)
                    Request/
                    Ack/
                   +Timeout+
                   V       |
                 +-|-------^-+
                 |           |
          +-Time-| Forwarded |<-------------+
          |  out +-----------+              |
          |                                 |
          |          (Retransmit Response)  |
          |                      Request    |
          V                      Ack        |
          |                    +-Timeout-+  |
          |                    V         |  |
        +---------+ Ack/ +|---------^+ |
 +-Time-|Response |<-Timeout--| Responded | |
 |  out |Discarded|           +----^------+ |
 |      +---------+                |        |
 |  +------------+                 |        |
 |  |            |->-Send Response-+        |
 |  |            |->-forward Request--------+
 +->| Processing |<----------------------+
 |  |            |<----------------+     |
 |  |            |<---|            |     |
 |  +-|--------^-+    |          Last    |
 | Receive     |      |          Request |
 |    |   Timeout   Single       Packet  |
 |    |        |    Packet         |   Timeout
 |    |        |    Request        ^     ^
 |    |        |      ^           +|-----|--+
 |  +-V--------|-+    |           |Receiving|<-+Time
 +->|  Awaiting  |->--+->Request->| Request |--+ out
    |  Request   |    |  (multi-  +---------+
    +------|-----+    ^  packet)
        Request       |
           |        Response
      Send Probe     to
           |        Probe
       +---V----+     |
       |Awaiting|     ^
       |Response|-->--+
       |to Probe|
       +--------+

             Figure 5-1:   Remote Client State Transitions

timeout corresponding to the time required to filter out duplicates, the

CSR returns either to the AwaitingRequest state or to the Processing
state.  Note that "Ack" refers to acknowledgment by a Notify operation.

A Request that is forwarded leaves the CSR in the Forwarded state.  In
the Forwarded state, the forwarded Request is retransmitted
periodically, expecting NotifyRemoteClient operations back from the
Server to which the Request was forwarded, analogous to the Client
behavior in the AwaitingResponse state.  In this state, a
NotifyRemoteClient from this Server acknowledges the Request or asks
that it be retransmitted or reports an error.  A retransmission of the
Request from the Client causes a NotifyVmtpClient to be returned to the
Client if APG is set.  The CSR leaves the Forwarded state after timing
out in the absence of NotifyRemoteClient operations from the forward
Server or on receipt of a NotifyRemoteClient operation indicating the
forward Server has sent a Response and received an acknowledgement.  It
then enters the ResponseDiscarded state.

Receipt of a new Request from the same Client aborts the current
transaction, independent of its state, and initiates a new transaction
unless the new Request is part of a run of message transactions.  If it
is part of a run of message transactions, the handling follows the state
diagram except the new Request is not Processed until there has been a
response sent to the previous transaction.


5.4. User Interface

The RPC or user interface to VMTP is implementation-dependent and may
use systems calls, functions or some other mechanism.  The list of
requests that follow is intended to suggest the basic functionality that
should be available.

AcceptMessage( reqmcb, segptr, segsize, client, transid, timeout ) 
                Accept a new Request message in the specified reqmcb
                area, placing the segment data, if any, in the area
                described by segptr and segsize.  This returns the
                Server in the entityId field of the reqmcb and actual
                segment size in the segsize parameters.  It also returns
                the Client and Transaction for this message transaction
                in the corresponding parameters.  This procedure
                supports message semantics for request processing.  When
                a server process executes this call, it blocks until a
                Request message has been queued for the server.
                AcceptMessage returns after the specified timeout period
                if a message has not been received by that time.

RespondMessage( responsemcb, client, transid, segptr ) 

                Respond to the client with the specified response
                message and segment, again with message semantics.

RespondCall( responsemcb, segptr ) 
                Respond to the client with the specified response
                message and segment, with remote procedure call
                semantics.  This procedure does not return.  The
                lightweight process that executes this procedure is
                matched to a stack, program counter, segment area and
                priority from the information provided in a
                ModifyService call, as specified in Appendix III.

ForwardMessage( requestmcb, transid, segptr, segsize, forwardserver ) 
                Forward the client to the specified forwardserver with
                the request specified in mcb.

ForwardCall( requestmcb, segptr, segsize, forwardserver ) 
                Forward the client transaction to the specified
                forwardserver with the request specified by requestmcb.
                This procedure does not return.

GetRemoteClientId()
                Return the entityId for the remote client on whose
                behave the process is executing.  This is only
                applicable in the procedure call model of request
                handling.

GetForwarder( client )
                Return the entity that forwarded this Request, if any.

GetProcess( client )
                Return an identifier for the process associated with
                this client entity-id.

GetPrincipal( client )
                Return the principal associated with this client
                entity-id.


5.5. Event Processing

The following events may occur in VMTP servers.

   - User Requests

        * Receive

        * Respond

        * Forward

        * GetForwarder

        * GetProcess

        * GetPrincipal

   - Packet Arrival

        * Request Packet

   - Management Operations

        * NotifyVmtpServer

   - Timeouts

        * Client State Record Timeout

The handling of these events is described in detail in the following
subsections.  The conventions of the previous chapter are followed,
including the use of the various subroutines in the description.


5.6. Server User-invoked Events

A user event occurs when a VMTP server invokes one of the VMTP interface
procedures.


5.6.1. Receive

AcceptMessage(reqmcb, segptr, segsize, client, transid, timeout)
    Locate server's request queue.
    if request is queued then
        Remember CSR associated with this Request.
        return Request in reqmcb, segptr and segsize
               and client and transaction id.
    Wait on server's request queue for next request
    up time timeout seconds.
end ReceiveCall

Notes:

   1. If a multi-packet Request is partially received at the time
      of the AcceptMessage, the process waits until it completes.

   2. The behavior of a process accepting a Request as a
      lightweight thread is similar except that the process
      executes using the Request data logically as part of the
      requesting Client process.


5.6.2. Respond

RespondCall is described as one case of the Respond transmission
procedure; RespondMessage is similar.

RespondCall( responsemcb, responsesegptr )
    Locate csr for this client.
    Check segment data accessible, if any
    if local client then
        Handle locally
        return
    endif
    if responsemcb.Code is RESPONSE_DISCARDED then
        Mark as RESPONSE_DISCARDED
        return
    SendPacketGroup( csr )
    set csr.State to Responded.
    if DGM reply then { Idempotent }
        release segment data
        Timeout( csr, TS4(csr.Client), FreeCsr );
    else { Await acknowledgement or new Request else ask for ack. }
        Timeout( csr, TS5(csr.Client), RemoteClientTimeout )
end RespondCall

Notes:

   1. RespondMessage is similar except the Server process must be
      synchronized with the release of the segment data (if any).

   2. The non-idempotent Response with segment data is sent first
      without a request for an acknowledgement.  The Response is
      retransmitted after time TS5(client) if no acknowledgment or
      new Request is received from the client in the meantime.  At
      this point, the APG bit is sent.

   3. The MCB of the Response is buffered in the client CSR, which
      remains for TS4 seconds, sufficient to filter old duplicates.
      The segment data (if any) must be retained intact until:  (1)

      after transmission if idempotent or (2) after acknowledged or
      timeout has occurred if not idempotent.  Techniques such as
      copy-on-write might be used to keep a copy of the Response
      segment data without incurring the cost of a copy.


5.6.3. Forward

Forwarding is logically initiating a new message transaction between the
Server (now acting as a Client) and the server to which the Request is
forwarded.  When the second server returns a Response, the same Response
is immediately returned to the Client.  The forwarding support in VMTP
preserves these semantics while providing some performance optimizations
in some cases.

ForwardCall( req, segptr, segsize, forwardserver )
    Locate csr for this client.
    Check segment data accessible, if any

    if local client or Request was multicast or secure
       or csr.ForwardCount == 15 then
        Handle as a new Send operation
        return
    if forwardserver is local then
        Handle locally
        return
    Set csr.funccode to Request
    Increment csr.ForwardCount
    Set csr.State to Responded
    SendPacketGroup( csr ) { To ForwardServer }
    Timeout( csr, TS4(csr.Client), FreeAlien )
end ForwardCall

Notes:

   1. A Forward is logically a new call or message transaction.  It
      must be really implemented as a new message transaction if
      the original Request was multicast or secure (with the
      optional further refinement that it can be used with a secure
      message transaction when the Server and ForwardServer are the
      same principal and the Request was not multicast).

   2. A Forward operation is never handled as an idempotent
      operation because it requires knowledge that the
      ForwardServer will treat the forwarded operation as
      idempotent as well.  Thus, a Forward operation that includes
      a segment should set APG on the first transmission of the

      forwarded Request to get an acknowledgement for this data.
      Once the acknowledgement is received, the forwarding Server
      can discard the segment data, leaving only the basic CSR to
      handle retransmissions from the Client.


5.6.4. Other Functions

GetRemoteClient is a simple local query of the CSR.  GetProcess and
GetPrincipal also extract this information from the CSR.  A server
module may defer the Probe callback to the Client to get that
information until it is requested by the Server (assuming it is not
using secure communication and duplicate suppression is adequate without
callback.)  GetForwarder is implemented as a callback to the Client,
using a GetRequestForwarder VMTP management operation.  Additional
management procedures for VMTP are described in Appendix III.


5.7. Request Packet Arrival

The basic packet reception follows that described for the Client
routines.  A Request packet is handled by the procedure HandleRequest.

HandleRequest( csr, p, psize )

    if LocalClient(csr) then
        { Forwarded Request on local Client }
        if csr.LocalTransaction != p.Transaction then return
        if csr.State != AwaitingResponse then return
        if p.ForwardCount < csr.ForwardCount then
           Discard Request and return.
        Find a CSR for Client as a remote Client.
        if not found then
            if packet group complete then
                handle as a local message transaction
                return
            Allocate and init CSR
            goto newTransaction
        { Otherwise part of current transaction }
        { Handle directly below. }n
    if csr.RemoteTransaction = p.Transaction then
      { Matches current transaction }
        if OldForward(p.ForwardCount,csr.ForwardCount) then
            return
        if p.ForwardCount > csr.ForwardCount then
          { New forwarded transaction }
            goto newTransaction

        { Otherwise part of current transaction }
        if csr.State = ReceivingRequest then
            if new segment data then retain in CSR segment area.
            if Request not complete then
               Timeout( csr, TS1(p.Client), RemoteClientTimeout )
               return;
            endif
            goto endPacketGroup
        endif
        if csr.State is Responded then
          { Duplicate }
            if csr.Code is RESPONSE_DISCARDED
               and Multicast(p) then
                return
            endif
            if not DGM(csr) then { Not idempotent }
                if SegmentData(csr) then set APG
                { Resend Response or Request, if Forwarded }
                SendPacketGroup( csr )
                timeout=if SegmentData(csr) then TS5(csr.Client)
                          else TS4(csr.Client)
                Timeout( csr, timeout, RemoteClientTimeout )
                return
            { Else idempotent - fall thru to newTransaction }
        else { Presume it is a retransmission }
            NotifyClient( csr, p, OK )
            return
   else if OldTransaction(csr.RemoteTransact,p.Transaction) then
        return
    { Otherwise, a new message transaction. }
newTransaction:
    Abort handling of previous transactions for this Client.

    if (NSRset(p) or NERset(p)) and NoStreaming then
        NotifyClient( csr, p, STREAMING_NOT_SUPPORTED )
        return
|   if NSRset(p) then { Streaming }
|     { Check that consecutive with previous packet group }
|       Find last packet group CSR from this client.
|      if p.Transaction not lastcsr.RemoteTransaction+1 mod 2**32
|         and not STIset(lastcsr) or
|        p.Transaction not lastcsr.RemoteTransaction+256 mod **32
|        then
|         { Out of order packet group }
|         NotifyClient(csr, p, BAD_TRANSACTION_ID )
|         return
|       endif

|       if lastcsr not completed then
|           NotifyClient( lastcsr, p, RETRY )
|       endif
|       if lastcsr available then use it for this packet group
|       else allocate and initialize new CSR
|       if CMG(lastcsr) then
|          Add segment data to lastcsr Request
|          Keep csr as record of this packet group.
|          Clear lastcsr.VerifyInterval
|      endif
|   else { First packet group }
        if MultipleRemoteClients(csr) then ScavengeCsrs(p.Client)
        Set csr.RemoteTransaction, csr.Priority
        Copy message and segment data to csr's segment area
         and set csr.PacketDelivery to that delivered.
        Clear csr.PacketDelivery
        Clear csr.VerifyInterval
        SaveNetworkAddress( csr, p )
    endif
    if packetgroup not complete then
        Timeout( csr, TS3(p.Client), RemoteClientTimeout )
        return;
    endif
endPacketGroup:
    { We have received complete packet group. }
    if APG(p) then NotifyClient( csr, p, OK )
    endif
|   if NERset(p) and CMG(p) then
|       Queue waiting for continuation packet group.
|       Timeout( csr, TS3(csr.Client), RemoteClientTimeout )
|       return
|   endif
    { Deliver request message. }
    if GroupId(csr.Server) then
        For each server identified by csr.Server
            Replicate csr and associated data segment.
            if CMDset(csr) and Server busy then
               Discard csr and data
            else
               Deliver or invoke csr for each Server.
            if not DGMset(csr) then queue for Response
            else Timeout( csr, TS4(csr.Client), FreeCsr )
        endfor
     else
       if CMDset(csr) and Server busy then
           Discard csr and data
        else

           Deliver or invoke csr for this server.
        if not DGMset(csr) then queue for Response
        else Timeout( csr, TS4(csr.Client), FreeCsr )
     endif
end HandleRequest

Notes:

   1. A Request received that specifies a Client that is a local
      entity should be a Request forwarded by a remote server to a
      local Server.

   2. An alternative structure for handling a Request sent to a
      group when there are multiple local group members is to
      create a remote CSR for each group member on reception of the
      first packet and deliver a copy of each packet to each such
      remote CSR as each packet arrives.

5.8. Management Operations

VMTP uses management operations (invoked as remote procedure calls) to
effectively acknowledge packet groups and request retransmissions.  The
following routine is invoked by the Server's management module on
request from the Client.

NotifyVmtpServer(server,clientId,transact,delivery,code)
    Find csr with same RemoteTransaction and RemoteClient
    as clientId and transact.
    if not found or csr.State not Responded then return
    if DGMset(csr) then
        if transmission of Response in progress then
            Abort transmission
            if code is migrated then
               restart transmission with new host addr.
        if Retry then Report protocol error
        return
    endif
    select on code
      case RETRY:
        if csr.RetransCount > MaxRetrans(clientId) then
             if response data segment then
                 Discard data and mark as RESPONSE_DISCARDED
|                if NERset(csr) and subsequent csr then
|                    Deallocate csr and use later csr for
|                    future duplicate suppression
|                endif
             return
        endif
        increment csr.RetransCount
        Set csr.TransmissionMask to missing segment blocks,
            as specified by delivery
        SendPacketGroup( csr )
        Timeout( csr, TS3(csr.Client), RemoteClientTimeout )
      case BUSY:
        if csr.TimeLimit exceeded then
            if response data segment then
                Discard data and mark as RESPONSE_DISCARDED
|               if NERset(csr) and subsequent csr then
|                   Deallocate csr and use later csr for
|                   future duplicate suppression
|               endif
             endif
        endif
        Set csr.TransmissionMask for full retransmission
        Clear csr.RetransCount

        Timeout( csr, TS3(csr.Server), RemoteClientTimeout )
        return

      case ENTITY_MIGRATED:
        Get new host address for entity
        Set csr.TransmissionMask for full retransmission
        Clear csr.RetransCount
        SendPacketGroup( csr )
        Timeout( csr, TS3(csr.Server), RemoteClientTimeout )
        return

      case default:
        Abort transmission of Response if in progress.
        if response data segment then
           Discard data and mark as RESPONSE_DISCARDED
           if NERset(csr) and subsequent csr then
               Deallocate csr and use later csr for
               future duplicate suppression
           endif
        return
    endselect
end NotifyVmtpServer

Notes:

   1. A NotifyVmtpServer operation requesting retransmission of
      the Response is acceptable only if the Response was not
      idempotent.  When the Response is idempotent, the Client must
      be prepared to retransmit the Request to effectively request
      retransmission of the Response.

   2. A NotifyVmtpServer operation may be received while the
      Response is being transmitted.  If an error return, as an
      efficiency, the transmission should be aborted, as suggested
      when the Response is a datagram.

   3. A NotifyVmtpServer operation indicating OK or an error
      allows the Server to discard segment data and not provide for
      subsequent retransmission of the Response.


5.8.1. HandleRequestNoCSR

When a Request is received from a Client for which the node has no CSR,
the node allocates and initializes a CSR for this Client and does a
callback to the Client's VMTP management module to get the Principal,
Process and other information associated with this Client.  It also

checks that the TransactionId is correct in order to filter out
duplicates.

HandleRequestNoCSR( p, psize )
|   if Secure(p) then
|       Allocate and init CSR
|       SaveSourceHostAddr( csr, p )
|       ProbeRemoteClient( csr, p, AUTH_PROBE )
|       if no response or error then
|          delete CSR
|          return
|       Decrypt( csr.Key, p, psize )
|        if p.Checksum not null then
|       if not VerifyChecksum(p, psize) then return;
|       if OppositeByteOrder(p) then ByteSwap( p, psize )
|       if psize not equal sizeof(VmtpHeader) + 4*p.Length then
|          NotifyClient(NULL, p, VMTP_ERROR )
|          return
|       HandleRequest( csr, p, psize )
|       return
    if Server does not exist then
        NotifyClient( csr, p, NONEXISTENT_ENTITY )
        return
    endif
    if security required by server then
        NotifyClient(csr, p, SECURITY_REQUIRED )
        return
    endif
    Allocate and init CSR
    SaveSourceHostAddr( csr, p );
    if server requires Authentication then
        ProbeRemoteClient( csr, p, AUTH_PROBE )
        if no response or error then
           delete CSR
           return
    endif
    { Setup immediately as a new message transaction }
    set csr.Server to p.Server
    set csr.RemoteTransaction to p.Transaction-1

    HandleRequest( csr, p, psize )
    endif

Notes:

   1. A Probe request is always handled as a Request not requiring
      authentication so it never generates a callback Probe to the

      Client.

   2. If the Server host retains remote client CSR's for longer
      than the maximum packet lifetime and the Request
      retransmission time, and the host has been running for at
      least that long, then it is not necessary to do a Probe
      callback unless the Request is secure.  A Probe callback can
      take place when the Server asks for the Process or
      PrincipalId associated with the Client.

5.9. Timeouts

The server must implement a timeout for remote client CSRs.  There is a
timeout for each CSR in the server.

RemoteClientTimeout( csr )
  select on csr.State
    case Responded:
        if RESPONSE_DISCARDED then
            mark as timed out
            Make a candidate for reuse.
            return
        if csr.RetransCount > MaxRetrans(Client) then
            discard Response
            mark CSR as RESPONSE_DISCARDED
            Timeout(csr, TS4(Client), RemoteClientTimeout)
            return
        increment csr.RetransCount
        { Retransmit Response or forwarded Request }
        Set APG to get acknowledgement.
        SendPacketGroup( csr )
        Timeout( csr, TS3(Client), RemoteClientTimeout )
        return
    case ReceivingRequest:
      if csr.RetransCount > MaxRetrans(csr.Client)
         or DGMset(csr) or NRTset(csr) then
          Modify csr.segmentSize and csr.MsgDelivery
          to indicate packets received.
          if MDMset(csr) then
              Invoke processing on Request
              return
          else
              discard Request and reuse CSR
              (Note: Need not remember Request discarded.)
              return
      increment csr.RetransCount
      NotifyClient( csr, p, RETRY )
      Timeout( csr, TS3(Client), RemoteClientTimeout )
      return
    default:
        Report error - invalid state for RemoteClientTimeout
    endselect
end RemoteClientTimeout

Notes:

   1. When a CSR in the Responded state times out after discarding

      the Response, it can be made available for reuse, either by
      the same Client or a different one.  The CSR should be kept
      available for reuse by the Client for as long as possible to
      avoid unnecessary callback Probes.