Scheduled Transfer Procotol
Frequently Asked Questions
ST History OS
Bypass
NIC Design
client-server oerations
Flow Control
transmission units
connection management
addressing
ST History and Evolution
Q: How did ST come into existence as an ANSI standard project?
ST was first proposed and developed as part of theANSI/NCITS
standard development for Hippi-6400, also known as GSN (Gigabyte System
Network). As Hippi-6400 progressed, two things became clear. First,
the ST work could not be completed at the same time as the Hippi-6400 PHY
standard; second, ST is useful on media other than GSN. The
Hippi-6400 draft standard was split into two documents: one for the PHY
and one for ST.
Q: What specific problems does ST address?
ST provides for traditional reliable transport, e.g. flow control,
error control, congestion management, while providing specialized facilities
for buffer management, zero-copy host transfers, remote direct memory
access, and OS Bypass-style programming interfaces. A high-bandwidth
network can create buffer overrun, i.e. congestion leading to performance
degradation, at a destination endpoint by consuming available buffers quickly
- before an adaptive feedback mechanism could react. End-to-end transfers
in ST take place only after endpoints have established that there is dedicated
buffer space at the destination. This avoids buffer congestion.
The control message exchange itself provides one level of flow control
and also gives both endpoint transaction identifiers for the buffers and
transfer. Information in the transaction identifier, specifically
bufx, mx, I_id, and R_id in the protocol, provide the basis
for a zero-copy host interface.
OS Bypass
Q: What is meant by OS Bypass?
When a hardware device, e.g. peripheral controller, can be directly
controlled from an application program without OS system calls, then the
device provides an OS Bypass mode of operation.
Q: When is OS Bypass useful?
Applications that are sensitive to message-passing latency can
benefit when operating system overhead for sending or receiving messages
is a significant component of overall message latency.
NIC Design
Q: What are the minimum requirements for a NIC, e.g. ethernet, to
support ST?
An unmodified ethernet adaptor could support ST, but not very well because
software in the end system would have to decode protocols headers and copy
data to or from application buffers.
Q: What are the minimum requirements for a NIC, e.g. ethernet, to
provide zero-copy ST transfers?
The receive side of an adaptor need only receive and recogenize the DATA
opcode and calculate a dma address from information found in the ST header.
Doing this correctly requires sufficient means for detecting and validating
a received ST header, checking the operation field for DATA, having one
or more lookup tables for translating the integer values found in ST header
fields to dma addresses and performing other checks and validations, and
having means for delivering a status descriptor to host software for each
received ST header. All other ST opcodes can be effectively implemented
in host software. An adaptor should also provide means for atomically
setting up and clearing state information held in an adaptor. On the output
side of an adaptor it can be helpful to provide processing power to format
output messages and perform the output tiling function (see section
6.3 of the ST specification for examples).
Q: what NIC facilities are needed to support low-latency OS Bypass operations?
A NIC needs to allow direct application writes to the hardware and
must provide validation support to assure that memory protections are
not violated by application clients.
Addressing
Q: what address types are defined
for ST?
n
Transmission Units
Q:What is the relationship between physical transmission units, e.g.
frames or packets,
and the message units defined by ST?
A: The ST Protocol defines a hierarchy
for dividing large data transfers into smaller components. An application
sends a transfer which may be d ivided into blocks
which are transmitted on the physical medium using one or more of the transmission
units defined for the medium - e.g. frames or packets. The smallest
of these units is generally referred to as an STU.
Q: What is an STU?
A: The ST protocol defines STU, or Scheduled
Transfer
Unit,
ras the data payload of an ST DATA operation, i.e. DATA message.
The protocol negotiates a maximum STU size, or data paylod size,
when a connection is established between endpoints. The term
STU is often in conversation to refer to a complete DATA message, i.e.
MAC header plus ST header plus data payload.
Q: Why is 256 bytes the minimum STU size?
A: The size was chosen to be greater than
or equal to the size of a cache line.
Q: What is the relationship between STUs and blocks?
A: A block is transmitted as one or more
STUs. Conversely, an ST implementation assembles a sequence of one
or more STUs in Destination memory to create the image of a block.
Q: Why must STU's be delivered in order? What happens if they're
not?
A: STUs are to be delivered in order
to avoid the complexity of out-of-order reassembly at this level
of the protocol. Blocks may be sent or delivered out of order.
On media or in networks that have the potential to reorder data,
an ST implementation should be decompose any transfer into a sequence of
blocks that each consist of a single STU.
Connection
Management
Q: Why does ST define both "ports" and "ULP-ports".
A: ST "ports" designate the endpoints of ST virtual connections and
"ULP-ports" designate
upper-layer protocol (ULP) endpoints. As an ST connection is
established, the initiating side uses the Request Connection message exchange
to request a connection to a particular ULP-port at the destination side.
The port numbers used in this exchange are as specified by IANA for IP-based
ULP services. Once a connection is established, the endpoints communicate
using port numbers exchanged during the setup handshake. This port
numbers are assigned by the ST implementation at each endpoint.
Q: Why does ST need connections?
The protocol specifies sequences of message exchanges that negotiate and
control data movement between endpoints. These sequences establish shared
state between endpoints which must be maintained. The connection
abstraction is a well-established means for maintaining this kind of state
information.
Q:Why are ST connections call "virtual connections"?
A single network interface card, or NIC, is expected to support many simultaneous
connections. Many of these connections may also be of the OS Bypass
variety.
Optional
Messages
Event
Notification
Flow
Control
Q: What objects are flow-controlled
in ST?
A:
Q: What is the flow control strategy
in ST?
A: The flow of transfers and blocks are
regulated by RTS/CTS. That is, transmission of data may not occur
unless both destination and source buffers are ready. The destination
descriptor queue is controlled by the credit values exchanged in RSR messages.
Client-Server
Operation
Q: If ST is used between multiple clients
A: