Tuning networking resources

STREAMS resources

The X Window System, networking services such as TCP/IP and NFS, applications that use streams pipes, and certain device drivers use STREAMS to perform I/O.

The STREAMS I/O system was designed to provide a simultaneous two-way (full duplex) connection between a process running in user space and a device driver (or pseudo-device driver) linked into the kernel. The topmost level within the kernel with which the user process communicates is known as the stream head.

Using STREAMS has the advantage that it allows the processing of I/O between an application and a device driver to be divided into a number of functionally distinct layers such as those required by network architectures that implement TCP/IP or the Open Systems Interconnection (OSI) 7-layer model.

The STREAMS I/O mechanism is based on the flow of messages from the stream head to a device driver, and in the opposite direction, from the device driver to the stream head. Messages that are passed away from the stream head toward the driver are said to be traveling downstream; messages going in the opposite direction are traveling upstream. Between the stream head and the driver, there may be a number of stream modules which process messages in addition to passing them to the next module. Each type of module is implemented as a separate driver linked into the kernel. For example, the udp driver implements the network module that applies the UDP protocol to messages. Each module has two separate queues for processing upstream and downstream-bound messages before handing them to the next module.

Implementation of networking protocols using STREAMS

A network protocol stack is built by linking STREAMS protocol modules. For example, the TCP/IP protocol stack is built by linking the Internet Protocol (IP) module, and the Transmission Control Protocol (TCP) module. Modules can also be multiplexed so that a module can talk to several stream heads, drivers or other modules. ``Implementation of networking protocols using STREAMS'' shows:

Two stream heads talking to the same TCP module.
The TCP and User Datagram Protocol (UDP) transport layer modules both connected to the underlying IP module.
Two different network adapter drivers interfaced to the IP module; this is necessary on systems that act as routers between two networks that use the Internet Protocol.

Note that the diagram simplifies the Link Layer Interface (LLI) for clarity. This layer consists of the Data Link Provider Interface (DLPI) and the SCO Media Access Control (MAC) Driver Interface.

For a more complete picture of the available protocol stacks and drivers, see ``Network hardware drivers''.

``Creating an Ethernet frame by successive encapsulation'' shows how the TCP/IP protocol stack encapsulates data from an application to be sent over a network that uses Ethernet as the physical layer. The Transport layer module adds a header to the data to convert it into a TCP segment or a UDP packet. The Internet layer module turns this into an IP datagram, and then passes it to the network driver which adds a header and CRC trailer. The resulting Ethernet frame is then ready for transmission over the physical medium.

Creating an Ethernet frame by successive encapsulation

To retrieve data from an Ethernet frame, the inverse process is applied; the received information is passed as a message upstream where it is processed by successive modules until its data is passed to the application. If the information is received by a router between two networks, the message will only travel upward as far as the Internet layer module from one network adapter before being passed back down to a different network adapter.

``Virtual and physical connections over a network'' shows protocol stacks on two machines linked via a physical connection technology such as Ethernet, Token Ring or Fiber Distributed Data Interface (FDDI). Applications appear to have a direct or virtual connection; they do not need to know how connection is established at the lower levels.

Virtual and physical connections over a network

The primary usage of memory by the STREAMS subsystem is for building messages. ``Memory structures used by STREAMS messages'' illustrates how a message is created from pieces dynamically allocated from the memory reserved for use by STREAMS. Each message consists of a fixed-size message header and one or more buffer headers attached to buffers. The buffers come in several different sizes and contain the substance of the message such as data, ioctl control commands (see ioctl(S), and streamio(M)), acknowledgements, and errors.

Message buffers are available in 15 sizes or classes:

16-byte class 0 buffers are stored within the buffer header.
Class 1 to class 14 buffers, ranging in size of powers of 2 from 64-byte to 512KB, use a separately allocated buffer structure pointed to by the buffer header.

Memory structures used by STREAMS messages

Four kernel parameters are important for the configuration of STREAMS: NSTRPAGES, STRSPLITFRAC, NSTREAM, and STRMAXBLK.

NSTRPAGES controls the total amount of physical memory that can be made available for messages. The kernel can dynamically allocate up to NSTRPAGES pages of memory for message headers, buffer headers, and buffers. If a message needs a buffer which is not currently available on the free list of buffers, a new buffer is dynamically allocated for use from memory. If more than STRSPLITFRAC percent of NSTRPAGES is in use and a suitable buffer is not available on the free list, the kernel will try to split a larger buffer for use and only allocates more memory if this fails.

The default value of STRSPLITFRAC is 80%; if you set this value lower, STREAMS will use less memory which will tend to become fragmented more quickly. When this happens, unallocated STREAMS memory exists as many small non-contiguous pieces which are unusable for large buffers. The STREAMS daemon, strd, manages memory on behalf of the STREAMS subsystem. If strd runs, it expends CPU time in system mode in order to release pages of STREAMS memory for use (this is known as garbage collection).

NSTREAM controls the number of stream heads that can be used. One stream head is needed for each application running on your machine that uses STREAMS to establish connections. Applications that use stream pipes require two stream heads per pipe.

Examples of applications that use stream heads are:

Remote login and file transfer programs such as ftp(TC), rcmd(TC), rcp(TC), rlogin(TC), and telnet(TC).
Remote X clients.
The power management daemon, pwrd(ADM), and each APM or UPS driver that talks to it.

NOTE: NSTREAM should be set to at least 256 on systems running SCO OpenServer software which mount several remote filesystems or invoke remote X clients. A program will not run if it cannot obtain a stream head and will output a message such as:

NOTICE: program: out of streams

If you see such a message, increase the value of NSTREAM, relink the kernel, and reboot.

Each configured stream head requires 80 bytes of memory. Apart from this overhead, the value of NSTREAM has no effect on performance.

STRMAXBLK controls the maximum size of a STREAMS message buffer. You must adjust the value of this parameter to 4096 if you are using older network cards whose LLI drivers use programmed DMA to transfer data directly between the interface card and memory. These drivers assume that the 4KB memory pages that compose a buffer are contiguous in physical memory. As STREAMS use dynamically allocated memory, this may not be the case. To avoid memory being corrupted, the maximum message buffer size must be set equal to the size of a memory page.