Traffic Control using tcng and HTB HOWTO

Traffic control is the term given to the entire packet queuing subsystem in a network or network device. Traffic control consists of several distinct operations. Classifying is a mechanism by which to identify packets and place them in individual flows or classes. Policing is a mechanism by which one limits the number of packets or bytes in a stream matching a particular classification. Scheduling is the decision-making process by which packets are ordered and re-ordered for transmission. Shaping is the process by which packets are delayed and transmitted to produce an even and predictable flow rate.

These many characteristics of a traffic control system can be combined in complex ways to reserve bandwidth for a particular flow (or application) or to limit the amount of bandwidth available to a particular flow or application.

One of the key concepts of traffic control is the concept of tokens. A policing or shaping implementation needs to calculate the number of bytes or packets which have passed at what rate. Each packet or byte (depending on the implementation), corresponds to a token, and the policing or shaping implementation will only transmit or pass the packet if it has a token available. A common metaphorical container in which an implementation keeps its token is the bucket. In short, a bucket represents the both the number of tokens which can be used instantaneously (the size of the bucket), and the rate at which the tokens are replenished (how fast the bucket gets refilled).

See the section called “What is htb?” for an example of buckets in a linux traffic control system.

Under linux, traffic control has historically been a complex endeavor. The tc command line tool provides an interface to the kernel structures which perform the shaping, scheduling, policing and classifying. The syntax of this command is, however, arcane. The tcng project provides a much friendlier interface to the human by layering a language on top of the powerful tc command line tool. By writing traffic control configurations in tcng they become easily maintainable, less arcane, and importantly also more portable.

Hierarchichal Token Bucket is a classful qdisc written by Martin Devera with a simpler set of configuration parameters than CBQ. There is a great deal of documentation on the author's site and also on Stef Coene's website about HTB and its uses. Below is a very brief sketch of the HTB system.

Conceptually, HTB is an arbitrary number of token buckets arranged in a hierarchy (yes, you probably could have figured that out without my sentence). Let's consider the simplest scenario. The primary egress queuing discipline on any device is known as the root qdisc.

The root qdisc will contain one class (complex scenarios could have multiple classes attached to the root qdisc). This single HTB class will be set with two parameters, a rate and a ceil. These values should be the same for the top-level class, and will represent the total available bandwidth on the link.

In HTB, rate means the guaranteed bandwidth available for a given class and ceil is short for ceiling, which indicates the maximum bandwidth that class is allowed to consume. Any bandwidth used between rate and ceil is borrowed from a parent class, hence the suggestion that rate and ceil be the same in the top-level class.

A number of children classes can be made under this class, each of which can be allocated some amount of the available bandwidth from the parent class. In these children classes, the rate and ceil parameter values need not be the same as suggested for the parent class. This allows you to reserve a specified amount of bandwidth to a particular class. It also allows HTB to calculate the ratio of distribution of available bandwidth to the ratios of the classes themselves. This should be more apparent in the examples below.

Hierarchical Token Bucket implements a classful queuing mechanism for the linux traffic control system, and provides rate and ceil to allow the user to control the absolute bandwidth to particular classes of traffic as well as indicate the ratio of distribution of bandwidth when extra bandwidth becomes available (up to ceil).

Keep in mind when choosing the bandwidth for your top-level class that traffic shaping only helps if you are the bottleneck between your LAN and the Internet. Typically, this is the case in home and office network environments, where an entire LAN is serviced by a DSL or T1 connection.

In practice, this means that you should probably set the bandwidth for your top-level class to your available bandwidth minus a fraction of that bandwidth.

Traffic Control Next Generation (tcng) is a project by Werner Almesberger to provide a powerful, abstract, and uniform language in which to describe traffic control structures. The tcc parser in the tcng distribution transforms tcng the language into a number of output formats. By default, tcc will read a file (specified as an argument or as STDIN) and print to STDOUT the series of tc commands (see iproute2 below) required to create the desired traffic control structure in the kernel.

Consult the parameter reference for tcng to see the supported queuing disciplines. Jacob Teplitsky, active on the LARTC mailing list and a contributor to the tcng project, wrote the htb support for tcng.

The tcc tool can produce a number of different types of output, but this document will only consider the conventional and default output. Consult the TCNG manual for more detailed information about the use of tcng.

The tcsim tool is a traffic control simulator which accepts tcng configuration files and reads a control language to simulate the behaviour of a kernel sending and receiving packets with the specified control structures. Although tcsim is a significant portion of the tcng project, tcsim will not be covered here at all.

The kernel requirements are very easy to meet. Kernel 2.4.20 and newer include support for HTB and dsmark, so simply be certain that these options are turned on in the QoS/Fair Queuing portion of your kernel configuration. For a brief summary of the options to select in kernel configuration, visit the DiffServ project kernel configuration notes.

For kernels older than 2.4.20, the following tarball containing a patch should be applied to your 2.4.17 or newer kernel tree.

The tc command is a part of the iproute2 utility suite. For general documentation on iproute2, see http://linux-ip.net/ and the iproute2 manual. The software itself is available directly from Alexey Kuznetsov'z FTP archive but commonly also via packages supplied with your linux distribution. If your distribution can make use of RPMS, you can download this SRPM and compile it on your own system.

If you need to compile iproute2 yourself, use the patch to tc from this tarball at Martin Devera's HTB site in order to provide support for HTB in tc.

Your tc will also need to support dsmark, the diffserv marking mechanism. Fortunately, this is a simple change to the Config file from the iproute2 source package. Simply change TC_CONFIG_DIFFSERV=n to TC_CONFIG_DIFFSERV=y and recompile.

The SRPM creates a tc binary with support for dsmark and for HTB, both of which are required for this example.

Support for tcng is the easiest part of the process. Simply untar the tcng source and run ./configure --no-tcsim before compiling.

If you are on an RPM-based system, you can use the SPEC file in tcng/build/tcng.spec to build for your distribution, or you can download and compile this SRPM. The SRPM produces two packages, tcc and tcc-devel. You need only tcc to create configurations.

In order to run the tcc parser, you will also need to have the cpp package installed. tcc uses cpp.

Many general concepts will be introduced with this example. This example can be compiled to its tc output with the command tcc class-selection-path.tcc.

Example 2. /etc/sysconfig/tcng/class-selection-path.tcc

/*
 * Simply commented example of a tcng traffic control file.
 *
 *   Martin A. Brown <martin@linux-ip.net>
 *
 * Example:  Using class selection path.
 *
 * (If you are reading the processed output in HTML, the callouts are
 *  clickable links to the description text.)
 *
 */

#include "fields.tc"     
#include "ports.tc"

#define INTERFACE  eth0  

dev INTERFACE {
    egress { 

        /* In class selection path, the filters come first!  DSmark */ 

        class ( <$ssh> )    if tcp_sport ==  22 && ip_tos_delay == 1 ;
        class ( <$audio> )  if tcp_sport == 554 || tcp_dport == 7070 ;
        class ( <$bulk> ) \
            if tcp_sport == PORT_SSH || tcp_dport == PORT_HTTP ; 
        class ( <$other> )  if 1 ; 

        /* section in which we configure the qdiscs and classes */ 

        htb () { 
            class ( rate 600kbps, ceil 600kbps ) { 
                $ssh   = class ( rate  64kbps, ceil 128kbps ) { sfq; } ; 
               $audio = class ( rate 128kbps, ceil 128kbps ) { sfq; } ;
                $bulk  = class ( rate 256kbps, ceil 512kbps ) { sfq; } ;
                $other = class ( rate 128kbps, ceil 384kbps ) { sfq; } ; 
            }
        }
    }
}
      

	The tcng language provides support for C-style include directives which can include any file. Files are included relative to the current directory or the tcng library (normally /usr/lib/tcng/include). Strictly speaking, it is not necessary to #include ports.tc and fields.tc, because tcc will include these by default. The use of #include can allow for flexible definition of variables and inclusion of common traffic control elements. See also the tcng manual on includes.
	These are CPP directives. The #define can be used to create macros or constants. For more on their use, you should see the tcng manual on variables.
	The egress keyword is synonymous with the dsmark keyword. The example here uses class selection path. It is the use of the egress keyword in this configuration which requires dsmark support in the kernel and tc.
	Class selection path is one approach to traffic shaping. In class selection path, the packet is marked (DiffServ mark) upon entry into the router. The router may take any number of actions or apply any number of policing, scheduling or shaping actions on the packet as a result of this initial classification. Consult the tcng manual on class selection path for further details.
	This example shows the use of names for the ports instead of numbers. This is one of the conveniences of tcng afforded by the automatic inclusion of ports.tc. The ports are named in accordance with IANA port names. See IANA's registered ports for these names or examine the file ports.tc. Names and numbers are equally acceptable and valid.
	Note this peculiar construct which classifies any packet which have not yet been classified. Any packet which has not been classified by the above classifiers is put into the class "$other" here. The if 1 construct can be used to classify the remainder of unclassified traffic.
	This is the creation of the root qdisc which is attached to device, eth0 in this case. Consult the reference material in the tcng appendix on queuing discipline parameters for valid parameters to each qdisc. Any qdisc parameters can be inserted into the parentheses in the same fashion as the class parameters further below in the example. If no parameters need be specified, the parentheses are optional.
	The top level class in this example sets the maximum bandwidth allowed through this class. Let's assume that eth0 is the inside network interface of a machine. This limits the total bandwidth to 600 kilobits per second transmitted to the internal network. The parameters rate and ceil should be familiar to anybody who has used HTB. These are HTB specific parameters and are translated properly by the tcc utility. See the table on tcng rate and speed specification.
	This is the assignment of a class to a variable. This is commonly done as part of class selection path.
	As suggested by Martin Devera on the HTB homepage, an embedded SFQ gives each class a fair queuing algorithm for distribution of resources to the contenders passing packets through that class. Note the absence of any parameters to the embedded queuing discipline. If no queuing discipline is specified for leaf classes, they contain the default, a pfifo_fast qdisc. The inclusion of a stochastic fair queuing qdisc in the leaf classes inhibits the ability of a single connection to dominate in a given class.