/*********************************************************************
 * DBS: Distributed Benchmark System
 * Copyright (c) 1995, 1996 Yukio Murayama
 * Copyright (c) 1995, 1996 Nara Institute of Science and Technology
 * All rights reserved.
 *
 * Permission to use, copy, modify and distribute this software and its
 * documentation is hereby granted, provided only with the following
 * conditions are satisfied:
 *
 * 1. Both the copyright notice and this permission notice appear in
 *    all copies of the software, derivative works or modified versions,
 *    and any portions thereof, and that both notices appear in
 *    supporting documentation.
 * 2. All advertising materials mentioning features or use of this
 *    software must display the following acknowledgment:
 *      This product includes software developed by Nara Institute of 
 *      Science and Technology and its contributors.
 * 3. Neither the name of Nara Institute of Science and Technology nor 
 *    the names of its contributors may be used to endorse or promote 
 *    products derived from this software without specific prior written
 *    permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE DEVELOPER ``AS IS'' AND NARA 
 * INSTITUTE OF SCIENCE AND TECHNOLOGY DISCLAIMS ANY LIABILITY OF 
 * ANY KIND FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF 
 * THIS SOFTWARE. ALSO, THERE IS NO WARRANTY IMPLIED OR OTHERWISE, 
 * NOR IS SUPPORT PROVIDED.
 *
 * Feedback of the results generated from any improvements or
 * extensions made to this software would be much appreciated.
 * Any such feedback should be sent to:
 *
 *  Yukio Murayama
 *  E-mail:  <yukio-m@is.aist-nara.ac.jp>
 *  URL:     <http://shika.aist-nara.ac.jp/member/yukio-m/index.html>
 *  Address: Graduate School of Information Science, 
 *           Nara Institute of Science and Technology,
 *           Takayama 8916-5, Ikoma, Nara, Japan
 *
 * Nara Institute of Science and Technology has the rights to 
 * redistribute these changes.
 *********************************************************************/

DBS (Distributed Benchmark System) User's Manual Ver. 1.1
                            Yukio Murayama (yukio-m@is.aist-nara.ac.jp)

1. What is DBS

A goal of this system is to provide a useful tool chest for evaluating
entire TCP functions.  Since there are three TCP control mechanisms
that are essential for evaluating TCP performance, i.e. flow control,
retransmission control and congestion avoidance control, we introduce
the following properties as the fundamental of DBS functions:

 - The DBS can treat multiple TCP connections spanning multiple hosts 
   simultaneously, and
 - The DBS has the capability of measuring the changes of application
   level throughput at every data transmissions.


Having these properties, the DBS will be able to evaluate the three
mechanisms of TCP mentioned above.  In a congested network, data
transmissions may cause TCP segment lost.  In this case,
retransmissions should be done for recovering the lost segments.  As
its result, both the retransmission control and the congestion
avoidance control are invoked.  On the other hand, segment
transmission in a less crowded network may invoke the TCP flow control
mechanism.

The DBS can generate various situations where the three controls are
working together.  Moreover, the DBS can cope with the fluctuation of
the network traffic in the case that the network load changes from
high to low or from low to high respectively.

With these functions, the DBS can evaluate TCP implemented in each OS
and evaluate various kinds of network components which integrated into
a single network system.


2. To Do Things Before using DBS 

You should install xntp for synchronizing the time of all your hosts
before using DBS.


3. Characteristics of DBS

 - Data can be transfer to hosts connected in multiple points utilizing
   TCP and UDP protocols.
 - Changes in throughput can be measured. 
 - Complicated traffic patterns such as MPEG can be send. 
 - The value can be recorded if TCP DEBUG function is implemented in OS 
   kernel.  TCP/IP pseudo-headers, retransmit segments, congestion window
   size, round trip time and other information can be identified utilizing
   this function. 


4. Programs and functions of DBS

dbsc : DBS controller

  The DBS controller is a program controlling the experiment of
  TCP/UDP data transfer.  Functions of the DBS controller are that (1)
  the DBS controller reads command from a command file, then (2) it
  asks the DBS daemons to start data transfer experiments, and (3)
  after receiving results from the daemons, the DBS controller will
  save them into the local files.

dbsd : DBS daemon

  The DBS daemon is a daemon program that is launch on the
  experimental hosts.  It sends and receives network traffic according
  to the commands instructed by the DBS controller.

dbs_view : DBS viewer

  The DBS viewer is a program for analysis data which is gathered by
  the DBS controller. It draws graphs to reveal the transitions of
  sequence numbers, changes of throughput, changes of delay times or
  other performance indexes. Perl and Gnu Plot V 3.5 are needed for
  executing this program.


5. Flow of DBS

 a command file
 +-----------------------------------------------------------------------+
 | {                                                                     |
 |   sender  {hostname=host1; port=20001; pattern{8192, 8192, 0.0, 0.0}} |
 |   receiver{hostname=host2; port=20001; pattern{8192, 8192, 0.0, 0.0}} |
 |   file = data/test1; send_times=100; protocol=TCP;                    |
 |   start_time=0.0; end_time=30;                                        |
 | }                                                                     |
 | {                                                                     |
 |   sender  {hostname=host3; port=20001; pattern{8192, 8192, 0.0, 0.0}} |
 |   receiver{hostname=host4; port=20001; pattern{8192, 8192, 0.0, 0.0}} |
 |   file = data/test2; send_times=100; protocol=TCP;                    |
 |   start_time=0.0; end_time=30;                                        |
 | }                                                                     |
 +-----------------------------------------------------------------------+
             | 
             |
             |(1)         DBS experimental hosts/experimental Network
             |                           host1
 +-----------+         Commands(2)-->   +----+   Data Transmissions(3)
 |           |             +----------> |dbsd| <-----------+
 |           |             |            +----+             |
 |           V             |             host2             |
 |    DBS control host     |            +----+             |
 |          +----+ <-------+  +-------> |dbsd| ---------+  |
 |          +    + <----------+         +----+          |  |
 |          |dbsc|                       host3          |  |
 |          +    + <----------+         +----+          |  |
 |          +----+ <-------+  +-------> |dbsd| ---------|--+
 |           |             |            +----+          |
 |           |             |             host4          |
 |           |             |            +----+          |
 |           |             +----------> |dbsd| <--------+
 |           |(5)      <-- Results(4)   +----+
 |           |
 |           V  Result Files
 |    +---------------------------------------------------------------------+
 |    | send_sequence send_size send_time recv_sequence recv_size recv_time |
 | +---------------------------------------------------------------------+  |
 | | send_sequence send_size send_time recv_sequence recv_size recv_time |  |
 | |       0       8192      0.001535          0       8192     0.127248 |  |
 | |    8192       8192      0.003842       8192       8192     0.138526 |  |
 | |   16384       8192      0.005675      16384       8192     0.147517 |  |
 | |   24576       8192      0.007400      24576       8192     0.155452 |  |
 | |   32768       8192      0.136132      32768       8192     0.166310 |  |
 | |   40960       8192      0.152961      40960       8192     0.171088 |  |
 | |   49152       8192      0.167777      49152       8192     0.182464 |  |
 | |     .           .         .             .           .           .   |  |
 | |     .           .         .             .           .           .   |--+
 | +---------------------------------------------------------------------+
 |         |
 |         |
 |         V
 +----->dbs_view (perl script using Gnuplot V3.5)
           |(6)
           V
      +-----------------------------------------------+
      |                                               |
      | Throughput(Mbps)                              |
      |   +---------------------------------------+   |
      |   |                              test1 -  |   |
      |   |      .       ....            test2 .  |   |
      |   |     . .     .    .         ------     |   |
      |   |     .  .....      .       /      \    |   |
      |   |---- .    --        . /----            |   |
      |   |  . \   /    \       .                 |   |
      |   |..   | |      |     |.                 |   |
      |   |     | |      |     | .                |   |
      |   |     | |      |     |  ......          |   |
      |   |     | |      |     |        .....     |   |
      |   |     | |      |     |             .    |   |
      |   +---------------------------------------+   |
      |                   Time (s)                    |
      |                                               |
      +-----------------------------------------------+


6. Usage

dbsc [-p port_number] [-d] [-v] command_file

   command_file     Specify file name of command file.
   -p port_number   Specify a port number. If no port number is specified,
                    dbsc will attempts to contact port number 10710.
   -d               Execute in Debug Mode.
   -v               Display version and help messages.

dbsd [-p port_number] [-d] [-D] [-v] [-h allow_host]

   -p port_number   Specify a port number. If no port number is specified,
                    dbsd will attempts to contact port number 10710.
   -d               Execute in Debug Mode.
   -h allow_host    Describe command host.
   -D               Execute by inetd.
   -v               Display version and help messages.

dbs_view

 There are a lot of options for DBS viewer. `-f' is indispensable.
 Usually we attach a `-p' option. If you want to display the 
 graph with the same parameter as before, you may omit the `-p'
 option. In this case, the process time will be shorter.
 Here is a summary of all the options.

 -f file file . . Specify command file name(Multiple designation is 
                  possible). This is not a output file. This is the file 
                  that is specified when you execute dbsc. 
 -a area          Describe the area of drawing the graph.
                  Specify like "[0:20][0:1000]".(double quotation is needed)
 -sq s|r|u|S|R|U  Display sequence numbers on the application level
                  s: display sending sequence number with Step Line
                  r: display receiving sequence number with Step Line
                  u: display UDP lost packet with Step Line
                  S: display sending sequence number with Dot
                  R: display receiving sequence number with Dot
                  U: display UDP lost packet with Dot
 -th s|r          display throughput on the application level
                  s: display sending throughput with Line
                  r: display receiving throughput with Line
                  t: display the total receiving throughput with Line
                  S: display sending throughput with Step Line
                  R: display receiving throughput with Step Line
                  T: display the total receiving throughput with Step Line
 -delay           display transmission delay time on the application level
 -jitter s|r      display jitters on the application level
                  s: display sending jitter with Step Line
                  r: display receiving jitter with Step Line
 -tsq s|r|a|d     display sequence numbers on the kernel level.
                  s: display sending sequence number with Step Line
                  r: display receiving sequence number with Step Line
                  a: display ACK sequence number with Step Line
                  d: display lost segments with Step Line
 -tth s|r         display throughput on the kernel level
                  s: display sending throughput with Line
                  r: display receiving throughput with Line
                  S: display sending throughput with Step Line
                  R: display receiving throughput with Step Line
 -twin w|s        display congestion window sizes and slow start thresholds
                  w: display Congestion Window
                  s: display Slow Start Threshold
 -tlost s|a       calculate lost segments
                  s: calculate lost segments
                  a: calculate lost ACK
 -trtt r|m|o      display round trip times and retransmission time outs.
                  r: display real RTT (round trip time) with Step Line
                  m: display measured RTT in the kernel with Step Line
                  o: display RTO (retransmission time out) with Step Line
 -tdelay s|a      displays delay times of the kernel level.
                  s: display sending delay time with Step Line
                  a: display ACK delay time with Step Line
 -tjitter s|r     display jitters on the kernel level
                  s: display sending jitter with Step Line
                  r: display receiving jitter with Step Line
 -ulost           calculate lost packets of UDP.
 -t time (sec)    Specify the grain degree of the time interval
                  (default is 0.1 seconds).
 -p               execute options `-tth', `-tlost', `-trtt', `-tdelay', 
                  `-tjitter' and `-ulost' in one time
 -ps              output a Postscript.
 -eps             output a EPS.
 -color           output with color PS or EPS (specify -ps or -eps same time)
 -title #         Specify the number of word in the command file to be
		  use as title of data. Examples are as follows.

                  the dbs_view creates many temporary files. Therefore, when
                  you specify a output file name in the command file, it is
                  recommended full path name such as "name/file" is specified.

                  For example:

                     file = test1/host1_host2/data1

                  In this case, please make the directory with mkdir before
                  experimenting.  With this configuration, dbs_view plots
                  the title of the data as follows.

                     -title isn't specified  -> "test1/host1_host2/data1"
                     -title -1               ->  no title
                     -title 1                -> "test1"
                     -title 2                -> "host1_host2"
                     -title 3                -> "data1"


7.  Executing Method of inetd

  If you like dbsd executed by inetd automatically, please make
  configuration as follows:

  1. copy dbsd to /usr/local/etc/dbsd.

  2. add the next line to /etc/services.

      dbs 10710/tcp

  3. add the next line to /etc/inetd.conf.

      dbs stream tcp nowait root /usr/local/etc/dbsd dbsd -D

  4. Investigate the process number of inetd, execute as follows:

      kill -HUP 'process number'

  With the above-mentioned procedure, inetd will execute dbsd anytime when
  requested by dbsc. But this will make a security problem in some
  environment connected with World Wide Internet.  Some villains called
  'cracker' can make your network congesting wrongly if they discover the
  above configuration. To avoid this problem dbsd can restrict accepting
  commands from only specific host that you think secure.  If you would
  like it, write "-h hostname" behind the dbsd -D, as follows:

      dbs stream tcp nowait root /usr/local/etc/dbsd dbsd -D -h cmdhost


8. Experimental Flow

 1. Write experiment method in a command file.

 2. Start DBS daemon on the all experimental hosts.  
    (This step is not necessary if dbsd is registered in inetd.conf)

 3. Start DBS controller and wait for the experiment to be done.
    the experimental end time is 'end_time' + 20 seconds + alpha.

 4. Display graphs using DBS viewer.


9. How to Write The Command File 

9.1 Command File Samples

There are command file samples.  command file is a free format,so new lines
and blanks can be written freely. '#' means to begin comments. # to the end
of lines are interpreted with comments.

  # Sample 1
  {                                             
          sender {
                  hostname  = host1;
                  hostname_cmd  = host1;
                  port      = 0;
                  so_debug  = OFF;
                  tcp_trace = OFF;
                  send_buff = 65535;
                  recv_buff = 65535;Sample 2 
                  mem_align = 2048;
                  pattern {2048,   2048,     0.0,      0.0}
          }
          receiver {
                  hostname  = host2;
                  hostname_cmd  = host2;
                  port      = 20001;
                  so_debug  = OFF;
                  tcp_trace = OFF;
                  recv_buff = 65535;
                  send_buff = 65535;
                  mem_align = 2048;
                  pattern {2048,   2048,     0.0,      0.0}
          }
          file           = ether/test1;
          protocol       = TCP;
          start_time     = 0.0;
          connection_mode= BEFORE;
          end_time       = 30;
          send_times     = 2048;
  }

  # Sample 2 
  # MPEG1 Traffic Model
  {                                             
          sender {
                  hostname  = host1;
                  port      = 12300;
                  mem_align = 2048;
                  pattern { 40000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;
                            10000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;
                            10000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;
                            10000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;
                             2000,  1024,    0.04,      0;}
            }
            receiver {
                    hostname  = host2;
                    port      = 12300;
                    mem_align = 2048;
                    pattern {2048,   2048,     0.0,      0.0}
            }
            file           = ether/test2;
            protocol       = UDP;
            start_time     = 0.0;
            end_time       = 30;
            send_times     = 100;
  }


9.2 Format of the command file 

 Format of the command file is as follows.  When multiple data streams are
 transfered in the same test, as many configurations should be written in
 the same file.  Please be careful port number and output file name in each
 configuration should not be the same.

  # First Configuration 
  { 
    sender   {'configurations of sender';   'sending traffic pattern(s)';}
    receiver {'configurations of receiver'; 'receiving message pattern(s)';} 
    'configurations of the connection';
  }
 
  # Second Configuration 
  { 
    sender   {'configurations of sender';   'sending traffic pattern(s)';}
    receiver {'configurations of receiver'; 'receiving message pattern(s)';} 
    'configurations of the connection';
  }
  .
  .
  .

9.2.1 Format of 'sending traffic pattern(s)'

  pattern 
  {
      'frame size', 'message size', 'interval', 'wait time';
      'frame size', 'message size', 'interval', 'wait time';
      . . . . ;
      . . . . ;
  }

  'data size':    an application data size 
  'message size': send size by one system call
  'interval':     time interval of sending the next message
  'wait time':    system overhead

  data size             |<-------------->|     |<-------->|   |<----------->|

  message size          |<-->|<-->|<-->|       |<->|<->|      |<---->|<---->|

                        +----+----+----+-+     +---+---+--+   +------+------+
  transmission segments |    |    |    | |     |   |   |  |   |      |      |
             (Datagrams)+----+----+----+-+     +---+---+--+   +------+------+

  wait time                              |<--->|          |<->|

  interval              |<-------------------->|<------------>|

  Time Transition   -------------------------------------------------------->


9.2.2 Format of 'receiving message pattern(s)'

  pattern {
      'buffer', 'message size', 'interval', 'wait time';
      'buffer', 'message size', 'interval', 'wait time';
      . . . . ;
      . . . . ;
  }

  'buffer size':  receive buffer size of the application
  'message size': size of receiving message at once
  'interval':     time interval of receiving the next message
  'Wait time':    system overhead

9.2.3 Format of 'configurations of sender' and 'configurations of receiver'

 hostname        experiment host name, interface name or IP address
 hostname_cmd    Interface name or IP address to receive commands.
                 (if hostname is the same hostname_cmd, this option can 
                 be omitted)
 port            port number used on experiment.
                 port number of client can be set to 0. In this case
                 OS selects the port number correctly.
 so_debug        ON :   use so_debug option.
                 OFF:   don't use so_debug option (default)
 tcp_trace       ON :   records TCP Control Block in the kernel every 
                        time when DBS calls the system call "send" or "recv".
                 OFF:   don't record TCP Control Block (default)
                 AFTER: records TCP Control Block after a transmission
                        or a receipt is end.
                        In order to use this option, it is necessary
                        that TCP_NDEEBUG of /sys/netinet/tcp_debug.h 
                        are set enough large number and rebuilded the kernel.
 no_delay        ON :   use no_delay option.
                 OFF:   don't use no_delay option (default)
 mss             size of segment size (OS depending)
 recv_buff       size of the receive buffer 
                 (default size is dependent host)
 send_buff       size of the transmission buffer 
                 (default size is dependent host).
 mem_align       arrange the address of transmission buffer or receipt 
                 buffer in page boundary.
 file            save file name.


9.2.4 configuration of connection

 protocol        transport protocol: TCP or UDP
 start_time      transmission start time (second).  
                 about 0.01 seconds level precision.
 connection_mode AFTER :   measure establishment of connection.
                 BEFORE:   connection establishment is before data transfering
                           which is measured.(default)
 server          RECEIVER: connection establishment
                           Sender(connect) -> Receiver(accept)
                 SENDER :   connection establishment
                           Receiver(connect) -> Sender(accept)
                           You may use this option if you have Fire Wall which
                           don't accept from forein TCP SYN packet.
 end_ time       End or break time of the measurement (second).
 send_times      Number of sending the traffic pattern


10. Output file

  DBS output files as follows:

  {file_name}.t       send/receive time of Application level
  {file_name}.c       connection establish/close time of Application level
  {file_name}.td_s_#  TCP DEBUG data of sender host
  {file_name}.td_r_#  TCP DEBUG data of receiver host

     # means as follows:
       0  TA_INPUT     when segment sends
       1  TA_OUTPUT    when segment receives
       2  TA_USER      when user system call
       3  TA_RESPOND   
       4  TA_DROP      when segment drops

10.1 send/receive time of application level(.t)

  send_sequence   send sequence number
  send_size       send size(byte)
  send_time       send time(second)
  recv_sequence   receive sequence number
  recv_size       receive size(byte)
  recv_time       receive time(second)

10.2 connection establish/close time of application level(.c)

  line 1 client                       server
  line 2 connection establish start   time receive connection establish time
  line 3 connection established time  connection close start time
  line 4 connection close start time  connection closed time
  line 5 connection closed time       connection closed time

10.3 results of tcp_trace(.td_s_# .td_r_#)

  td_time         iptime() :ms since midnight, UTC
  act             TA_DROP, TA_INPUT, TA_OUTPUT, TA_USER
  th_seq          sequence number
  th_ack          acknowledgement number
  len             data length
  th_flags        ACK, FIN, PUSH, RST, SYN, URG
  t_state         connection state
  t_rxtshift      log(2) of rexmt exp. backoff
  t_rxtcur        current retransmission timeout
  t_dupacks       #consecutive duplicate ACKs received
  t_maxseg        maximum segment size to send
  t_force         1 if forcing out a byte
  t_flags         TF_ACKNOW, TF_DELACK, TF_NODELAY, ...
  snd_una         send unacknowledged
  snd_nxt         send next
  snd_wl1         window update seg seq number
  snd_wl2         window update seg ack number
  iss             initial send sequence number
  snd_wnd         send window
  rcv_wnd         receive window
  rcv_nxt         receive next
  irs             initial receive sequence number
  rcv_adv         advertised window by other end
  snd_max         highest sequence number sent
  snd_cwnd        congestion-controlled window
  snd_ssthresh    snd_cwnd size threshhold
  t_idle          inactivity time
  t_rtt           round-trip time
  t_rtseq         sequence number being timed
  t_srtt          smoothed round-trip time
  t_rttvar        variance in round-trip time
  max_sndwnd      largest window peer has offered

  When you need more information about these, I recommend you read the book
  as follows.

  Gray R. Wright, W. Richard Stevens, 
  "TCP/IP Illustrated, Volume2 --The Implementation--",
  Addison-Wesley Publishing Company, 1995, 
  ISBN 0-201-63354-X

