As per Relevance of the word specific, we have this rfc below:
Network Working Group S.
Request for Comments: 2544 Harvard
Obsoletes: 1944 J.
Category: Informational NetScout
March 1999
Benchmarking Methodology for Network Interconnect
Status of this
This memo provides information for the Internet community. It
not specify an Internet standard of any kind. Distribution of
memo is unlimited
Copyright
Copyright (C) The Internet Society (1999). All Rights Reserved
IESG
This document is a republication of RFC 1944 correcting the
for the IP addresses which were assigned to be used as the
addresses for networking test equipment. (See section C.2.2 ).
RFC replaces and obsoletes RFC 1944.
This document discusses and defines a number of tests that may
used to describe the performance characteristics of a
interconnecting device. In addition to defining the tests
document also describes specific formats for reporting the results
the tests. Appendix A lists the tests and conditions that we
should be included for specific cases and gives
information about testing practices. Appendix B is a
listing of maximum frame rates to be used with specific frame
on various media and Appendix C gives some examples of frame
to be used in testing
1.
Vendors often engage in "specsmanship" in an attempt to give
products a better position in the marketplace. This often
"smoke & mirrors" to confuse the potential users of the products
Bradner & McQuaid Informational [Page 1]
RFC 2544 Benchmarking Methodology March 1999
This document defines a specific set of tests that vendors can use
measure and report the performance characteristics of
devices. The results of these tests will provide the user
data from different vendors with which to evaluate these devices
A previous document, "Benchmarking Terminology for
Interconnect Devices" (RFC 1242), defined many of the terms that
used in this document. The terminology document should be
before attempting to make use of this document
2. Real
In producing this document the authors attempted to keep in mind
requirement that apparatus to perform the described tests
actually be built. We do not know of "off the shelf"
available to implement all of the tests but it is our opinion
such equipment can be constructed
3. Tests to be
There are a number of tests described in this document. Not all
the tests apply to all types of devices under test (DUTs).
should perform all of the tests that can be supported by a
type of product. The authors understand that it will take
considerable period of time to perform all of the recommended
nder all of the recommended conditions. We believe that the
are worth the effort. Appendix A lists some of the tests
conditions that we believe should be included for specific cases
4. Evaluating the
Performing all of the recommended tests will result in a great
of data. Much of this data will not apply to the evaluation of
devices under each circumstance. For example, the rate at which
router forwards IPX frames will be of little use in selecting
router for an environment that does not (and will not) support
protocol. Evaluating even that data which is relevant to
particular network installation will require experience which may
be readily available. Furthermore, selection of the tests to be
and evaluation of the test data must be done with an understanding
generally accepted testing practices regarding repeatability
variance and statistical significance of small numbers of trials
Bradner & McQuaid Informational [Page 2]
RFC 2544 Benchmarking Methodology March 1999
5.
In this document, the words that are used to define the
of each particular requirement are capitalized. These words are
* "MUST" This word, or the words "REQUIRED" and "SHALL" mean
the item is an absolute requirement of the specification
* "SHOULD" This word or the adjective "RECOMMENDED" means
there may exist valid reasons in particular circumstances
ignore this item, but the full implications should
understood and the case carefully weighed before choosing
different course
* "MAY" This word or the adjective "OPTIONAL" means that
item is truly optional. One vendor may choose to include
item because a particular marketplace requires it or because
enhances the product, for example; another vendor may omit
same item
An implementation is not compliant if it fails to satisfy one or
of the MUST requirements for the protocols it implements.
implementation that satisfies all the MUST and all the
requirements for its protocols is said to be "
compliant"; one that satisfies all the MUST requirements but not
the SHOULD requirements for its protocols is said to
"conditionally compliant".
6. Test set
The ideal way to implement this series of tests is to use a
with both transmitting and receiving ports. Connections are
from the sending ports of the tester to the receiving ports of
DUT and from the sending ports of the DUT back to the tester. (
Figure 1) Since the tester both sends the test traffic and
it back, after the traffic has been forwarded but the DUT, the
can easily determine if all of the transmitted packets were
and verify that the correct packets were received. The
functionality can be obtained with separate transmitting
receiving devices (see Figure 2) but unless they are
controlled by some computer in a way that simulates the
tester, the labor required to accurately perform some of the
(particularly the throughput test) can be prohibitive
Bradner & McQuaid Informational [Page 3]
RFC 2544 Benchmarking Methodology March 1999
+------------+
| |
+------------| tester |<-------------+
| | | |
| +------------+ |
| |
| +------------+ |
| | | |
+----------->| DUT |--------------+
| |
+------------+
Figure 1
+--------+ +------------+ +----------+
| | | | | |
| sender |-------->| DUT |--------->| receiver |
| | | | | |
+--------+ +------------+ +----------+
Figure 2
6.1 Test set up for multiple media
Two different setups could be used to test a DUT which is used
real-world networks to connect networks of differing media type
local Ethernet to a backbone FDDI ring for example. The tester
support both media types in which case the set up shown in Figure 1
would be used
Two identical DUTs are used in the other test set up. (see Figure 3)
In many cases this set up may more accurately simulate the
world. For example, connecting two LANs together with a WAN link
high speed backbone. This set up would not be as good at
a system where clients on a Ethernet LAN were interacting with
server on an FDDI backbone
+-----------+
| |
+---------------------| tester |<---------------------+
| | | |
| +-----------+ |
| |
| +----------+ +----------+ |
| | | | | |
+------->| DUT 1 |-------------->| DUT 2 |---------+
| | | |
+----------+ +----------+
Figure 3
Bradner & McQuaid Informational [Page 4]
RFC 2544 Benchmarking Methodology March 1999
7. DUT set
Before starting to perform the tests, the DUT to be tested MUST
configured following the instructions provided to the user
Specifically, it is expected that all of the supported protocols
be configured and enabled during this set up (See Appendix A). It
expected that all of the tests will be run without changing
configuration or setup of the DUT in any way other than that
to do the specific test. For example, it is not acceptable to
the size of frame handling buffers between tests of frame
rates or to disable all but one transport protocol when testing
throughput of that protocol. It is necessary to modify
configuration when starting a test to determine the effect of
on throughput, but the only change MUST be to enable the
filter. The DUT set up SHOULD include the normally
routing update intervals and keep alive frequency. The
version of the software and the exact DUT configuration,
what functions are disabled, used during the tests MUST be
as part of the report of the results
8. Frame
The formats of the test frames to use for TCP/IP over Ethernet
shown in Appendix C: Test Frame Formats. These exact frame
SHOULD be used in the tests described in this document for
protocol/media combination and that these frames will be used as
template for testing other protocol/media combinations. The
formats that are used to define the test frames for a particular
series MUST be included in the report of the results
9. Frame
All of the described tests SHOULD be performed at a number of
sizes. Specifically, the sizes SHOULD include the maximum and
legitimate sizes for the protocol under test on the media under
and enough sizes in between to be able to get a full
of the DUT performance. Except where noted, at least five
sizes SHOULD be tested for each test condition
Theoretically the minimum size UDP Echo request frame would
of an IP header (minimum length 20 octets), a UDP header (8 octets
and whatever MAC level header is required by the media in use.
theoretical maximum frame size is determined by the size of
length field in the IP header. In almost all cases the
maximum and minimum sizes are determined by the limitations of
media
Bradner & McQuaid Informational [Page 5]
RFC 2544 Benchmarking Methodology March 1999
In theory it would be ideal to distribute the frame sizes in a
that would evenly distribute the theoretical frame rates.
recommendations incorporate this theory but specify frame sizes
are easy to understand and remember. In addition, many of the
frame sizes are specified on each of the media types to allow
easy performance comparisons
Note: The inclusion of an unrealistically small frame size on some
the media types (i.e. with little or no space for data) is to
characterize the per-frame processing overhead of the DUT
9.1 Frame sizes to be used on
64, 128, 256, 512, 1024, 1280, 1518
These sizes include the maximum and minimum frame sizes permitted
the Ethernet standard and a selection of sizes between these
with a finer granularity for the smaller frame sizes and higher
rates
9.2 Frame sizes to be used on 4Mb and 16Mb token
54, 64, 128, 256, 1024, 1518, 2048, 4472
The frame size recommendations for token ring assume that there is
RIF field in the frames of routed protocols. A RIF field would
present in any direct source route bridge performance test.
minimum size frame for UDP on token ring is 54 octets. The
size of 4472 octets is recommended for 16Mb token ring instead of
theoretical size of 17.9Kb because of the size limitations imposed
many token ring interfaces. The reminder of the sizes are
to permit direct comparisons with other types of media. An IP (i.e
not UDP) frame may be used in addition if a higher data rate
desired, in which case the minimum frame size is 46 octets
9.3 Frame sizes to be used on
54, 64, 128, 256, 1024, 1518, 2048, 4472
The minimum size frame for UDP on FDDI is 53 octets, the minimum
of 54 is recommended to allow direct comparison to token
performance. The maximum size of 4472 is recommended instead of
theoretical maximum size of 4500 octets to permit the same type
comparison. An IP (i.e. not UDP) frame may be used in addition if
higher data rate is desired, in which case the minimum frame size
45 octets
Bradner & McQuaid Informational [Page 6]
RFC 2544 Benchmarking Methodology March 1999
9.4 Frame sizes in the presence of disparate
When the interconnect DUT supports connecting links with
MTUs, the frame sizes for the link with the *larger* MTU SHOULD
used, up to the limit of the protocol being tested. If
interconnect DUT does not support the fragmenting of frames in
presence of MTU mismatch, the forwarding rate for that frame
shall be reported as zero
For example, the test of IP forwarding with a bridge or router
joins FDDI and Ethernet should use the frame sizes of FDDI when
from the FDDI to the Ethernet link. If the bridge does not support
fragmentation, the forwarding rate for those frames too large
Ethernet should be reported as zero
10. Verifying received
The test equipment SHOULD discard any frames received during a
run that are not actual forwarded test frames. For example, keep
alive and routing update frames SHOULD NOT be included in the
of received frames. In any case, the test equipment SHOULD
the length of the received frames and check that they match
expected length
Preferably, the test equipment SHOULD include sequence numbers in
transmitted frames and check for these numbers on the
frames. If this is done, the reported results SHOULD include
addition to the number of frames dropped, the number of frames
were received out of order, the number of duplicate frames
and the number of gaps in the received frame numbering sequence
This functionality is required for some of the described tests
11.
It might be useful to know the DUT performance under a number
conditions; some of these conditions are noted below. The
results SHOULD include as many of these conditions as the
equipment is able to generate. The suite of tests SHOULD be
run without any modifying conditions and then repeated under each
the conditions separately. To preserve the ability to compare
results of these tests any frames that are required to generate
modifying conditions (management queries for example) will
included in the same data stream as the normal test frames in
of one of the test frames and not be supplied to the DUT on
separate network port
Bradner & McQuaid Informational [Page 7]
RFC 2544 Benchmarking Methodology March 1999
11.1 Broadcast
In most router designs special processing is required when
addressed to the hardware broadcast address are received. In
(or in bridge mode on routers) these broadcast frames must be
to a number of ports. The stream of test frames SHOULD be
with 1% frames addressed to the hardware broadcast address.
frames sent to the broadcast address should be of a type that
router will not need to process. The aim of this test is
determine if there is any effect on the forwarding rate of the
data in the stream. The specific frames that should be used
included in the test frame format document. The broadcast
SHOULD be evenly distributed throughout the data stream, for example
every 100th frame
The same test SHOULD be performed on bridge-like DUTs but in
case the broadcast packets will be processed and flooded to
outputs
It is understood that a level of broadcast frames of 1% is
higher than many networks experience but, as in drug
evaluations, the higher level is required to be able to gage
effect which would otherwise often fall within the normal
of the system performance. Due to design factors some test
will not be able to generate a level of alternate frames this low
In these cases the percentage SHOULD be as small as the equipment
provide and that the actual level be described in the report of
test results
11.2 Management
Most data networks now make use of management protocols such as SNMP
In many environments there can be a number of management
sending queries to the same DUT at the same time
The stream of test frames SHOULD be augmented with one
query as the first frame sent each second during the duration of
trial. The result of the query must fit into one response frame.
response frame SHOULD be verified by the test equipment. One
of the specific query frame that should be used is shown in
C
11.3 Routing update
The processing of dynamic routing protocol updates could have
significant impact on the ability of a router to forward data frames
The stream of test frames SHOULD be augmented with one routing
frame transmitted as the first frame transmitted during the trial
Bradner & McQuaid Informational [Page 8]
RFC 2544 Benchmarking Methodology March 1999
Routing update frames SHOULD be sent at the rate specified
Appendix C for the specific routing protocol being used in the test
Two routing update frames are defined in Appendix C for the TCP/
over Ethernet example. The routing frames are designed to change
routing to a number of networks that are not involved in
forwarding of the test data. The first frame sets the routing
state to "A", the second one changes the state to "B". The
MUST be alternated during the trial
The test SHOULD verify that the routing update was processed by
DUT
11.4
Filters are added to routers and bridges to selectively inhibit
forwarding of frames that would normally be forwarded. This
usually done to implement security controls on the data that
accepted between one area and another. Different products
different capabilities to implement filters
The DUT SHOULD be first configured to add one filter condition
the tests performed. This filter SHOULD permit the forwarding of
test data stream. In routers this filter SHOULD be of the form
forward input_protocol_address to output_protocol_
In bridges the filter SHOULD be of the form
forward destination_hardware_
The DUT SHOULD be then reconfigured to implement a total of 25
filters. The first 24 of these filters SHOULD be of the form
block input_protocol_address to output_protocol_
The 24 input and output protocol addresses SHOULD not be any that
represented in the test data stream. The last filter SHOULD
the forwarding of the test data stream. By "first" and "last"
mean to ensure that in the second case, 25 conditions must be
before the data frames will match the conditions that permit
forwarding of the frame. Of course, if the DUT reorders the
or does not use a linear scan of the filter rules the effect of
sequence in which the filters are input is properly lost
The exact filters configuration command lines used SHOULD be
with the report of the results
Bradner & McQuaid Informational [Page 9]
RFC 2544 Benchmarking Methodology March 1999
11.4.1 Filter
Two sets of filter addresses are required, one for the single
case and one for the 25 filter case
The single filter case should permit traffic from IP
198.18.1.2 to IP address 198.19.65.2 and deny all other traffic
The 25 filter case should follow the following sequence
deny aa.ba.1.1 to aa.ba.100.1
deny aa.ba.2.2 to aa.ba.101.2
deny aa.ba.3.3 to aa.ba.103.3
...
deny aa.ba.12.12 to aa.ba.112.12
allow aa.bc.1.2 to aa.bc.65.1
deny aa.ba.13.13 to aa.ba.113.13
deny aa.ba.14.14 to aa.ba.114.14
...
deny aa.ba.24.24 to aa.ba.124.24
deny all
All previous filter conditions should be cleared from the
before this sequence is entered. The sequence is selected to test
see if the router sorts the filter conditions or accepts them in
order that they were entered. Both of these procedures will
in a greater impact on performance than will some form of
coding
12. Protocol
It is easier to implement these tests using a single logical
of data, with one source protocol address and one
protocol address, and for some conditions like the filters
above, a practical requirement. Networks in the real world are
limited to single streams of data. The test suite SHOULD be first
with a single protocol (or hardware for bridge tests) source
destination address pair. The tests SHOULD then be repeated
using a random destination address. While testing routers
addresses SHOULD be random and uniformly distributed over a range
256 networks and random and uniformly distributed over the full
range for bridges. The specific address ranges to use for IP
shown in Appendix C
Bradner & McQuaid Informational [Page 10]
RFC 2544 Benchmarking Methodology March 1999
13. Route Set
It is not reasonable that all of the routing information necessary
forward the test stream, especially in the multiple address case
will be manually set up. At the start of each trial a routing
MUST be sent to the DUT. This routing update MUST include all of
network addresses that will be required for the trial. All of
addresses SHOULD resolve to the same "next-hop". Normally this
be the address of the receiving side of the test equipment.
routing update will have to be repeated at the interval required
the routing protocol being used. An example of the format
repetition interval of the update frames is given in Appendix C
14. Bidirectional
Normal network activity is not all in a single direction. To
the bidirectional performance of a DUT, the test series SHOULD be
with the same data rate being offered from each direction. The sum
the data rates should not exceed the theoretical limit for the media
15. Single stream
The full suite of tests SHOULD be run along with whatever
conditions that are relevant using a single input and output
port on the DUT. If the internal design of the DUT has
distinct pathways, for example, multiple interface cards each
multiple network ports, then all possible types of pathways SHOULD
tested separately
16. Multi-
Many current router and bridge products provide many network ports
the same module. In performing these tests first half of the
are designated as "input ports" and half are designated as "
ports". These ports SHOULD be evenly distributed across the
architecture. For example if a DUT has two interface cards each
which has four ports, two ports on each interface card are
as input and two are designated as output. The specified tests
run using the same data rate being offered to each of the
ports. The addresses in the input data streams SHOULD be set so
a frame will be directed to each of the output ports in sequence
that all "output" ports will get an even distribution of packets
this input. The same configuration MAY be used to perform
bidirectional multi-stream test. In this case all of the ports
considered both input and output ports and each data stream
consist of frames addressed to all of the other ports
Bradner & McQuaid Informational [Page 11]
RFC 2544 Benchmarking Methodology March 1999
Consider the following 6 port DUT
--------------
---------| in A out X|--------
---------| in B out Y|--------
---------| in C out Z|--------
--------------
The addressing of the data streams for each of the inputs SHOULD be
stream sent to input A
packet to out X, packet to out Y, packet to out
stream sent to input B
packet to out X, packet to out Y, packet to out
stream sent to input
packet to out X, packet to out Y, packet to out
Note that these streams each follow the same sequence so that 3
packets will arrive at output X at the same time, then 3 packets
Y, then 3 packets at Z. This procedure ensures that, as in the
world, the DUT will have to deal with multiple packets addressed
the same output at the same time
17. Multiple
This document does not address the issue of testing the effects of
mixed protocol environment other than to suggest that if such
are wanted then frames SHOULD be distributed between all of the
protocols. The distribution MAY approximate the conditions on
network in which the DUT would be used
18. Multiple frame
This document does not address the issue of testing the effects of
mixed frame size environment other than to suggest that if such
are wanted then frames SHOULD be distributed between all of
listed sizes for the protocol under test. The distribution
approximate the conditions on the network in which the DUT would
used. The authors do not have any idea how the results of such a
would be interpreted other than to directly compare multiple DUTs
some very specific simulated network
19. Testing performance beyond a single DUT
In the performance testing of a single DUT, the paradigm can
described as applying some input to a DUT and monitoring the output
The results of which can be used to form a basis of
of that device under those test conditions
Bradner & McQuaid Informational [Page 12]
RFC 2544 Benchmarking Methodology March 1999
This model is useful when the test input and output are
(e.g., 64-byte IP, 802.3 frames into the DUT; 64 byte IP, 802.3
frames out), or the method of test can distinguish between
input/output. (E.g., 1518 byte IP, 802.3 frames in; 576 byte
fragmented IP, X.25 frames out.)
By extending the single DUT test model, reasonable
regarding multiple DUTs or heterogeneous environments may
collected. In this extension, the single DUT is replaced by a
of interconnected network DUTs. This test methodology would
the benchmarking of a variety of device/media/service/
combinations. For example, a configuration for a LAN-to-WAN-to-
test might be
(1) 802.3-> DUT 1 -> X.25 @ 64kbps -> DUT 2 -> 802.3
Or a mixed LAN configuration might be
(2) 802.3 -> DUT 1 -> FDDI -> DUT 2 -> FDDI -> DUT 3 -> 802.3
In both examples 1 and 2, end-to-end benchmarks of each system
be empirically ascertained. Other behavior may be
through the use of intermediate devices. In example 2,
configuration may be used to give an indication of the FDDI to
capability exhibited by DUT 2.
Because multiple DUTs are treated as a single system, there
limitations to this methodology. For instance, this methodology
yield an aggregate benchmark for a tested system. That
alone, however, may not necessarily reflect asymmetries in
between the DUTs, latencies introduce by other apparatus (e.g.,
CSUs/DSUs, switches), etc
Further, care must be used when comparing benchmarks of
systems by ensuring that the DUTs' features/configuration of
tested systems have the appropriate common denominators to
comparison
20. Maximum frame
The maximum frame rates that should be used when testing
connections SHOULD be the listed theoretical maximum rate for
frame size on the media
Bradner & McQuaid Informational [Page 13]
RFC 2544 Benchmarking Methodology March 1999
The maximum frame rate that should be used when testing
connections SHOULD be greater than the listed theoretical
rate for the frame size on that speed connection. The higher
for WAN tests is to compensate for the fact that some vendors
various forms of header compression
A list of maximum frame rates for LAN connections is included
Appendix B
21. Bursty
It is convenient to measure the DUT performance under steady
load but this is an unrealistic way to gauge the functioning of a
since actual network traffic normally consists of bursts of frames
Some of the tests described below SHOULD be performed with
steady state traffic and with traffic consisting of repeated
of frames. The frames within a burst are transmitted with
minimum legitimate inter-frame gap
The objective of the test is to determine the minimum
between bursts which the DUT can process with no frame loss.
each test the number of frames in each burst is held constant and
inter-burst interval varied. Tests SHOULD be run with burst sizes
16, 64, 256 and 1024 frames
22. Frames per
Although it is possible to configure some token ring and
interfaces to transmit more than one frame each time that the
is received, most of the network devices currently available
only one frame per token. These tests SHOULD first be
while transmitting only one frame per token
Some current high-performance workstation servers do transmit
than one frame per token on FDDI to maximize throughput. Since
may be a common feature in future workstations and servers
interconnect devices with FDDI interfaces SHOULD be tested with 1, 4,
8, and 16 frames per token. The reported frame rate SHOULD be
average rate of frame transmission over the total trial period
23. Trial
A particular test consists of multiple trials. Each trial
one piece of information, for example the loss rate at a
input frame rate. Each trial consists of a number of phases
a) If the DUT is a router, send the routing update to the "input
port and pause two seconds to be sure that the routing has settled
Bradner & McQuaid Informational [Page 14]
RFC 2544 Benchmarking Methodology March 1999
b) Send the "learning frames" to the "output" port and wait 2
seconds to be sure that the learning has settled. Bridge
frames are frames with source addresses that are the same as
destination addresses used by the test frames. Learning frames
other protocols are used to prime the address resolution tables
the DUT. The formats of the learning frame that should be used
shown in the Test Frame Formats document
c) Run the test trial
d) Wait for two seconds for any residual frames to be received
e) Wait for at least five seconds for the DUT to restabilize
24. Trial
The aim of these tests is to determine the rate
supportable by the DUT. The actual duration of the test trials
be a compromise between this aim and the duration of the
test suite. The duration of the test portion of each trial SHOULD
at least 60 seconds. The tests that involve some form of "
search", for example the throughput test, to determine the
result MAY use a shorter trial duration to minimize the length of
search procedure, but the final determination SHOULD be made
full length trials
25. Address
The DUT SHOULD be able to respond to address resolution requests
by the DUT wherever the protocol requires such a process
26. Benchmarking tests
Note: The notation "type of data stream" refers to the
modifications to a frame stream with a constant inter-frame gap,
example, the addition of traffic filters to the configuration of
DUT
26.1
Objective: To determine the DUT throughput as defined in RFC 1242.
Procedure: Send a specific number of frames at a specific
through the DUT and then count the frames that are transmitted by
DUT. If the count of offered frames is equal to the count of
frames, the fewer frames are received than were transmitted, the
of the offered stream is reduced and the test is rerun
Bradner & McQuaid Informational [Page 15]
RFC 2544 Benchmarking Methodology March 1999
The throughput is the fastest rate at which the count of test
transmitted by the DUT is equal to the number of test frames sent
it by the test equipment
Reporting format: The results of the throughput test SHOULD
reported in the form of a graph. If it is, the x coordinate SHOULD
the frame size, the y coordinate SHOULD be the frame rate.
SHOULD be at least two lines on the graph. There SHOULD be one
showing the theoretical frame rate for the media at the various
sizes. The second line SHOULD be the plot of the test results
Additional lines MAY be used on the graph to report the results
each type of data stream tested. Text accompanying the graph
indicate the protocol, data stream format, and type of media used
the tests
We assume that if a single value is desired for advertising
the vendor will select the rate for the minimum frame size for
media. If this is done then the figure MUST be expressed in
per second. The rate MAY also be expressed in bits (or bytes)
second if the vendor so desires. The statement of performance
include a/ the measured maximum frame rate, b/ the size of the
used, c/ the theoretical limit of the media for that frame size,
d/ the type of protocol used in the test. Even if a single value
used as part of the advertising copy, the full table of
SHOULD be included in the product data sheet
26.2
Objective: To determine the latency as defined in RFC 1242.
Procedure: First determine the throughput for DUT at each of
listed frame sizes. Send a stream of frames at a particular
size through the DUT at the determined throughput rate to a
destination. The stream SHOULD be at least 120 seconds in duration
An identifying tag SHOULD be included in one frame after 60
with the type of tag being implementation dependent. The time
which this frame is fully transmitted is recorded (timestamp A).
receiver logic in the test equipment MUST recognize the
information in the frame stream and record the time at which
tagged frame was received (timestamp B).
The latency is timestamp B minus timestamp A as per the
definition frm RFC 1242, namely latency as defined for store
forward devices or latency as defined for bit forwarding devices
The test MUST be repeated at least 20 times with the reported
being the average of the recorded values
Bradner & McQuaid Informational [Page 16]
RFC 2544 Benchmarking Methodology March 1999
This test SHOULD be performed with the test frame addressed to
same destination as the rest of the data stream and also with each
the test frames addressed to a new destination network
Reporting format: The report MUST state which definition of
(from RFC 1242) was used for this test. The latency results
be reported in the format of a table with a row for each of
tested frame sizes. There SHOULD be columns for the frame size,
rate at which the latency test was run for that frame size, for
media types tested, and for the resultant latency values for
type of data stream tested
26.3 Frame loss
Objective: To determine the frame loss rate, as defined in RFC 1242,
of a DUT throughout the entire range of input data rates and
sizes
Procedure: Send a specific number of frames at a specific
through the DUT to be tested and count the frames that
transmitted by the DUT. The frame loss rate at each point
calculated using the following equation
( ( input_count - output_count ) * 100 ) / input_
The first trial SHOULD be run for the frame rate that corresponds
100% of the maximum rate for the frame size on the input media
Repeat the procedure for the rate that corresponds to 90% of
maximum rate used and then for 80% of this rate. This
SHOULD be continued (at reducing 10% intervals) until there are
successive trials in which no frames are lost. The
granularity of the trials MUST be 10% of the maximum rate, a
granularity is encouraged
Reporting format: The results of the frame loss rate test SHOULD
plotted as a graph. If this is done then the X axis MUST be
input frame rate as a percent of the theoretical rate for the
at the specific frame size. The Y axis MUST be the percent loss
the particular input rate. The left end of the X axis and the
of the Y axis MUST be 0 percent; the right end of the X axis and
top of the Y axis MUST be 100 percent. Multiple lines on the
MAY used to report the frame loss rate for different frame sizes
protocols, and types of data streams
Note: See section 18 for the maximum frame rates that SHOULD be used
Bradner & McQuaid Informational [Page 17]
RFC 2544 Benchmarking Methodology March 1999
26.4 Back-to-back
Objective: To characterize the ability of a DUT to process back-to
back frames as defined in RFC 1242.
Procedure: Send a burst of frames with minimum inter-frame gaps
the DUT and count the number of frames forwarded by the DUT. If
count of transmitted frames is equal to the number of
forwarded the length of the burst is increased and the test is rerun
If the number of forwarded frames is less than the
transmitted, the length of the burst is reduced and the test
rerun
The back-to-back value is the number of frames in the longest
that the DUT will handle without the loss of any frames. The
length MUST be at least 2 seconds and SHOULD be repeated at least 50
times with the average of the recorded values being reported
Reporting format: The back-to-back results SHOULD be reported in
format of a table with a row for each of the tested frame sizes
There SHOULD be columns for the frame size and for the
average frame count for each type of data stream tested.
standard deviation for each measurement MAY also be reported
26.5 System
Objective: To characterize the speed at which a DUT recovers from
overload condition
Procedure: First determine the throughput for a DUT at each of
listed frame sizes
Send a stream of frames at a rate 110% of the recorded
rate or the maximum rate for the media, whichever is lower, for
least 60 seconds. At Timestamp A reduce the frame rate to 50% of
above rate and record the time of the last frame lost (Timestamp B).
The system recovery time is determined by subtracting Timestamp
from Timestamp A. The test SHOULD be repeated a number of times
the average of the recorded values being reported
Reporting format: The system recovery results SHOULD be reported
the format of a table with a row for each of the tested frame sizes
There SHOULD be columns for the frame size, the frame rate used
the throughput rate for each type of data stream tested, and for
measured recovery time for each type of data stream tested
Bradner & McQuaid Informational [Page 18]
RFC 2544 Benchmarking Methodology March 1999
26.6
Objective: To characterize the speed at which a DUT recovers from
device or software reset
Procedure: First determine the throughput for the DUT for
minimum frame size on the media used in the testing
Send a continuous stream of frames at the determined throughput
for the minimum sized frames. Cause a reset in the DUT. Monitor
output until frames begin to be forwarded and record the time
the last frame (Timestamp A) of the initial stream and the
frame of the new stream (Timestamp B) are received. A
interruption reset test is performed as above except that the
to the DUT should be interrupted for 10 seconds in place of causing
reset
This test SHOULD only be run using frames addressed to
directly connected to the DUT so that there is no requirement
delay until a routing update is received
The reset value is obtained by subtracting Timestamp A from
B
Hardware and software resets, as well as a power interruption
be tested
Reporting format: The reset value SHOULD be reported in a simple
of statements, one for each reset type
27. Security
Security issues are not addressed in this document
Bradner & McQuaid Informational [Page 19]
RFC 2544 Benchmarking Methodology March 1999
28. Editors'
Scott
Harvard
1350 Mass. Ave, room 813
Cambridge, MA 02138
Phone: +1 617 495-3864
Fax: +1 617 496-8500
EMail: sob@harvard.
Jim
NetScout
4 Westford Tech Park
Westford, MA 01886
Phone: +1 978 614-4116
Fax: +1 978 614-4004
EMail: mcquaidj@netscout.
Bradner & McQuaid Informational [Page 20]
RFC 2544 Benchmarking Methodology March 1999
Appendix A: Testing
A.1 Scope Of This
This appendix discusses certain issues in the
methodology where experience or judgment may play a role in the
selected to be run or in the approach to constructing the test with
particular DUT. As such, this appendix MUST not be read as
amendment to the methodology described in the body of this
but as a guide to testing practice
1. Typical testing practice has been to enable all protocols to
tested and conduct all testing with no further configuration
protocols, even though a given set of trials may exercise only
protocol at a time. This minimizes the opportunities to "tune"
DUT for a single protocol
2. The least common denominator of the available filter
should be used to ensure that there is a basis for
between vendors. Because of product differences, those
and evaluating tests must make a judgment about this issue
3. Architectural considerations may need to be considered.
example, first perform the tests with the stream going
ports on the same interface card and the repeat the tests with
stream going into a port on one interface card and out of a
on a second interface card. There will almost always be a
case and worst case configuration for a given DUT architecture
4. Testing done using traffic streams consisting of mixed
has not shown much difference between testing with
protocols. That is, if protocol A testing and protocol B
give two different performance results, mixed protocol
appears to give a result which is the average of the two
5. Wide Area Network (WAN) performance may be tested by setting
two identical devices connected by the appropriate short-
versions of the WAN modems. Performance is then measured
a LAN interface on one DUT to a LAN interface on the other DUT
The maximum frame rate to be used for LAN-WAN-LAN configurations is
judgment that can be based on known characteristics of the
system including compression effects, fragmentation, and gross
speeds. Practice suggests that the rate should be at least 110%
the slowest link speed. Substantive issues of testing
itself are beyond the scope of this document
Bradner & McQuaid Informational [Page 21]
RFC 2544 Benchmarking Methodology March 1999
Appendix B: Maximum frame rates
(Provided by Roger Beeman, Cisco Systems
Size Ethernet 16Mb Token Ring
(bytes) (pps) (pps) (pps
64 14880 24691 152439
128 8445 13793 85616
256 4528 7326 45620
512 2349 3780 23585
768 1586 2547 15903
1024 1197 1921 11996
1280 961 1542 9630
1518 812 1302 8138
Ethernet
Preamble 64
Frame 8 x N
Gap 96
16Mb Token Ring
SD 8
AC 8
FC 8
DA 48
SA 48
RI 48 bits ( 06 30 00 12 00 30 )
DSAP 8
SSAP 8
Control 8
Vendor 24
Type 16
Data 8 x ( N - 18)
FCS 32
ED 8
FS 8
Tokens or idles between packets are not
FDDI
Preamble 64
SD 8
FC 8
DA 48
SA 48
Bradner & McQuaid Informational [Page 22]
RFC 2544 Benchmarking Methodology March 1999
DSAP 8
SSAP 8
Control 8
Vendor 24
Type 16
Data 8 x ( N - 18)
FCS 32
ED 4
FS 12
Bradner & McQuaid Informational [Page 23]
RFC 2544 Benchmarking Methodology March 1999
Appendix C: Test Frame
This appendix defines the frame formats that may be used with
tests. It also includes protocol specific parameters for TCP/IP
Ethernet to be used with the tests as an example
C.1.
The general logic used in the selection of the parameters and
design of the frame formats is explained for each case within
TCP/IP section. The same logic has been used in the other sections
Comments are used in these sections only if there is a
specific feature to be explained. Parameters and frame formats
additional protocols can be defined by the reader by using the
logic
C.2. TCP/IP
The following section deals with the TCP/IP protocol suite
C.2.1 Frame Type
An application level datagram echo request is used for the test
frame in the protocols that support such a function. A
protocol is used to minimize the chance that a router might expect
specific session initialization sequence, as might be the case for
reliable stream protocol. A specific defined protocol is used
some routers verify the protocol field and refuse to forward
protocols
For TCP/IP a UDP Echo Request is used
C.2.2 Protocol
Two sets of addresses must be defined: first the addresses
to the router ports, and second the address that are to be used
the frames themselves and in the routing updates
The network addresses 192.18.0.0 through 198.19.255.255 are have
assigned to the BMWG by the IANA for this purpose. This
was made to minimize the chance of conflict in case a testing
were to be accidentally connected to part of the Internet.
specific use of the addresses is detailed below
Bradner & McQuaid Informational [Page 24]
RFC 2544 Benchmarking Methodology March 1999
C.2.2.1 Router port protocol
Half of the ports on a multi-port router are referred to as "input
ports and the other half as "output" ports even though some of
tests use all ports both as input and output. A contiguous series
IP Class C network addresses from 198.18.1.0 to 198.18.64.0 have
assigned for use on the "input" ports. A second series
198.19.1.0 to 198.19.64.0 have been assigned for use on the "output
ports. In all cases the router port is node 1 on the
network. For example, a two port DUT would have an IP address
198.18.1.1 on one port and 198.19.1.1 on the other port
Some of the tests described in the methodology memo make use of
SNMP management connection to the DUT. The management access
for the DUT is assumed to be the first of the "input"
(198.18.1.1).
C.2.2.2 Frame
Some of the described tests assume adjacent network routing (
reboot time test for example). The IP address used in the test
is that of node 2 on the appropriate Class C network. (198.19.1.2
example
If the test involves non-adjacent network routing the phantom
are located at node 10 of each of the appropriate Class C networks
A series of Class C network addresses from 198.18.65.0
198.18.254.0 has been assigned for use as the networks
through the phantom routers on the "input" side of DUT. The
of Class C networks from 198.19.65.0 to 198.19.254.0 have
assigned to be used as the networks visible through the
routers on the "output" side of the DUT
C.2.3 Routing Update
The update interval for each routing protocol is may have to
determined by the specifications of the individual protocol. For
RIP, Cisco IGRP and for OSPF a routing update frame or frames
precede each stream of test frames by 5 seconds. This frequency
sufficient for trial durations of up to 60 seconds. Routing
must be mixed with the stream of test frames if longer trial
are selected. The frequency of updates should be taken from
following table
IP-RIP 30
IGRP 90
OSPF 90
Bradner & McQuaid Informational [Page 25]
RFC 2544 Benchmarking Methodology March 1999
C.2.4 Frame Formats - detailed
C.2.4.1 Learning
In most protocols a procedure is used to determine the
between the protocol node address and the MAC address. The
Resolution Protocol (ARP) is used to perform this function in TCP/IP
No such procedure is required in XNS or IPX because the MAC
is used as the protocol node address
In the ideal case the tester would be able to respond to ARP
from the DUT. In cases where this is not possible an ARP
should be sent to the router's "output" port. This request should
seen as coming from the immediate destination of the test
stream. (i.e. the phantom router (Figure 2) or the end node
adjacent network routing is being used.) It is assumed that
router will cache the MAC address of the requesting device. The
request should be sent 5 seconds before the test frame stream
in each trial. Trial lengths of longer than 50 seconds may
that the router be configured for an extended ARP timeout
+--------+ +------------+
| | | phantom |------ P
IN A------| DUT |------------| |------ P
| | OUT A | router |------ P
+--------+ +------------+
Figure 2
In the case where full routing is being
C.2.4.2 Routing Update
If the test does not involve adjacent net routing the tester
supply proper routing information using a routing update. A
routing update is used before each trial on each "destination"
(see section C.24). This update includes the network addresses
are reachable through a phantom router on the network attached to
port. For a full mesh test, one destination network address
present in the routing update for each of the "input" ports.
test stream on each "input" port consists of a repeating sequence
frames, one to each of the "output" ports
Bradner & McQuaid Informational [Page 26]
RFC 2544 Benchmarking Methodology March 1999
C.2.4.3 Management Query
The management overhead test uses SNMP to query a set of
that should be present in all DUTs that support SNMP. The
for a single interface only are read by an NMS at the
intervals. The list of variables to retrieve follow
C.2.4.4 Test
The test frame is an UDP Echo Request with enough data to fill
the required frame size. The data should not be all bits off or
bits on since these patters can cause a "bit stuffing" process to
used to maintain clock synchronization on WAN links. This
will result in a longer frame than was intended
C.2.4.5 Frame Formats - TCP/IP on
Each of the frames below are described for the 1st pair of DUT ports
i.e. "input" port #1 and "output" port #1. Addresses must be
if the frame is to be used for other ports
C.2.6.1 Learning
ARP Request on
-- DATAGRAM
offset data (hex)
00 FF FF FF FF FF FF dest MAC address send
broadcast
06 xx xx xx xx xx xx set to source MAC
12 08 06 ARP
14 00 01 hardware type Ethernet = 1
16 08 00 protocol type IP = 800
18 06 hardware address length 48
on
19 04 protocol address length 4
for
20 00 01 opcode request = 1
22 xx xx xx xx xx xx source MAC
28 xx xx xx xx source IP
32 FF FF FF FF FF FF requesting DUT's MAC
38 xx xx xx xx DUT's IP
Bradner & McQuaid Informational [Page 27]
RFC 2544 Benchmarking Methodology March 1999
C.2.6.2 Routing Update
-- DATAGRAM
offset data (hex)
00 FF FF FF FF FF FF dest MAC address is
06 xx xx xx xx xx xx source hardware
12 08 00
-- IP
14 45 IP version - 4, header length (4
byte units) - 5
15 00 service
16 00 EE total
18 00 00
20 40 00 flags (3 bits) 4 (do
fragment),
fragment offset-0
22 0A
23 11 protocol - 17 (UDP
24 C4 8D header
26 xx xx xx xx source IP
30 xx xx xx destination IP
33 FF host part = FF for
-- UDP
34 02 08 source port 208 =
36 02 08 destination port 208 =
38 00 DA UDP message
40 00 00 UDP
-- RIP
42 02 command =
43 01 version = 1
44 00 00 0
-- net 1
46 00 02 family =
48 00 00 0
50 xx xx xx net 1 IP
53 00 net not
54 00 00 00 00 0
58 00 00 00 00 0
62 00 00 00 07 metric 7
-- net 2
66 00 02 family =
68 00 00 0
70 xx xx xx net 2 IP
Bradner & McQuaid Informational [Page 28]
RFC 2544 Benchmarking Methodology March 1999
73 00 net not
74 00 00 00 00 0
78 00 00 00 00 0
82 00 00 00 07 metric 7
-- net 3
86 00 02 family =
88 00 00 0
90 xx xx xx net 3 IP
93 00 net not
94 00 00 00 00 0
98 00 00 00 00 0
102 00 00 00 07 metric 7
-- net 4
106 00 02 family =
108 00 00 0
110 xx xx xx net 4 IP
113 00 net not
114 00 00 00 00 0
118 00 00 00 00 0
122 00 00 00 07 metric 7
-- net 5
126 00 02 family =
128 00 00 0
130 00 net 5 IP
133 00 net not
134 00 00 00 00 0
138 00 00 00 00 0
142 00 00 00 07 metric 7
-- net 6
146 00 02 family =
148 00 00 0
150 xx xx xx net 6 IP
153 00 net not
154 00 00 00 00 0
158 00 00 00 00 0
162 00 00 00 07 metric 7
C.2.4.6 Management Query
To be defined
C.2.6.4 Test
UDP echo request on
Bradner & McQuaid Informational [Page 29]
RFC 2544 Benchmarking Methodology March 1999
-- DATAGRAM
offset data (hex)
00 xx xx xx xx xx xx set to dest MAC
06 xx xx xx xx xx xx set to source MAC
12 08 00
-- IP
14 45 IP version - 4 header length 5 4
byte
15 00
16 00 2E total length
18 00 00
20 00 00 flags (3 bits) - 0
offset-0
22 0A
23 11 protocol - 17 (UDP
24 C4 8D header checksum
26 xx xx xx xx set to source IP address**
30 xx xx xx xx set to destination IP address**
-- UDP
34 C0 20 source
36 00 07 destination port 07 =
38 00 1A UDP message length
40 00 00 UDP
-- UDP
42 00 01 02 03 04 05 06 07 some data***
50 08 09 0A 0B 0C 0D 0E 0
* - change for different length
** - change for different logical
*** - fill remainder of frame with incrementing octets
repeated if required by frame
Values to be used in Total Length and UDP message length fields
frame size total length UDP message
64 00 2E 00 1
128 00 6E 00 5
256 00 EE 00 9
512 01 EE 01 9
768 02 EE 02 9
1024 03 EE 03 9
1280 04 EE 04 9
1518 05 DC 05 C
Bradner & McQuaid Informational [Page 30]
RFC 2544 Benchmarking Methodology March 1999
Full Copyright
Copyright (C) The Internet Society (1999). All Rights Reserved
This document and translations of it may be copied and furnished
others, and derivative works that comment on or otherwise explain
or assist in its implementation may be prepared, copied,
and distributed, in whole or in part, without restriction of
kind, provided that the above copyright notice and this paragraph
included on all such copies and derivative works. However,
document itself may not be modified in any way, such as by
the copyright notice or references to the Internet Society or
Internet organizations, except as needed for the purpose
developing Internet standards in which case the procedures
copyrights defined in the Internet Standards process must
followed, or as required to translate it into languages other
English
The limited permissions granted above are perpetual and will not
revoked by the Internet Society or its successors or assigns
This document and the information contained herein is provided on
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED,
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Bradner & McQuaid Informational [Page 31]
if you see any problems within the linking, don't worry be happy,
this is version 0.1 of the Relevance System and you gotta expect some crappy subroutines sometimes,
just be content we did not write this in Java, which would have made this "bigger and better" HAHAHHA.
RFC documents can be found at I.E.T.F.
Relevance System Copyright © 2002 Spectrum WorldResearch
other technical nosh by ServerMasters Corporation
collaboration of BobX
