As per Relevance of the word synchronization, we have this rfc below:
Network Working Group D.L.
Request for Comments: 957 M/A-COM
September 1985
Experiments in Network Clock
Status of this
This RFC discusses some experiments in clock synchronization in
ARPA-Internet community, and requests discussion and suggestions
improvements. Distribution of this memo is unlimited
Table of
1.
2. Design of the Synchronization
2.1. The Logical
2.2. Linear Phase
2.3. Nonlinear Phase
3. Synchronizing Network
3.1. Reference Clocks and Reference
3.2. Distribution of Timing
4. Experimental Validation of the
4.1. Experiment
4.2. Experiment
4.3. Discussion of
4.3.1. On Power-Grid
4.3.2. On Clocks Synchronized via Network
4.3.3. On the Accuracy of Radio
4.3.3.1. The Spectracom 8170 WWVB Radio
4.3.3.2. The True Time 468-DC GOES Radio
4.3.3.3. The Heath GC-1000 WWV Radio
4.3.4. On Handling
4.4. Additional
5. Summary and
6.
List of
Figure 1. Clock
Figure 2. Network
Mills [Page 1]
RFC 957 September 1985
Experiments in Network Clock
List of
Table 1. Experiment
Table 2. Link
Table 3. First Derivative of
Table 4. GOES Radio Clock
Table 5. WWV Radio Clock
Table 6. ISI-MCON-GW Clock
Table 7. LL-GW Clock
Table 8. LL-GW Clock
1.
One of the services frequently neglected in computer network
is a high-quality, time-of-day clock capable of generating
timestamps with small residual errors compared to intrinsic one-
network delays. Such a service would be useful for tracing
progress of complex transactions, synchronizing cached data bases
monitoring network performance and isolating problems
Several mechanisms have been specified in the Internet protocol
to record and transmit the time at which an event takes place
including the ICMP Timestamp message [6], Time Protocol [7],
protocol [8] and IP Timestamp option [9]. A new Network
Protocol [12] has been proposed as well. Additional information
network time synchronization can be found in the References at
end of this document. Synchronization protocols are described in [3]
and [12] and synchronization algorithms in [2], [5], [10] and [11].
Experimental results on measured roundtrip delays in the Internet
discussed in [4]. A comprehensive mathematical treatment of
synchronization can be found in [1].
Several mechanisms have been specified in the Internet protocol
to record and transmit the time at which an event takes place
including the ICMP Timestamp message [6], Time protocol [7],
protocol [8] and IP Timestamp option [9]. Issues on
synchronization are discussed in [4] and synchronization
in [2] and [5]. Experimental results on measured roundtrip delays
the Internet are discussed in [2]. A comprehensive
treatment of the subject can be found in [1], while an
discussion on mutual-synchonization techniques can be found in [10].
There are several ways accurate timestamps can be generated. One
to provide at every service point an accurate, machine-readable
synchronized to a central reference, such as the National Bureau
Standards (NBS). Such clocks are readily available in several
ranging in accuracies of a few hundred milliseconds to less than
Mills [Page 2]
RFC 957 September 1985
Experiments in Network Clock
millisecond and are typically synchronized to special ground-based
satellite-based radio broadcasts. While the expense of the
themselves, currently in the range $300 to $3000, can often
justified, all require carefully sited antennas well away
computer-generated electromagnetic noise, as well as
connections to the clocks. In addition, these clocks can require
lengthy synchonization period upon power-up, so that a battery-
power supply is required for reliable service in the event of
interruptions
If the propagation delays in the network are stable or can
predicted accurately, timestamps can be generated by a
server, equipped with a clock such as described above, in response
requests from remote service points. However, there are
instances where the trans-network delay to obtain a timestamp
be intolerable, such as when timestamping a message
transmission. In addition, propagation delays are usually
predictable with precisions in the order required, due
probabilistic queuing and channel-contention delays
In principle, a clock of sufficient accuracy can be provided at
service point using a stable, crystal-controlled clock which
corrected from time to time by messages from a central server
Suitable inexpensive, crystal-controlled clock interfaces
available for virtually any computer. The interesting
remaining is the design of the synchronization algorithm and
used to transmit the corrections. In this document one such
will be described and its performance assessed. This design has
incorprated as an integral part of the network routing and
protocols of the Distributed Computer Network (DCnet)
[5], clones of which have been established at several sites in the
and Europe. These protocols have been in use since 1979 and
continuously tested and refined since then
2. Design of the Synchronization
The synchronization algorithm is distributed in nature, with
peers maintained in every host on the network. Peers
with each other on a pairwise basis using special control messages
called Hello messages, exchanged periodically over the ordinary
links between them. The Hello messages contain information
for each host to calculate the delay and offset between the
clock of the host and the clock of every other host on the
and are also used to drive the routing algorithm
The synchronization algorithm includes several features to
the accuracy and stability of the local clock in the case of host
Mills [Page 3]
RFC 957 September 1985
Experiments in Network Clock
link failures. In following sections the design of the algorithm
summarized. Full design details are given in [5] along with a
description of the Hello protocol
2.1. The Logical
In the DCnet model each service point, or host, is equipped with
hardware clock, usually in the form of an off-the-shelf interface
Using this and software registers, a logical clock is
including a 48-bit Clock Register, which increments at a 1000
rate, a 32-bit Clock-Adjust Register, which is used to slew the
Register in response to raw corrections received over the net, and
Counter Register, which is used in some interface designs as
auxilliary counter. The configuration and decimal point of
registers are shown in Figure 1.
Clock Register
0 16 32
+---------------+---------------+---------------+
| | | |
+---------------+---------------+---------------+
A
decimal point
Clock-Adjust Register
0 16
+---------------+---------------+
| | |
+---------------+---------------+
A
decimal point
Counter Register
0 16
+---------------+
| |
+---------------+
A
decimal point
Figure 1. Clock
The Clock Register and Clock-Adjust Register are implemented
memory. In typical clock interface designs such as the DEC KMV11-
Mills [Page 4]
RFC 957 September 1985
Experiments in Network Clock
the Counter Register is implemented in the interface as a
counter driven by a crystal oscillator. A counter overflow
signalled by an interrupt, which results in an increment of the
Register at bit 15 and the propagation of carries as required.
time of day is determined by reading the Counter Register, which
not disturb its counting process, and adding its value to that of
Clock Register with decimal points aligned
In other interface designs such as the simple LSI-11 event-
mechanism, each tick of the clock is signalled by an interrupt
intervals of 10, 16-2/3 or 20 ms, depending on interface and
source. When this occurs the appropriate number of milliseconds
expressed to 32 bits in precision, is added to the Clock
with decimal points aligned
It should be noted at this point that great care in operating
design is necessary in order to preserve the full accuracy
timestamps with respect to the application program, which must
protected from pre-emption, excessive device latencies and so forth
In addition, the execution times of all sequences operating with
interrupt system disabled must be strictly limited. Since the PDP11
operating system most often used in the DCnet (the "Fuzzball
operating system) has been constructed with these
foremost in mind, it has been especially useful for detailed
performance testing and evaluation. Other systems, in particular
various Unix systems, have not been found sufficiently accurate
this purpose
Left uncorrected, the host logical clock runs at the rate of
intrinsic oscillator, whether derived from a crystal or the
frequency. The correction mechanism uses the Clock-Adjust Register
which is updated from time to time as raw corrections are received
The corrections are computed using roundtrip delays and
derived from the routing algorithm, described later in this document
which are relatively noisy compared to the precision of the
clock. A carefully designed smoothing mechansim insures stability
as well as isolation from large transients that occur due to
retransmissions, host reboots and similar disruptions
Mills [Page 5]
RFC 957 September 1985
Experiments in Network Clock
2.2. Linear Phase
The correction is introduced as a signed 32-bit integer
milliseconds. If the magnitude of the correction is less than 128
ms, the low-order 16 bits replaces bits 0-15 in the Clock-
register. At suitable intervals, depending on the jitter of
intrinsic oscillator, the value of this register is divided by
fixed value, forming a quotient which is first added to the
Register, then subtracted from the Clock-Adjust Register.
technique has several advantages
1. The clock never runs backwards; that is,
timestamps always increase monotonically
2. In the event of loss of correction information, the
slews to the last correction received
3. The rate of slew is proportional to the magnitude of the
correction. This allows rapid settling in case of
corrections, but provides high stability in case of
corrections
4. The sequence of computations preserves the highest
and minimizes the propagation of round-off errors
Experience has indicated the choice of 256 as appropriate for
dividend above, which yields a maximum slew-rate magnitude less
0.5 ms per adjustment interval and a granularity of about 2.0
microseconds, which is of the same order as the intrinsic
of the crystal oscillators used in typical clock interfaces. In
case of crystal-derived clocks, an adjustment interval of
seconds has worked well, which yields a maximum slew-rate
of 125 microseconds per second. In the case of power-
clocks or especially noisy links, the greatly increased
requires shorter adjustment intervals in the range of 0.5 second
which yields a maximum slew-rate magnitude of 1.0 ms per second
In most cases, independent corrections are generated over each
at intervals of 30 seconds or less. Using the above choices a
sample error of 128 ms causes an error at the next sample interval
greater than about 7.5 ms with the longer adjustment interval and 30
ms with the shorter. The number of adjustment intervals to
the residual error by half is about 177, or about 12 minutes with
longer interval and about 1.5 minutes with the shorter.
completely characterizes the linear dynamics of the mechanism
Mills [Page 6]
RFC 957 September 1985
Experiments in Network Clock
2.3. Nonlinear Phase
When the magnitude of the correction exceeds 128 ms, the
exists that the clock is so far out of synchronization with
reference host that the best action is an immediate and
replacement of Clock Register contents, rather than a
slewing as described above. In practice the necessity to do this
rare and occurs when the local host or reference host is rebooted
for example. This is fortunate, since step changes in the clock
result in the clock apparently running backward, as well as
delay and offset measurements of the synchronization
itself
However, it sometimes happens that, due to link retransmissions
occasional host glitches, a single correction sample will be
with magnitude exceeding 128 ms. In practice this happens
enough that a special procedure has been incorporated into
design. If a sample exceeding the limit is received, its value
saved temporarily and does not affect the Clock-Adjust Register.
addition, a timer is initialized, if not already running, to
down to zero in a specified time, currently 30 seconds
If the timer is already running when a new correction sample
magnitude exceeeding 128 ms arrives, its value and the saved
value are averaged with equal weights to form a new saved
value. If a new correction sample arrives with magnitude less
128 ms, the timer is stopped and the saved sample value discarded
If the timer counts down to zero, the saved sample value replaces
contents of the Clock Register and the Clock-Adjust Register is
to zero. This procedure has the effect that occasional spikes
correction values are discarded, but legitimate step changes
prefiltered and then used to reset the clock after no more than
30-second delay
3. Synchronizing Network
The algorithms described in the previous section are designed
achieve a high degree of accuracy and stability of the logical
in each participating host. In this section algorithms will
described which synchronize these logical clocks to each other and
standard time derived from NBS broadcasts. These algorithms
designed to minimize the cumulative errors using linear
techniques. See [10] for a discussion of algorithms using
techniques
Mills [Page 7]
RFC 957 September 1985
Experiments in Network Clock
3.1. Reference Clocks and Reference
The accuracy of the entire network of logical clocks depends on
accuracy of the device used as the reference clock. In the
clones the reference clock takes the form of a precision
oscillator which is synchronized via radio or satellite with the
standard clocks in Boulder, CO. The date and time derived from
oscillator can be sent continuously or read upon command via a
asynchronous line. Stand-alone units containing the oscillator
synchronizing receiver and controlling microprocessor are
from a number of manufacturers
The device driver responsible for the reference clock uses
serial-line protocol to derive both an "on-time" timestamp
to the logical clock of the reference host and an absolute
encoded in messages sent by the clock. About once every 30
the difference between these two quantities is calculated and used
correct the logical clock according to the mechanisms
previously. The corrected logical clock is then used to correct
other logical clocks in the network. Note the
nomenclature: The term "reference clock" applies to the
clock itself, while the term "reference host" applies to the
clock of the host to which it is connected. Each has an
address, delay and offset in synchronizing messages
There are three different commercial units used as reference
in DCnet clones. One of these uses the LF radio broadcasts on 60
from NBS radio WWVB, another the HF radio broadcasts on 5, 10 and 15
MHz from NBS radio WWV or WWVH and the third the UHF broadcasts
a GOES satellite. The WWVB and GOES clocks claim accuracies in
one-millisecond range. The WWV clock claims accuracies in the 100-
range, although actual accuracies have been measured somewhat
than that
All three clocks include some kind of quality indication in
messages, although of widely varying detail. At present
indication is used only to establish whether the clock
synchronized to the NBS clocks and convey this information to
network routing algorithm as described later. All clocks
some provision to set the local-time offset and propagation delay
although for DCnet use all clocks are set at zero offset relative
Universal Time (UT). Due to various uncertaincies in
delay, serial-line speed and interrupt latencies, the
accuracy of timestamps derived from a reference host equipped with
WWVB or GOES reference clock is probably no better than a couple
milliseconds
Mills [Page 8]
RFC 957 September 1985
Experiments in Network Clock
3.2. Distribution of Timing
The timekeeping accuracy at the various hosts in the network
critically on the precision whith which corrections can
distributed throughout the network. In the DCnet design
distributed routing algorithm is used to determine minimum-
routes between any two hosts in the net. The algorithms used,
are described in detail in [5] and only in outline form here,
reachability and delay information, as well as clock offsets,
neighboring hosts by means of periodic exchanges of routing
called Hello messages. This information is then incorporated into
set of routing tables maintained by each host and spread
the network by means of the Hello messages
The detailed mechanisms to accomplish these functions have
carefully designed to minimize timing uncertaincies. For instance
all timestamping is done in the drivers themselves, which are
the highest priority for resource access. The mechanism to
roundtrip delays on the individual links is insensitive to the
inherent in the processing of the Hello message itself, as well
the intervals between transmissions. Finally, care is taken
isolate and discard transient timing errors that occur when a host
rebooted or a link is restarted
The routing algorithm uses a table called the Host Table,
contains for each host in the network the computed roundtrip
and clock offset, in milliseconds. In order to separately
each reference clock, if there is more than one in the network,
separate entry is used for each clock, as well as each host.
delay and offset fields of the host itself are set to zero, as is
delay field of each attached reference clock. The offset field
each attached reference clock is recomputed periodically as
above
Hello messages containing a copy of the Host Table are
periodically to each neighbor host via the individual
connecting them. In the case of broadcast networks the Hello
is broadcast to all hosts sharing the same cable. The Hello
also contains a timestamp inserted at the time of transmission,
well as information used to accurately compute the roundtrip delay
point-to-point links
A host receiving a Hello message processes the message for each
in turn, including those corresponding to the reference clocks.
adds the delay field in the message to the previously
roundtrip link delay and compares this with the entry already in
Host Table. If the sum is greater than the delay field in the
Mills [Page 9]
RFC 957 September 1985
Experiments in Network Clock
Table, nothing further is done. If the sum is less, an
procedure is executed. The update procedure, described in detail
[5], causes the new delay to replace the old and the routing to
amended accordingly
The update procedure also causes a new correction sample to
computed as the difference between the timestamp in the Hello
and the local clock, which is used to correct the local clock
described above. In addition, the sum of this correction sample
the offset field in the Hello message replaces the offset field
the Hello Table. The effect of these procedures is that the
clock is corrected relative to the neighbor clock only if
neighbor is on the path of least delay relative to the
reference clock. If there is no route to the reference clock,
determined by the routing algorithm, no corrections are
and the local clock free-runs relative to the last
correction
As the result of the operation of the routing algorithm, all
clocks in the network will eventually stabilize at a constant
relative to the reference clock, depending upon the drift rates
the individual oscillators. For most applications the offset
small and can be neglected. For the most precise measurements
computed offset in the Host Table entry associated with any host
including the reference clock, can be used to adjust the
time relative to the local clock of that host. However, since
computed offsets are relatively noisy, it is necessary to smooth
using some algorithm depending upon application. For this reason
the computed offsets are provided separately from the local time
4. Experimental Validation of the
The original DCnet was established as a "port expander" net
to an ARPAnet IMP in 1978. It was and is intended as an
testbed for the development of protocols and measurement of
performance. Since then the DCnet network-layer protocols
evolved to include multi-level routing, logical addressing
multicasting and time synchronization. Several DCnet clones
been established in the US and Europe and connected to the
Internet system. The experiments described below were
using the DCnet clone at Linkabit East, near Washington, DC,
another at Ford Motor Division, near Detroit, MI. Other clones
Ford Aerospace and the Universities of Maryland and Michigan
used for calibration and test, while clones at various sites
Norway and Germany were used for occasional tests.
Mills [Page 10]
RFC 957 September 1985
Experiments in Network Clock
ARPANET gateways of the WIDEBAND/EISN satellite system were
included in the experiments in order to determine the feasability
synchronizing clocks across the ARPANET
There were four principal issues of interest in the experiments
1. What are the factors affecting accuracy of a network of
using the power grid as the basic timing source, together
corrections broadcast from a central point
2. What are the factors affecting accuracy of a network of
synchronized via links used also to carry ordinary data
3. How does the accuracy of the various radio clocks - WWVB,
and WWV compare
4. What is the best way to handle disruptions, such as a
second
These issues will be discussed in turn after presentation of
experiment design and execution
4.1. Experiment
Figure 2 shows the configuration used in a series of tests
during late June and early July of 1985. The tests involved
hosts, three reference clocks and several types of
links. The tests were designed to coincide with the insertion of
leap second in the standard time broadcast by NBS, providing
interesting test of system stability in the face of such disruptions
The test was also designed to test the feasability of using the
grid as a reference clock, with corrections broadcast as
above, but not used to adjust the local clock
Mills [Page 11]
RFC 957 September 1985
Experiments in Network Clock
ARPAnet |
- - - - - - - - - - - - - - - - - - | - - - - - - - - - -
56K |
+---------+ +---------+ +----+----+ 1.2 +---------+
| WWV | 1.2 | | 4.8 | +-----+ WWVB |
| radio +-----+ DCN6 +-----+ DCN1 |async| radio |
| clock |async| |DDCMP| +--+ | clock |
+---------+ +---------+ +----+----+ | +---------+
Ethernet | |
DCnet ===o===============o=======o=== | 1822/DH
| | |
+----+----+ +----+----+ +----+----+
power | | | | | |
freq <--+ DCN3 | | DCN5 | | DCN7 +--> freq
60 Hz | | | | | | 60
+---------+ +----+----+ +---------+
9.6 |
- - - - - - - - - - - - - - | - - - - - - - - - - - - - -
| DDCMP
+----+----+ +---------+
| | 1.2 | GOES |
FORDnet | FORD1 +-----+satellite|
| |async| clock |
+---------+ +---------+
Figure 2. Network
Only those hosts and links directly participating in the tests
shown in Figure 2. All hosts shown operate using the DCnet
and timekeeping algorithms summarized in this document and
in [5]. The DCnet hosts operate as one self-contained net of
Internet systems, while the FORDnet hosts operate as another
distinct net numbers. The gateway functions connecting the two
are distributed in the DCN5 and FORD1 hosts and the link
them. This means that, although the clock offsets of
DCnet hosts are visible to other DCnet hosts and the clock offsets
individual FORDnet hosts are visible to other FORDnet hosts, only
clock offset of the gateway host on one net is visible to hosts
the other net
In Figure 2 the links are labelled with both the intrinsic speed,
kilobits per second, as well as the link protocol type. The
links use microprocessor-based DMA interfaces that retransmit in
of message failure. The 1822/DH link connecting DCN1 and DCN
operates at DMA speeds over a short cable. The Ethernet link
Mills [Page 12]
RFC 957 September 1985
Experiments in Network Clock
DMA interfaces that retransmit only in case of collisions.
asynchronous links are used only to connect the reference clocks
the hosts over a short cable
While all hosts and links were carrying normal traffic throughout
test period, the incidence of retransmissions was very low,
no more than a few times per day on any link. However, the
link protocol includes the use of short control messages
between the microprocessors about once per second in the absence
link traffic. These messages, together with retransmissions when
occur, cause small uncertaincies in Hello message delays
contribute to the total measurement error. An additional
(less than 0.5 per-cent on average) in Hello message length can
introduced when the link protocol makes use of character-stuffing
bit-stuffing techniques to achieve code transparency, such as
the LAPB link-level protocol of X.25. However, the particular
used in the tests use a count field in the header, so that
stuffing is required
Although the timekeeping algorithms have been carefully designed
be insensitive to traffic levels, it sometimes happens that
intense burst of traffic results in a shortage of memory buffers
the various hosts. In the case of the Ethernet interfaces,
have internal buffers, this can result in additional delays while
message is held in the interface pending delivery to the host
Conditions where these delays become significant occur perhaps
or twice a day in the present system and were observed
during the tests. As described above, the correction-
processing incorporates a filtering procedure that discards the
majority of glitches due to this and other causes
4.2. Experiment
The series of experiments conducted in late June and early July
1985 involved collecting data on the delays and offsets of the
hosts and three reference clocks shown in Figure 2. In order
accomplish this, a special program was installed in a Unix 4.2
system connected to the Ethernet link but not shown in Figure 2.
program collected each 128-octet Hello message broadcast from DCN
every 16 seconds and appended it bit-for-bit to the data base.
total volume of raw data collected amounted to almost 0.7
per day
The raw Hello-message data were processed to extract only
timestamp and measured clock offsets for the hosts shown in Table 1
and then reformatted as an ASCII file, one line per Hello message
Mills [Page 13]
RFC 957 September 1985
Experiments in Network Clock
Host Clock Drift Experiment Use
Name ID (ppm)
------------------------------------------------------
DCN1 WWVB -2.5 WWVB reference host
DCN3 - 60-Hz power-grid (unlocked)
DCN5 DCN1 6.8 Ethernet host
DCN6 DCN1 -1.7 DDCMP host, WWV reference host
DCN7 DCN1 60-Hz power-grid (locked)
FORD1 GOES 17.9 GOES reference host
WWV - - WWV reference clock
WWVB - - WWVB reference clock
Table 1. Experiment
In Table 1 the Clock ID column shows the reference host selected
the master clock for each host shown. In this
configuration host DCN1 was locked to host WWVB, while hosts DCN5,
DCN6 and DCN7 were locked to DCN1. Although the offset of GOES
not be directly determined from the Hello messages exchanged
DCnet and FORDnet hosts, the offset of FORD1 relative to GOES
determined by observation to be in the order of a millisecond, so
all practical purposes the offset of FORD1 represents the offset
GOES. In addition, since the WWVB clock was considered by
the most accurate and reliable and the offset of DCN1 relative
WWVB was negligible, DCN1 was considered the reference clock
offset zero relative to the NBS clocks
During the setup phase of the experiments the intrinsic drift
of the crystal oscillators in the four hosts DCN1, DCN5, DCN6
FORD1 equipped with them was measured as shown in the "Drift"
in Table 1. The two hosts DCN3 and DCN7 are equipped
line-frequency clocks. For experimental purposes DCN3 was
and allowed to free-run at the line-frequency rate, while DCN
remained locked
An ASCII file consisting of about 0.2 megabyte of reformatted data
was collected for each Universal-Time (UT) day of
beginning on 28 June and continuing through 8 July. Each file
processed by a program that produces an eight-color display
measured offsets as a function of time of observation. Since
display technique uses a bit-map display and each
overwrites the bit-map in an inclusive-OR fashion, the
dispersion is immediately apparent. Over eight samples per pixel
the time axis are available in a 24-hour collection period. On
other hand, the fine granularity of almost four samples per
allows zooming the display to focus on interesting short-
fluctuations, such as in the case of the WWV clock
Mills [Page 14]
RFC 957 September 1985
Experiments in Network Clock
4.3. Discussion of
Each of the four previously mentioned issues of interest will
discussed in following subsections
4.3.1. On Power-Grid
Telephone interviews with operators and supervisors of the
Electric Power Company (PEPCO), the electric utility serving
Washington, DC, area, indicate that there are three major
regions or grids, one east of the Rockies, a second west of
Rockies and a third in parts of Texas. The member electric
in each grid operate on a synchronous basis, so that clocks
within the grid should keep identical time. However, in the
case when a utility drops off the grid, no attempt is made
re-establish correct time upon rejoining the grrd. In the much
common case when areas within the grid are isolated due to
thunderstorms, for example, clock synchronization is also disrupted
The experiments provided an opportunity to measure with
precision the offset between a clock connected to the eastern
(DCN3) and the NBS clocks. The results, confirmed by the
interviews, show a gradual gain in time of between four and
seconds during the interval from about 1700 local time to 0800
next morning, followed by a more rapid loss in time between 0800
1700. If the time was slewed uniformly throughout these extremes
the rate would be about 100 ppm
The actual slewing rates depend on the demand, which itself is
function of weather, day of the week and season of the year.
effects occur in the western and Texas grids, with more
variations in the Texas grid due to the smaller inertia of
system, and less extreme variations in the western grid, due
smaller extremes in temperature, less total industrial demand and
larger fraction of hydro-electric generation
The uilities consider timekeeping a non-tariffed service provided
a convenience to the customer. In the eastern grid a control
in Ohio manually establishes the baseline system output,
indirectly affects the clock offset and slewing rate. The local
is determined at the control station with respect to a WWVB
clock. The maximum slewing rate is specified as .025 Hz (about 400
ppm), which is consistent with the maximum rates observed. In
western grid the baseline system output is adjusted
using a servomechanism driven by measured offsets from the
clocks
Mills [Page 15]
RFC 957 September 1985
Experiments in Network Clock
Given the considerations above, it would seem feasable for hosts
synchronize logical clocks to a particular power grid, but only
corrections were transmitted often enough to maintain the
accuracy and these corrections were delivered to the
essentially at the same time. Assuming a worst-case 400-ppm
rate and one minute between correction broadcasts, for example,
would in principle be possible to achieve accuracies in the 20-
range. There are a number of prediction and smoothing
that could be used to inhance accuracy and reduce the overhead of
broadcasts
Host DCN3, which uses a line-frequency clock interface, was
during the experiment period so that the offset between the
clock, which is locked to the eastern power grid, could be
with respect to the reference host DCN1. Host DCN7, which uses
same interface, remained locked to DCN1. In spite of the
noted instability of the power grid, DCN7 remained typically
30 ms of DCN1 and only infrequently exceeded 100 ms in the
of large changes in system load that occured near 0800 and 1700
time. Over the seven-day period from 2 July through 8 July the
offset was less than a millisecond with standard deviation about 24
ms, while the maximum was 79 ms and minimum -116 ms
Experiments were also carried out using ICMP Timestamp messages
hosts known to use line-frequency clock interfaces in California
Norway and Germany. The results indicated that the western
grid is rather more stable than the eastern grid and that
overseas grids are rather less stable. In the Oslo, Munich
Stuttgart areas, for example, the diurnal variation was observed
exceed ten seconds
4.3.2. On Clocks Synchronized via Network
As mentioned previously, all network links used to synchronize
clocks were carrying normal data traffic throughout the
period. It would therefore be of interest to investigate how
affects the accuracy of the individual clocks
Table 2 summarizes the mean and standard deviation of the
offsets between the WWVB radio clock and various hosts as shown
Figure 2. Measurements were made over the 24-hour period for each
several days during the experimental period. Each entry shown
Table 2 includes the mean of the statistic over these days,
with the maximum variation
Mills [Page 16]
RFC 957 September 1985
Experiments in Network Clock
Host Mean Deviation Link Type and Speed
-----------------------------------------------------------
DCN1 .08/.02 0.53/.02 WWVB radio clock (1200 bps
DCN5 -13.61/.04 1.1/0.4 Ethernet (10 Mbps)
DCN6 0.27/.18 5.8/1.0 DDCMP (4800 bps)
FORD1 38.5/1.6 2.5/0.5 DDCMP (9600 bps)
Table 2. Link
The departure of the mean shown in Table 2 from zero is related
the drift of the crystal oscillator used in the hardware
(see Table 1). As described previously, FORD1 was synchonized to
GOES radio clock with neglible offset, so that the mean and
deviation shown can be accurately interpreted to apply to the
radio clock as well
The results show that the uncertaincies inherent in
synchronization algorithm and protocols is in the same order as
of the reference clocks and is related to the speed of the
connected the reference hosts to the other hosts in the network
Further discussion on the FORD1/GOES statistics can be found in
next section
Further insight into the error process can be seen in Table 3,
shows the first derivative of delay
Host Dev Max Min Error
-------------------------------------
DCN3 2.3 12 -17 10
DCN5 1.5 45 -45 5
DCN6 9 94 -54 40
DCN7 1.4 6 -7 5
FORD1 3.4 68 -51 15
Table 3. First Derivative of
The mean and standard deviation of delay were computed for all
on a typical day during the experimental period. In all cases
magnitude of the mean was less than one. The standard deviation
maximum and minimum for each link is summarized by host in Table 3.
A common characteristic of the distribution in most cases was
only a handful of samples approached the maximum or minimum extrema
while the vast majority of samples were much less than this.
"Error" colum in Table 3 indicates the magnitude of the
error when these extrema are discarded
Mills [Page 17]
RFC 957 September 1985
Experiments in Network Clock
A very interesting feature of the observations was the
low standard deviation of DCN3, which was locked to the power
and thus would be expected to show wide variations. Upon analysis
this turned out to be a natural consequence of the fact that
Hello messages are generated as the result of interrupts based on
line frequency when the local clock had just been incremented
1/60th of a second
The synchronizing protocol and implementation were
constructed to minimize the loss of accuracy due to sharing of
network links between data and control traffic, as long as
resources (in particular, packet buffers) are available. Since
various network links shown in Figure 2 operate over a wide range
rates, it is possible that undisciplined bursts of traffic can
a host or gateway and precipitate a condition of buffer starvation
While most hosts using paths through the experimental
were relatively well-disciplined in their packetization
retransmission policies, some Unix 4.2bsd systems were
exceptions. On occasion these hosts were observed sending floods
packets, with only a small amount of data per packet, together
excessive retransmissions. As expected, this caused
congestion, unpredictable link delays and occasional
synchronizing errors
The synchronizing algorithms described above successfully cope
almost all instances of congestion as described, since delay-
errors tend to be isolated, while inherent anti-spike and
properties of the synchronizing algorithm help to preserve
in any case. Only one case was found during the ten-day
period where a host was mistakenly synchronized outside
linear-tracking window due to congestion. Even in this case the
was quickly resynchronized to the correct time when the
was cleared
4.3.3. On the Accuracy of Radio
One of the more potent motivations for the experiments was to
the accuracy of the various radio clocks and to determine whether
WWV radio clock was an appropriate replacement for the expensive
or GOES clocks. A secondary consideration, discussed further in
next section, was how the various clocks handled disruptions due
power interruptions, leap seconds and so forth
Mills [Page 18]
RFC 957 September 1985
Experiments in Network Clock
4.3.3.1. The Spectracom 8170 WWVB Radio
As the result of several years of experience with the WWVB
clock, which is manufactured by Spectracom Corporation as Model 8170,
it was chosen as the reference for comparison for the GOES and
radio clocks. Washington, DC, is near the 100-microvolt/
countour of the WWVB transmitter at Boulder, CO, well in excess
the 25-microvolt/meter sensitivity of the receiver. The antenna
located in a favorable location on the roof of a four-storey
in an urban area
Using the data from the instruction manual, the propagation delay
the path from Boulder to Washington is about 8 ms, while
intrinsic receiver delay is about 17 ms. The clock is read via
1200-bps asynchronous line, which introduces an additional delay
about 7 ms between the on-time transition of the first character
the interrupt at the middle of the first stop bit. Thus, the
radio clock indications should be late by 8 + 17 + 7 = 32 ms
to NBS standard time. While it is possible to include this
directly in the clock indication, this was not done in
experiments. In order to account for this, 32 ms should
subtracted from all indications derived from this clock.
uncertaincy in the indication due to all causes is estimated to be
couple of milliseconds
4.3.3.2. The True Time 468-DC GOES Radio
The GOES radio clock is manufactured by True Time Division
Kinemetrics, Incorporated, as Model 468-DC. It uses
Geosynchronous Orbiting Environmental Satellite (GOES),
includes an NBS-derived clock channel. Early in the
period there was some ambiguity as to the exact longitude of
satellite and also whether the antenna was correctly positioned
This was reflected in the rather low quality-of-signal
and occasional signal loss reported by the clock and also
apparent offset compared with the other radio clocks
Table 4 shows a summary of offset statistics for the GOES radio
by day (all day numbers refer to July, 1985).
Mills [Page 19]
RFC 957 September 1985
Experiments in Network Clock
Day Mean Dev Max Min
------------------------------------
2 31.6 9.4 53 -76
3 19.8 22.1 53 -64
4 42.8 17.1 >150 19
5 39.3 2.2 54 -45
6 37.8 2.7 53 19
7 62.2 13.0 89 22
8 38.2 2.8 90 -7
Table 4. GOES Radio Clock
On all days except days 5, 6 and 8 long periods of poor-
signal reception were evident. Since the antenna and
configuration are known to be marginal, these conditions are
considered representative of the capabilities of the clock. When
data from these days are discarded, the mean offset is 38.4 ms
standard deviation in the range 2.2 to 2.8. The maximum offset is 90
ms and the minimum is -45 ms; however, only a very small number
samples are this large - most excursions are limited to 10 ms of
mean
In order to compute the discrepancy between the GOES and WWVB clocks
it is necessary to subtract the WWVB clock delay from the
offsets computed above. Thus, the GOES clock indications are 38.4 -
32 = 6.4 ms late with respect to the WWVB clock indications.
is probably within the bounds of experiment error
4.3.3.3. The Heath GC-1000 WWV Radio
The WWV radio clock is manufactured by Heath Company as
GC-1000. It uses a three-channel scanning WWV/WWVH receiver on 5, 10
and 15 MHz together with a microprocessor-based controller.
receiver is connected to an 80-meter dipole up about 15 meters
located in a quiet suburban location. Signal reception from the
Collins transmitters was average to poor during the experiment
due to low sunspot activity together with a moderate level
geomagnetic disturbances, but was best during periods of
over the path. The clock locked at one of the frequencies
varying periods up to an hour from two to several times a day
The propagation delay on the path between Fort Collins and
is estimated at about 10 ms and can vary up to a couple
milliseconds over the day and night. While it is possible to
this delay in the clock indications, which are already corrected
Mills [Page 20]
RFC 957 September 1985
Experiments in Network Clock
the intrinsic receiver delay, this was not done in the experiments
During periods of lock, the clock indications are claimed to
accurate to within 100 ms
Table 5 shows a summary of offset statistics for the WWV radio
by day (all day numbers refer to July, 1985).
Day Mean Dev Max Min
------------------------------------
2 -31 36 110 -119
3 -42 38 184 -141
4 -21 38 61 -133
5 -31 37 114 -136
6 -48 42 53 -160
7 -100 80 86 -315
8 -71 70 115 -339
Table 5. WWV Radio Clock
On inspection of the detailed plots of offsets versus time the
reveal an interesting sawtooth variation with period about 25
per hour and amplitude about 90 ms. Once the clock has locked
some time the variation decreases in frequency and
disappears. This behavior is precisely what would be expected of
phase-locked oscillator and accounts for the rather large
deviations in Table 5.
On inspection of the plots of offsets versus time, it is
that by far the best accuracies are obtained at or in the periods
lock, which is most frequent during periods of darkness over
propagation path, which occured roughly between 0800 UT and 1100
during the experiment period. Excluding all data except
collected during this period, the mean offset is -21.3 ms
standard deviation in the range 29-31. The maximum offset is 59
and the minimum is -118 ms
In order to compute the discrepancy between the WWV and WWVB clocks
it is necessary to subtract the total of the propagation delay
WWVB clock delay from the mean offsets computed above. Thus, the
clock indications are -21.3 - 10 - 32 = -72.3 ms late (72.3 ms early
with respect to the WWVB clock indications. Considering the
standard deviations noted above, it is probably not worthwhile
include this correction in the WWV clock indications
On exceptional occasions excursions in offset over 300 ms relative
the WWVB clock were observed. Close inspection of the data
that this was due to an extended period (a day or more) in which
Mills [Page 21]
RFC 957 September 1985
Experiments in Network Clock
was not achieved on any frequency. The master oscillator uses
3.6-MHz crystal oscillator trimmed by a digital/analog converter
register which is loaded by the microprocessor. The
excursions in offset were apparently due to incorrect register
as the result of noisy reception conditions and excessive
between lock. On occasion the oscillator frequency was observed
error over 4 ppm due to this cause, which could result in
cumulative error of almost 400 ms per day if uncorrected
4.3.4. On Handling
The experiment period was intentionally selected to coincide with
insertion of a leap second in the worldwide time broadcasts.
intent was to examine the resulting behavior of the various
clocks and the synchronization algorithm when an additional
was introduced at 2400 UT on 30 June
As it turned out, radio reception conditions at the time of
were quite poor on all WWV frequencies, the WWVB frequency and
GOES frequency. Thus, all three clocks took varying periods up
several hours to resynchonize and correct the indicated time.
fact, the only time signals heard around the time of interest
those from Canadian radio CHU, but the time code of the
broadcasts is incompatible with the of the US broadcasts
As mentioned above, the WWVB clock was used as the master during
experiment period. About two hours after insertion of the
second the clock resynchronized and all hosts in the
network were corrected shortly afterwards. Since the magnitude
the correction exceeded 128 ms, the correction was of a step nature
but was not performed simultaneously in all hosts due to
individual timing of the Hello messages. Thus, if timing-
network operations happened to take place during the
process, inconsistent timestamps could result
The lesson drawn from this experience is quite clear. Accurate
synchronization requires by its very nature long integration times
so that epochal events which disrupt the process must be predicted
advance and applied in all hosts independently. In principle,
would not be hard to do and could even be integrated into
operation of the step-correction procedure described earlier,
in the form of bits included in Hello messages which trigger
one-second correction at the next rollover from 2400 to 0000 hours
In order for such an out-of-band correction to be effective,
notice of the leap second must be available. At present,
information is not available in the broadcast format and must
Mills [Page 22]
RFC 957 September 1985
Experiments in Network Clock
obtained via the news media. In fact, there are spare bits in
broadcast format that could be adapted for this purpose, but
would require reprogramming both the transmitting and
equipment. Nevertheless, this feature should be considered for
systems
4.4. Additional
A set of experiments was performed using two WIDEBAND/EISN
equipped with WWVB radio clocks and connected to the ARPANET.
experiments were designed to determine the limits of accuracy
comparing these clocks via ARPANET paths. One of the
(ISI-MCON-GW) is located at the Information Sciences Institute
Los Angeles, while the other (LL-GW) is located at
Laboratories near Boston. Both gateways consist of PDP11/44
computers running the EPOS operating system and clock-
boards with oscillators phase-locked to the WWVB clock
The clock indications of the WIDEBAND/EISN gateways were
with the DCNet WWVB reference clock using ICMP Timestamp
[6], which record the individual timestamps with a precision of
millisecond. This technique is not as accurate as the one
in Section 3, since the protocol implementation involves
user-process level, which can be subject to minor delays due
process scheduling and interprocess-message queueing. However
calibration measurements made over several of the links shown
Figure 2 indicate that the measurement errors are dominated by
individual link variations and not by the characteristics of
measurement technique itself
Measurements were made separately with each gateway by sending
ICMP Timestamp Request message from the ARPANET address of DCN1
the ARPANET address of the gateway and computing the round-trip
and clock offset from the ICMP Timestamp Reply message. This
was continued for 1000 message exchanges, which took about
minutes. Table 6 shows the statistics obtained with ISI-MCON-GW
Table 7 those with LL-GW (all numbers are milliseconds).
Mills [Page 23]
RFC 957 September 1985
Experiments in Network Clock
ISI-MCON-GW Mean Dev Max Min
--------------------------------------------
Offset -16 40 126 -908
Delay 347 59 902 264
Table 6. ISI-MCON-GW Clock
LL-GW (a) Mean Dev Max Min
--------------------------------------------
Offset -23 15 32 -143
Delay 310 25 536 252
Table 7. LL-GW Clock
The smaller values of standard deviation and extreme for LL-GW
probably due to the shorter ARPANET path involved. The confidence
the mean offset can be estimated by dividing the standard
by the square root of the number of samples (1000), which
that the mean offsets are accurate to within a couple of miliseconds
The mean offsets of the WIDEBAND/EISN clocks as a group relative
the DCN1 clock may thus indicate a minor discrepancy in the
of the delay-compensation switches
It is well known that ARPANET paths exhibit wide variations
delays, with occasional delays reaching surprising values up to
seconds. In order to improve the estimates a few samples
removed from both the offset and delay data, including all those
magnitude greater than one second
The above experiments involve a burst of activity over a
short time during which the ratio of the measurement traffic to
network traffic may be nontrivial. Another experiment with LL-GW
designed with intervals of ten seconds between ICMP messages
operated over a period of about three hours. The results are
in Table 8.
LL-GW (b) Mean Dev Max Min
--------------------------------------------
Offset -16 93 990 -874
Delay 371 108 977 240
Table 8. LL-GW Clock
Mills [Page 24]
RFC 957 September 1985
Experiments in Network Clock
Note that the standard deviations and extrema are higher than in
previous experiments, but the mean offset is about the same
The results of these experiments suggest that time
via ARPANET paths can yield accuracies to the order of a
milliseconds, but only if relatively large numbers of samples
available. The number of samples can be reduced and the
improved by using the techniques of Section 3 modified for
Timestamp messages and the longer, more noisy paths involved
5. Summary and
The experiments described above were designed to verify the
operation of the DCnet time-synchronization algorithms and
under a variety of scenarios, including the use of line-
clocks, three types of radio clocks and various types
interprocessor links. They involved the collection and processing
many megabytes of data collected over a ten-day period that
the insertion of a leap second in the standard NBS time scale.
the lessons learned were the following
1. The algorithms and protocols operate as designed,
accuracies throughout the experimental net in the order of
few milliseconds to a few tens of milliseconds, depending
the topology and link type
2. Glitches due to congestion, rebooted hosts and link
are acceptably low, even in the face of massive
resulting from inappropriate host implementations elsewhere
the Internet
3. A synchronization scenario where the clocks in all hosts
locked to the line frequency and corrections are
from a central time standard will work only if all hosts
on the same power grid, which is unlikely in the
Internet configuration, but may be appropriate for
applications
4. In spite of the eastern power grid wandering over as much
six seconds in a day, it is possible to achieve accuracies
the 30-ms range using line-frequency interface clocks
corrections broadcast on the local net
5. Radio clocks can vary widely in accuracy depending on
reception conditions. Absolute time can be determined
within a couple of milliseconds using WWVB and GOES
clocks, but only if they are calibrated using an
Mills [Page 25]
RFC 957 September 1985
Experiments in Network Clock
standard such as a portable clock. The inexpensive WWV
perform surprisingly well most of the time, but can be
error up to a significant fraction of a second under
conditions
6. Adjustments in the time scale due to leap seconds must
anticipated before they occur. The synchronization
must include a mechanism to broadcast an adjustment in
of its occurance, so that it can be incorporated in each
simultaneously. There is a need to incorporate advance
of leap seconds in the broadcast time code
7. Time synchronization via ARPANET paths can yield accuracies
the order of a few milliseconds, but only if relatively
numbers of samples are available. Further work is needed
develop efficient protocols capable of similar accuracies
using smaller numbers of samples
6.
1. Lindsay, W.C., and A.V. Kantak. Network Synchronization
Random Signals. IEEE Trans. Comm. COM-28, 8 (August 1980),
1260-1266.
2. Mills, D.L. Time Synchronization in DCNET Hosts. DARPA
Project Report IEN-173, COMSAT Laboratories, February 1981.
3. Mills, D.L. DCNET Internet Clock Service. DARPA Network
Group Report RFC-778, COMSAT Laboratories, April 1981.
4. Mills, D.L. Internet Delay Experiments. DARPA Network
Group Report RFC-889, M/A-COM Linkabit, December 1983.
5. Mills, D.L. DCN Local-Network Protocols. DARPA Network
Group Report RFC-891, M/A-COM Linkabit, December 1983.
6. Postel, J. Internet Control Message Protocol. DARPA
Working Group Report RFC-792, USC Information Sciences Institute
September 1981.
7. Postel, J. Time Protocol. DARPA Network Working Group
RFC-868, USC Information Sciences Institute, May 1983.
8. Postel, J. Daytime Protocol. DARPA Network Working Group
RFC-867, USC Information Sciences Institute, May 1983.
Mills [Page 26]
RFC 957 September 1985
Experiments in Network Clock
9. Su, Z. A Specification of the Internet Protocol (IP)
Option. DARPA Network Working Group Report RFC-781.
International, May 1981.
10. Marzullo, K., and S. Owicki. Maintaining the Time in
Distributed System. ACM Operating Systems Review 19, 3 (
1985), 44-54.
11. Mills, D.L. Algorithms for Synchronizing Network Clocks.
Network Working Group Report RFC-956, M/A-COM Linkabit,
1985.
12. Mills, D.L. Network Time Protocol (NTP). DARPA Network
Group Report RFC-958, M/A-COM Linkabit, September 1985.
Mills [Page 27]
if you see any problems within the linking, don't worry be happy,
this is version 0.1 of the Relevance System and you gotta expect some crappy subroutines sometimes,
just be content we did not write this in Java, which would have made this "bigger and better" HAHAHHA.
RFC documents can be found at I.E.T.F.
Relevance System Copyright © 2002 Spectrum WorldResearch
other technical nosh by ServerMasters Corporation
collaboration of BobX
