As per Relevance of the word automatic, we have this rfc below:
Network Working Group D.
Request for Comments: 1297 Merit Network, Inc
January 1992
NOC Internal Integrated Trouble Ticket
Functional Specification
("NOC TT REQUIREMENTS")
Status of the
This memo provides information for the Internet community. It
not specify an Internet standard. Distribution of this memo
unlimited
Professional quality handling of network problems requires some
of problem tracking system, herein referred to as a "trouble ticket
system. A basic trouble ticket system acts like a hospital chart
coordinating the work of multiple people who may need to work on
problem
Once the basic trouble ticket system is in place, however, there
many extensions that can aid Network Operations efficiency
Information in the tickets can be used to produce
reports. Operator efficiency and accuracy may be increased
automating trouble ticket entry with information from the
Alert system. The Alert system may be used to monitor trouble
progress. Trouble tickets may be also used to communicate
health information between NOCs, to telcom vendors, and to
internal sales and engineering audiences
This document explores competing uses, architectures, and
features of integrated internal trouble ticket systems for
and other Operations Centers
This RFC describes general functions of a Trouble Ticket system
could be designed for Network Operations Centers. The document
being distributed to members of the Internet community in order
stimulate discussions of new production-oriented operator-
application tools for network operations. Hopefully, this
result both in more ideas for improving NOC performance, and in
available tools that incorporate those ideas
Johnson [Page 1]
RFC 1297 NOC TT REQUIREMENTS January 1992
PURPOSES OF A NOC TROUBLE TICKET
A good Network Operations Trouble Ticket System should serve
purposes
1) SHORT-TERM MEMORY AND COMMUNICATION ("Hospital Chart").
primary purpose of the trouble ticket system is to act as short
term memory about specific problems for the NOC as a whole. In
multi-operator or multi-shift NOC, calls and problem updates
in without regard to who worked last on a particular problem
Problems extend over shifts, and problems may be addressed
several different operators on the same shift. The trouble
(like a hospital chart) provides a complete history of
problem, so that any operator can come up to speed on a
and take the next appropriate step without having to consult
other operators who are working on something else, or have
home, or are on vacation. In single-room NOCs, an operator
ask out loud if someone else knows about or is working on
problem, but the system should allow for more formal
as well
2) SCHEDULING and WORK ASSIGNMENT. NOCs typically work with
simultaneous problems with different priorities. An on-
trouble ticket system can provide real time (or even
displayed and updated) lists of open problems, sorted by priority
This would allow operators to sort their work at the beginning
a shift, and to pick their next task during the shift. It
would allow supervisors and operators to keep track of the
NOC workload, and to call in and assign additional staff
appropriate
It may be useful to allow current priorities of tickets
according to time of day, or in response to timer alerts
3) REFERRALS AND DISPATCHING. If the trouble ticket system
thoroughly enough integrated with a mail system, or if the
is used by Network Engineers as well as Network Operators,
some problems can be dispatched simply by placing the
Engineer or Operator name in an "assigned to" field of the
ticket
4) ALARM CLOCK. Typically, most of the time a trouble ticket
open, it is waiting for something to happen. There should
always be a timer associated with every wait. If a ticket
referred to a phone company, there will be an escalation
before which the phone company is supposed to call back with
update on the problem. For tickets referred to remote
personnel, there may be other more arbitrary timeouts such
Johnson [Page 2]
RFC 1297 NOC TT REQUIREMENTS January 1992
"Monday morning". Tickets referred to local engineers
programmers should also have timeouts ("Check in a couple of
if you don't hear back from me"). A good trouble ticket
will allow a timeout to be set for each ticket. This alarm
generate an alert for that ticket at the appropriate time
Preferably, the system should allow text to be attached to
timer with a shorthand message about what the alert
("Remind Site: TT xxx") (The full story can always be found
checking the trouble ticket). These alerts should feed into
NOC's standard alert system
The Alarm Clock can also assist (or enforce!)
escalation. An escalation timer could automatically be set
on the type of network, severity of the problem, and the time
outage occurred
5) OVERSIGHT BY ENGINEERS AND CUSTOMER/SITE REPRESENTATIVES.
frequently operate more than one network, or at least have
(engineers, customer representatives, etc) who are responsible
subsets of the total network. For these
representatives, summaries of trouble tickets can be filtered
network or by node, and delivered electronically to the
engineers or site representatives. Each of these reports
a summary of the previous day's trouble tickets for those sites,
listing of older trouble tickets still open, and a section
recurrent problems. These reports allow the site reps to
aware the current outages and trends for their particular sites
The trouble ticket system also allows network access to the
details of individual trouble tickets, so those receiving
general reports can get more detail on any of their problems
referencing the trouble ticket number
6) STATISTICAL ANALYSIS. The fixed-form fields of trouble
allow categorizations of tickets, which are useful for
equipment and NOC performance. These include, Mean Time
Failure and Mean Time to Repair reports for specific equipment
The fields may also be of use for generating statistical
control reports, which allow deteriorating equipment to
detected and serviced before it fails completely.
breakdowns by network a NOC costs to be apportioned appropriately
and help in developing staffing and funding models. A
trouble ticket system should make this statistical information
a format suitable for spreadsheets and graphics programs
7) FILTERING CURRENT ALERTS. It would be possible to use
status information from the trouble ticket system to filter
alerts that are displayed on the alert system. For instance,
node XXX is known to be down because the trouble ticket
Johnson [Page 3]
RFC 1297 NOC TT REQUIREMENTS January 1992
currently open on it, the alert display for that node
automatically be acknowledged. Trouble tickets could
contain much further information useful for expert system
of current network alert information
8) ACCOUNTABILITY ("CYA"), FACILITATING CUSTOMER FOLLOW-THROUGH
AND NOC IMAGE). Keeping user-complaint tickets facilities
kind of follow through with end-users that generates happy
(and good NOC image) for normal trouble-fixing situations.
also, by their nature, NOCs deal with crises; they
find themselves with major outages, and angry users
administrators. The trouble ticket system documents the NOC'
(and the rest of the organization's) efforts to solve problems
case of complaints
FIXED FIELDS, FREE-FORM FIELDS, and TT
Information in trouble tickets can be placed in either fixed
freeform fields. Fixed fields have the advantage that they can
used more easily for searches. A series of fixed fields also acts
a template, either encouraging or requiring the operators to fill
certain standard data. Fixed fields can facilitate data
(e.g., making sure an entered name is in an attached
database, or verifying that a phone number consists of ten
characters). Fixed fields are also appropriate for data that
automatically entered by the system, such as the operator's login id
the name of the node that was clicked on if the trouble ticket
opened via an alert tool, or names and phone numbers that
automatically entered into the ticket based on other entries (e.g.,
filling in a contact name and phone based on a machine name).
Unfortunately, fixed fields work best where the problem-
environment is uniform, well-understood, and stable; that is,
tickets work best when their fields are well tailored to the
problem at hand. It is easy to set up a large number of fields (
even required fields) that are irrelevant to a given problem;
slows down and confuses the operators. Adding structure and
checking to a field tends to make the data more consistent
reliable, but it also tends to force the operators into
procedures like menus to get the get the data accepted by the system
It also forces there to be more maintenance on those
systems (adding new entries as they become new legal options), and
some ways it reduces the accuracy of the system by forcing
to choose "canned" or authorized responses that may not
represent the situation accurately. Where statistical
reports are a primary purpose of the trouble ticket system,
fixed fields may be appropriate. If the primary intent of the
is to keep notes for individual problems and to
Johnson [Page 4]
RFC 1297 NOC TT REQUIREMENTS January 1992
communication between operators, then fixed fields may tend to be
hindrance. One reasonable guideline would be that fixed fields
used ONLY where they are automatically filled in by the
system, or where the information in that field is explicitly used
a report or standard search procedure
Because of this close relationship between the structure of
ticket and the problem to be solved, it is very very useful to
able to define different ticket types for different classes
problems. This becomes even more true for those many NOCs
staff are responsible for other types of operations:
operations, workstation administration, help desk functions, or
of the other real-time response functions. Network operations
justify the expense of an operations center. This kind of
makes economic sense, and is becoming more prevalent. In these
of situations it is vital that the same tools that are used
network operations also be available for the other operations.
means that the trouble ticket configurations need to be modifiable
local staff. Commercial RDBMS forms builder and report
packages and "fourth-generation languages" offer a good start
this, although it is sometimes difficult to integrate full
ticket functionality through these systems
TROUBLE TICKET
1) HEADERS. Inevitably, a trouble ticket begins with a number
fixed fields. These generally include
Time and Date of problem start
Initials or signon of the operator opening the ticket
Severity of the problem (possibly separating the "
severity" and the "NOC priority", since these could be different).
A one-line description of the problem for use in reports
There can be many other fixed fields for specific purposes.
may also be different kinds of tickets for different problems,
the ticket format differs mainly in fixed fields. These include
Who reported the problem? (Name, organization, phone
email address
Machine(s) involved
Network involved (for multi-network NOCs).
User's machine address
Destination machine address
Next Action
Time and date for alarm on this ticket
Who should the ticket be dispatched to
Ticket "owner" (one person designated to be responsible overall).
Johnson [Page 5]
RFC 1297 NOC TT REQUIREMENTS January 1992
2) INCIDENT UPDATES. The main body of trouble tickets is usually
series of freeform text fields. Optimally, each of these fields
automatically marked with the time and date of the update, and
the signon of the operator making the update. Since updates
frequently recorded sometime after the problem is fixed, however,
is useful to allow the operators to override the current time
with the time the update was actually made. (In
implementations, both times will be kept internally).
The first incident update usually is a description of the problem
Since the exact nature of the problem is usually not known when
ticket is first opened, this description may be complex
imprecise. For problems that are reported by electronic mail, it
useful to be able to paste the original message in the ticket
particularly if it contains cryptic or extensive information (such
a user's traceroute output). At least one such arbitrarily-
freeform field seems necessary to contain this kind of output
although it is better to allow arbitrarily long messages at any
(e.g., so future complex messages can also be archived in
ticket).
Subsequent update fields may be as simple as "Called site;
answer". Some systems allow these kinds of updates to be coded
fixed fields; most use freeform text
There should always be an indication of what the next action for
ticket ought to be. Again, this may be implemented as a
fixed field, or by convention of using the last line of text
Advanced systems may also need a facility to allocate the amount
time a ticket is open between multiple sources. A serious NOC
want to use its trouble ticket system to statistically track
performance on responding to problems. (e.g., Mean Time
Failure and Mean Time To Repair reports). Frequently, though
repairs are stopped at the customer's request. ("It's not
important a machine and I don't feel like coming in--can you defer
until Monday Morning?"). In these cases the ticket needs to
open, but there needs to be a notation that the ticket is now
"customer time" rather than "NOC time". The durations of "
time" need to be excluded from MTBF and MTTR reports.
repairs could move back and forth between "NOC time" "customer time
repeatedly. This probably implies that each Incident Update may
a time and date of status change, and that these status changes
be read and aggregated by by reporting programs
3) RESOLUTION DATA. Once a problem is resolved, it is useful
summarize the problem for future statistical analysis. The
fields have been found to be useful
Johnson [Page 6]
RFC 1297 NOC TT REQUIREMENTS January 1992
- Time and Date of resulation (for outage duration).
- Durations (can be calculated from time of resolution
incident report "customer/NOC time" stamps).
- Resolution (one line of description of what happened,
reports).
- Key component affected (for MTBF and similar reports).
- Checked By -- a field for supervisors to sign off on
review
- Escalated to -- for reports on how many problems
non-NOC help
- Temp - a database field that can be used to store
"check marks" while making statistical investigations
USER, TROUBLE, and ENGINEERING Ticket System(s
The primary level of an Network Operations trouble ticket is
"problem" or "trouble": a single malfunctioning piece of hardware
software that breaks at some time, has various efforts to fix it,
eventually is fixed at some given time
The primary level of an Network Information Center ticket, however
might well be the "user complaint". A single network failure
well produce a large number of individual user phone calls and
"user complaint" tickets. A NIC may want to use tickets to
each one of these calls, e.g., to make sure each user is informed
satisfied about the eventual resolution of the single
problem
In addition, NOCs (or Engineering Staffs) may want to
systematic problems. The staff may know, for instance, that
particular router is old and fragile, or that a particular section
their network doesn't have enough redundancy. It may be useful
open an "Engineering Ticket" on these known problems, providing
place to record history and notes about the problem, for use
further engineering or funding discussions
Even further "Meta" tickets could be described, having to do
such issues as whether the current trouble ticket fields, reports
and operation procedures were sufficient to handle current problems
It would be very convenient to be able to build all of these
on the same platform, and to allow each type of ticket to
reference other types. Multiple "user complaint" tickets, then
might might explicitly point to a single "trouble" ticket.
trouble tickets representing independent failures would then point
a single "engineering" ticket, which described the
problem. Multiple engineering tickets could point to a single "meta
ticket, if appropriate
Johnson [Page 7]
RFC 1297 NOC TT REQUIREMENTS January 1992
ASSISTED ENTRY AND DATA
Data (particularly in fixed fields) is only useful for searching
it is entered in consistent formats. A trouble ticket system
to help operators fill these fields with the correct format
information. This can be done using assisted entry (menus
acceptable choices), verification routines which check
internal lists or external databases (see next section), or
computer checking
Some database systems allow a customized "help" screen to
associated with each field, helping new (and experienced)
by making context-sensitive trouble ticket system
available at every field
Very complicated help or operator-guidance systems can be built
of Expert System technology. This could be as simple as
screens, or help screens with database information inserted (e.g.,
site contact names and phone numbers). Or it could involve hints
the operator, based on current network conditions. Or it might
ask the operator to run tests and to type in the results. (
EXPERT SYSTEMS, below).
To be maximally efficient and useful, a Trouble Ticket system
to integrate well with most of the rest of the NOC tools.
include
1) OPERATOR WINDOW ENVIRONMENT. A NOC Operator needs access
many pieces of information simultaneously, and therefore is
served by a good windowing environment. The Trouble Ticket
needs to run within this larger windowing system, so that
operator can debug, consult databases, use Email, field alerts
and keep an eye out for other emergencies while working on
trouble ticket. It is also useful to be able to run two
ticket sessions simultaneously, for example, to allow an
to search for related tickets while he is in the middle
updating another ticket. Cut and Paste between these
screens is mandatory, to allow easy recording of technical
in the trouble tickets
2) ALERT MONITORING SYSTEM. Trouble tickets are often opened
response to machine alerts; it ought to be easy to open a
ticket directly from the alert tool. When a ticket is opened
way, information about the alert and the machine involved ought
be automatically filled into the ticket. (There are
opinions about whether trouble tickets ought to be
Johnson [Page 8]
RFC 1297 NOC TT REQUIREMENTS January 1992
automatically without operator intervention. This operator'
opinion is that an operator acknowledgement should be required
but this point is debated enough that designers of a new
probably should support either option).
The Alarm Clock feature of the trouble ticket system
generates alerts. These alerts ought to feed gracefully into
Alert Monitor system, so that the operators will get all of
alerts from one place
3) DATABASE CONNECTIONS. A good trouble ticketing system
query NOC databases to automatically fill out trouble
fields where possible. This can be used for
- Filling out Network Operator information (e.g., phone number
based on the NetOp's signon id
- Filling in contact information based on machine name
- Filling in circuit numbers based on link description
- Filling in alarm clock or escalation time fields based on
machine or link name and on time of day
- Filling in machine serial numbers based on configuration database
4) MACHINE QUERYABLE INFORMATION. It could also be possible for
trouble ticket system to make standard queries of the
itself when a trouble ticket is opened: e.g., the system
request and store current machine configurations whenever a
was opened for that machine. On some systems, hardware
numbers are obtainable by software query directly to the machine
5) ELECTRONIC MAIL. Problem notification often comes
electronic mail; it must be possible to easily open a ticket
include the original mail message within the ticket as part of
initial problem description. When extremely technical
come in from network engineers, it is useful to allow
messages to be included verbatim, rather than forcing
technical network operators to rephrase the messages or to
them into predefined formats. Later update messages should
be easily includable. Possibly: tickets could be
automatically for mail messages to certain mailboxes. A
system saying "Your request has been received and assigned
number ####" might be desirable
Information within trouble tickets must also be easily
(possibly just via the windowing system) for inclusion in
messages to engineers and others
Scheduled (e.g., daily) reports must also be easily generated
the Email system
Johnson [Page 9]
RFC 1297 NOC TT REQUIREMENTS January 1992
6) DISPATCHING AND NOTIFICATION SYSTEMS. An important real-
aspect of Network Operations is notifying users,
contacts, and administrators of various classes of problems.
rules for who gets notified of what can be very arbitrary
complex, and can involve electronic mail, notices in
conferences, automatic beeper pages, and synthesized
announcements. It would be good for a trouble ticket system
provide for automatic (or operator initiated) notification of
appropriate channels for the current ticket (based on network
machine, severity of problem, duration of the problem,
guidelines, etc).
Databases associated with the trouble ticket system may also
lists of specific people to contact about outages for
machines. These "who to inform" lists can facilitate
notification messages directly from the trouble ticket system
It may also be possible to dispatch experts directly from
trouble tickets system. IBM's ECCO system allows allows
to directly dispatch Service Engineers from machine interactions
Many NOCs also use computer hooked to modems to automatically
engineers. This kind of dispatching should be available
within the trouble ticket system (along with an automatic
into the trouble ticket that the engineer has been dispatched).
7) OTHER TROUBLE TICKET SYSTEMS. When the NOC generates a
ticket, it often immediately calls up a telco or another
NOC, who proceed to open their own ticket. The
Engineering Task Force User Connectivity Working Group is
proposing a national trouble ticket tracking system, which
need updating from individual NOC trouble ticket systems.
state-of-the-art trouble ticket system could have provisions
transferring tickets and ticket information in and out of
such systems
8) NETWORK ACCESS. Some older trouble ticket systems assumed
anyone with a need to access the information would obtain a
and learn to use that system. The range of people with a need
trouble ticket information is now too great to allow
assumption. A good system now needs to be able to support
query for tickets and summary reports, as well as Email
of scheduled reports
9) SUBROUTINE INTERFACE. To allow for ad-hoc and
unanticipated needs, the trouble ticket system needs to support
full-function set of subroutine calls. These subroutines
allow construction of further trouble ticket functionality not
specified
Johnson [Page 10]
RFC 1297 NOC TT REQUIREMENTS January 1992
10) EXPERT SYSTEMS. Network debugging is a very promising
for expert system and artificial intelligence applications.
such an algorithm should require access to the alert
system, configuration and change control systems, to the
itself, and also to the information in the trouble ticket system
A good future system then needs to make this information
(probably via the subroutine interface mentioned above), and
also allow the Network Operators to invoke the
intelligent debugging from within a trouble ticket (including
output as part of the ticket dialogue).
11) GRAPHICS/REPORT Capability. Statistical and
displays about trouble ticket data need to be compatible
tools used to generate reports, news letters, etc
OTHER CONSIDERATIONS
1) INTERACTIVE SPEED. The system must be fast enough to be
interactively. NetOps need to answer questions over the phone
real time; good answers cannot be given if every query takes
couple of minutes. More importantly, the NetOps need the
ticket system in order to get information necessary to fix
network. If looking for old or currently-open tickets takes
than a few seconds, it won't be done. If updates take very
to make, then updates won't be recorded, or they will be
long after the event (with corresponding loss of accuracy). (
Operators have asked for a single-line "update this ticket
this message" utility that would let them avoid even
the ticket for simple updates!) Any time spent waiting
NetOp productivity and Network reliability
2) BACKUPS AND RELIABILITY. The trouble ticket system
absolutely crucial to both immediate and long-term operation
the NOC. Good systems could back up all data several times
hour to an auxiliary processor. That processor should
accessible for immediate use in case of failure of the
system
3) HISTORY AND ARCHIVING. A trouble ticket system is
constantly-growing database system. Old tickets need to
removed from the system at some interval (a year? several years?)
and archived. These archives should also be restorable for long
term history processing
4) PRIVACY AND SECURITY. The ability to enter, append, and
tickets should be controlled by id and password.
should be specifiable on a per-field basis. General read
to tickets (or portions of tickets) also needs to be restricted
Johnson [Page 11]
RFC 1297 NOC TT REQUIREMENTS January 1992
or else NetOps will be reluctant to be full and candid in
reporting
There are quite a few ideas in this "Wishlist". Ultimately, what
Operations Center needs is a totally integrated set of tools
completely model all of its activities, and which integrates
with all backup, peer, and vendor NOCs. It is hard to imagine
this whole system could come out of a shrinkwrapped box, even
the local configuration. But most of these facilities do exist, now
in some system. Hopefully, this document will foster an
discussion of ways in which NOC operator-level tools are used in
operations, and will encourage systems implementors and vendors
bring some of this functionality to the aid of real operations.
might even inspire current Operations Centers to add useful
to their current operations
Security
This paper does not pose specific new security issues. The
described herein would be host database applications, however,
even distributed host database applications. All of the
security considerations for that kind of system would apply
Multiple classes of user access need to be specified for classes
ticket data. Possible security threats include disclosure of
information, disclosure of confidential material (e.g.,
numbers or home phone numbers), and denial of service to the
Operations Center leading to degradation of network service
Author's
Dale S.
Merit
1075 Beal
Ann Arbor, MI 48109-2112
Phone: (313) 936-2270
Email: dsj@merit.
Discussion/comments may be sent to noc-tt-req@merit.edu. The
is maintained by noc-tt-req-request@merit.edu
Johnson [Page 12]
if you see any problems within the linking, don't worry be happy,
this is version 0.1 of the Relevance System and you gotta expect some crappy subroutines sometimes,
just be content we did not write this in Java, which would have made this "bigger and better" HAHAHHA.
RFC documents can be found at I.E.T.F.
Relevance System Copyright © 2002 Spectrum WorldResearch
other technical nosh by ServerMasters Corporation
collaboration of BobX
