Advanced Trouble Ticket

Systems
Steve Peck
Operations Manager
Global Research NOC
I2 Joint Techs July 18, 2006

Advanced Trouble Ticket Systems

If you look at R&E networks as a system,
there are quite a few components that need to
interoperate well to achieve the desired result,
the advanced networking that we know and
love so well.

Advanced Trouble Ticket Systems

While the primary focus of advanced networking is
the incredible bandwidth, the dazzling speeds, the
amazing advanced services, and the remarkable
high-end applications that run over the networks,
there are some basic operational elements that help
provide the foundation of support for all of the
great gee-whiz bang stuff that gets the front page
coverage.

Advanced Trouble Ticket Systems

One of the basic operational elements that help
facilitate the R&E system is the Trouble Ticket
System (TTS).

With the tremendous changes and advancement in

networking, the trouble ticket system can no longer
be just basic, a glorified word processor, or a
simplistic database like thingy that doubles as a
work order system, or project management tool.

Advanced Trouble Ticket Systems

Advanced networks require

advanced trouble ticket
systems.

Advanced Trouble Ticket Systems

Flexibility
- While a TTS needs to be robust, it does need to
be lightweight and flexible enough for
customization and integration with other NOC
systems and tools.
- Large enterprise wide behemoths like Peregrine or
Remedy can be an obstacle to the flexibility
required of a NOC.

Advanced Trouble Ticket Systems

Flexibility
- The TTS should scale to the needs of both the

NOC and the user community.

- Members of the user community like to have a

place to look up the status of a ticket on the web.

NOCs should offer this service.
- Sometimes it is helpful for users to actually access

the system itself, with limited user permissions, to

look at their tickets, and even to provide updates.

Advanced Trouble Ticket Systems

Intuitive User Interface
- A helpful way to organize the user interface is to

nest certain fields within the schema so as to not

inundate the user with 17 trillion fields laid out on a
flat screen.
- Should organize certain fields into ticket types,

based on the action needed. These fields reside in

nested dependencies.
- This way related fields can be organized sensibly,

and only seen when needed.

Advanced Trouble Ticket Systems

Ticket Types

Advanced Trouble Ticket Systems

Scheduled Maintenance Ticket-Type Window

Advanced Trouble Ticket Systems

Provision/Modify/Decom Ticket Type Window

Advanced Trouble Ticket Systems

Another fundamental question asked with
dependent nested fields is Is this network
impacting?

Advanced Trouble Ticket Systems

Intuitive User Interface

- Define fields with different user permissions write or read only, some for Engineering, some for
the Service Desk (front-line staff), etc.
- Make certain fields mandatory before changing
the tickets status, handing it back to the other
group, and especially before closing the ticket.
- Helps to capture the information that really is
needed and necessary.

Advanced Trouble Ticket Systems

Advanced Reporting Capabilities

- Network

availability based on node, circuit, etc.

- Trend reporting:
- Specific device failures (cards, etc.)
- Problem source
- How long a ticket is in the hands of the Service
Desk, Engineering, Vendor, Customer.
- Other options, if system is robust enough

Advanced Trouble Ticket Systems

Advanced Reporting Capabilities

- Time

to resolution
- Business activity during a specific time frame
- How many tickets created
- What kind of tickets
- When created
- Created by who
- Break stats out by network or support
operation, etc.

Advanced Trouble Ticket Systems

Advanced Reporting Capabilities

- Make sure you have the granularity needed to
perform the reports and searches you need. More
fields may mean a bit more work, but it saves you
time because
- NOTHING can be WORSE than staying up all
night MANUALLY going through 215 outage
tickets, looking for RMA numbers for a report!!!

Advanced Trouble Ticket Systems

CARDINAL RULE OF TICKETING!

Your system is only as accurate as the

data you put in it!!!

Advanced Trouble Ticket Systems

Managing Work Flow

- Master Ticket and Sub-Task (parent/child
relationships)
- Helps with issues that require multiple separate
actions, often by different support groups (Field
Service Requests, RMAs, customer notification,
etc.)

- Helps keep tickets tidier, rather than the infamous

massively long ticket with everything in it including
the kitchen sink and the bit bucket.

Advanced Trouble Ticket Systems

Managing Work Flow

- How do you handle the issue of a problem with
one of your networks, causing a related problem
with another network you support?
- Each deserves a separate ticket, for record keeping
purposes, etc.
- Must have a way to automatically copy the original
ticket to the other network project (or queue), then
edit the new ticket as appropriate.

Advanced Trouble Ticket Systems

Managing Work Flow

- Need ways to segregate tickets by the specific
networks you are supporting.
- But also need ways to see an aggregation of
tickets, to help manage workflow, for reporting
purposes, etc.

Advanced Trouble Ticket Systems

Managing Work Flow

- An excellent method to manage NOC work flow is
through a series of Next Action fields: Item, Day,
Time.
- These can be viewed through a tool like TickMon.
Sort by date and time (expired, upcoming)
Lists Next Action Item
Sorts by network, support group, or user

Advanced Trouble Ticket Systems

Managing Work Flow

Advanced Trouble Ticket Systems

Managing Work Flow

- Automated escalation to management, network
administration, etc.
- Switch time to UTC.
- Links to internal documentation: field definitions,
time zone converters, databases, tools, linkage to
vendor sites for case and RMA info, shipping
updates (Fed Ex, etc.)

Advanced Trouble Ticket Systems

Customer Notification
- TTS should provide the raw data for customer
notification.
- Make sure editing is done in the ticketing system,
and not through the notification tool interface.
- This way the ticket information is always the most
up-to-date.

Advanced Trouble Ticket Systems

Customer Notification Tool

Advanced Trouble Ticket Systems

Calendar Functions
- Have the capability to automatically publish
calendars from the TTS for:
- Scheduled Maintenances
- Field Service Requests
- RMAs
- Maintenance Freezes
- Outages

Advanced Trouble Ticket Systems

Database Integration
- Find Circuit(s) from Route
- Find Element(s) from Element
- Find Circuit(s) from Customer
- Find Circuit(s) from Element
- Need solid database back-end in your TSS!!!

Advanced Trouble Ticket Systems

Trouble Ticket Exchange

- Designed to allow quick and clear sharing of
information between trouble ticket systems in a
secure and reliable environment.
- The exchange of this data is handled via
automated email between the ticket systems.
- Scripts and parsers in the respective ticket
systems receive such email and insert the data into
the appropriate tickets and fields.

Advanced Trouble Ticket Systems

Trouble Ticket Exchange

- The end user (home ticket system), will see their
own ticket properly updated with data from a
foreign user (external ticket system), and vice versa.
- Clearly developed workflow guidelines and rules
are needed to ensure a trouble-free ticket exchange.
- Loops between the ticketing systems are another
possible danger in an automated ticket exchange.

Advanced Trouble Ticket Systems

Trouble Ticket Exchange

- Potentially a very powerful tool for NOC-toNOC communication.
- Global NOC doing it within OSG between
Support Centers (IU -> Fermi Lab, EGEE)
- Currently not done very often. Difficult and time
consuming.
- Easier between like systems (Remedy to Remedy)

Advanced Trouble Ticket Systems

Trouble Ticket Exchange
- There are many challenges, as each ticketing

system has its own unique features, characteristics,

and restrictions.
- Trouble ticket exchange is an ever changing

process as ticket systems will evolve, schema

changes, etc. Serious challenge to keep updated.
- But how does this scale when your NOC needs to

exchange information and tickets with a number of

NOCs, each with their own TTS?

Advanced Trouble Ticket Systems

Conclusion
- With the challenges being presented to NOCs
today - optical networking, dynamic provisioning,
monitoring issues, security, etc, the need for
seemless and meaningful communication between
NOCs continues to grow.
- NOC-to-NOC communication is essential in the
advanced R&E world.

Advanced Trouble Ticket Systems

Conclusion
-The sharing of information between NOCs must
increase. We must view the NOCs, and the
important roles they play, as a key part of the R&E
system. TTS needs to have a central role in this.
- I invite the community to think about these
communication and operational issues.

Advanced Trouble Ticket Systems

QUESTIONS?

Advanced Trouble Ticket Systems

Steve Peck
Global Research NOC
speck@iu.edu

http://globalnoc.iu.edu