Internet2 IP Network Operations Report
09/24/2007 - 09/30/2007

Internet2 IP Network Operations Center
at Indiana University

New Customers

None


Upgrades

None

New Services

None



Unscheduled Outages Summary







Ticket Number
Customer Impact
Network Impact
Title
Outage Type
Start Time (UTC)

End Time (UTC)





109951-Critical
1-Critical

Internet2 IP Network Core Node CHIC Available Update
Software
09/24/2007 2:26 PM
09/24/2007 2:28 PM


109961-Critical
1-Critical

Internet2 IP Network Core Node NEWY Available
Software
09/24/2007 4:07 PM
09/24/2007 4:09 PM


110391-Critical
1-Critical

Internet2 IP Network Core Node WASH FPC0 Available
Software
09/28/2007 3:07 PM
09/28/2007 3:11 PM






Unscheduled Outages Detail






Ticket No.:10995:45



Subject:Internet2 IP Network Core Node CHIC Available Update



Affected:Connectors, Peers, and Participants off of FPC2
BackBone Circuit ATLA-CHIC



Start Time:Monday, September 24, 2007, 2:26 PM (1426) UTC



End Time:Monday, September 24, 2007, 2:28 PM (1428) UTC


Description:Internet2 IP Network Core Node CHIC was experiencing
instability.  The Node will be reloaded with new
software and is now stable.







Ticket No.:10996:45



Subject:Internet2 IP Network Core Node NEWY Available



Affected:Backbone circuit NEWY-WASH
IXP: MAN LAN
Connector: NOX
Peer: SINET, Surfnet



Start Time:Monday, September 24, 2007, 4:07 PM (1607) UTC



End Time:Monday, September 24, 2007, 4:09 PM (1609) UTC


Description:Internet2 IP Network Core Node NEWY FPC Card Crashed,
traffic was only affected for connectors on FPC 0 and
the backbone to WASH.  Internet2 Engineers determined
the crash was due to a software bug.  Code on the NEWY
core node was updated to resolve the issue.







Ticket No.:11039:45



Subject:Internet2 IP Network Core Node WASH FPC0 Available



Affected:Backbone Circuit NEWY-WASH
IXP NGIX-East and its Peers
Connector OARNet



Start Time:Friday, September 28, 2007, 3:07 PM (1507) UTC



End Time:Friday, September 28, 2007, 3:11 PM (1511) UTC


Description:There was a brief crash of FPC0 in the WASH T640 due to
a software bug. A software upgrade tonight is expected
to resolve the bug issue. Please note that only the
above listed entities were affected, not the entire
Core Node.







Scheduled Maintenances Summary





Ticket Number
Customer Impact
Network Impact
Title

Maintenance Type
Start Time (UTC)
End Time (UTC)





11000
3-Elevated
1-Critical

Internet2 IP Network Core Node CHIC Emergency Maintenance Completed
Software
09/25/2007 6:39 AM
09/25/2007 6:43 AM


11009
3-Elevated
1-Critical

Internet2 IP Network Core Node NEWY Maintenance Completed
Software
09/26/2007 9:17 AM
09/26/2007 9:19 AM


11008
3-Elevated
2-High

Internet2 IP Network Backbone Circuit CHIC-NEWY Maintenance Completed
Circuit
09/27/2007 5:19 AM
09/27/2007 5:25 AM


11023
3-Elevated
1-Critical

Internet2 IP Network Core Node WASH Emergency Maintenance Completed
Software
09/29/2007 9:21 AM
09/29/2007 9:23 AM


11024
3-Elevated
1-Critical

Internet2 IP Network Core Node ATLA Emergency Maintenance Completed
Software
09/29/2007 9:40 AM
09/29/2007 9:43 AM


11025
3-Elevated
1-Critical

Internet2 IP Network Core Node KANS Emergency Maintenance Completed
Software
09/29/2007 10:00 AM
09/29/2007 10:03 AM


11026
3-Elevated
1-Critical

Internet2 IP Network Core Node HOUS Emergency Maintenance Completed
Software
09/29/2007 10:20 AM
09/29/2007 10:22 AM


11027
3-Elevated
1-Critical

Internet2 IP Network Core Node SEAT Emergency Maintenance Completed
Software
09/29/2007 10:40 AM
09/29/2007 10:42 AM


11028
3-Elevated
1-Critical

Internet2 IP Network Core Node SALT Emergency Maintenance Completed
Software
09/29/2007 11:00 AM
09/29/2007 11:03 AM


11029
3-Elevated
1-Critical

Internet2 IP Network Core Node LOSA Emergency Maintenance Completed
Software
09/29/2007 11:20 AM
09/29/2007 11:22 AM




Scheduled Maintenances Detail





Ticket No.:11000:45



Subject:Internet2 IP Network Core Node CHIC Emergency Maintenance Completed



Affected:Connectors, Peers, and Participants Off Of FPC2
BackBone Circuit ATLA-CHIC



Start Time:Tuesday, September 25, 2007, 6:39 AM (0639) UTC



End Time:Tuesday, September 25, 2007, 6:43 AM (0643) UTC


Description:   Internet2 IP Network Core Node CHIC was briefly
unavailable to the community while Internet2 Engineers
performed a software reload. Maintenance has been
completed.
*Traffic re-routed.



  







Ticket No.:11009:45



Subject:Internet2 IP Network Core Node NEWY Maintenance Completed



Affected:Core Node NEWY and its Connectors, Peers, and Participants
IP Backbone Circuits: NEWY-WASH, CHIC-NEWY
IXP: MAN LAN



Start Time:Wednesday, September 26, 2007, 9:17 AM (0917) UTC



End Time:Wednesday, September 26, 2007, 9:19 AM (0919) UTC


Description:   Internet2 Engineers performed an emergency software
update on the New York Internet2 core node. A brief
outage was detected. Maintenance has been completed.



  







Ticket No.:11008:45



Subject:Internet2 IP Network Backbone Circuit CHIC-NEWY Maintenance Completed



Affected:IP Backbone Circuit CHIC-NEWY
Grooming Backbone Circuit BOST-NEWY
ESNET IP Backbone CHIC-NEWY
ESNET SDN Backbone Circuit CHIC-NEWY
HOPI Backbone CHIC-NEWY
Connector NoX



Start Time:Thursday, September 27, 2007, 5:19 AM (0519) UTC



End Time:Thursday, September 27, 2007, 5:25 AM (0525) UTC


Description:   Level(3) technicians replaced a faulty OAM
card at the Level(3) facility in Chepachet, RI.
Estimated downtime during this maintenance was expected
to be 5 minutes maintenance has been completed.



  







Ticket No.:11023:45



Subject:Internet2 IP Network Core Node WASH Emergency Maintenance Completed



Affected:Core Node WASH and Its Connectors, Participants, and Peers



Start Time:Saturday, September 29, 2007, 9:21 AM (0921) UTC



End Time:Saturday, September 29, 2007, 9:23 AM (0923) UTC


Description:   Internet2 engineers performed software upgrades on the
Internet2 core nodes.  Maintenance has been completed.



  







Ticket No.:11024:45



Subject:Internet2 IP Network Core Node ATLA Emergency Maintenance Completed



Affected:Core Node ATLA and Its Connectors, Participants, and Peers



Start Time:Saturday, September 29, 2007, 9:40 AM (0940) UTC



End Time:Saturday, September 29, 2007, 9:43 AM (0943) UTC


Description:   Internet2 engineers performed software upgrades on the
Internet2 core nodes.  Maintenance has been completed.



  







Ticket No.:11025:45



Subject:Internet2 IP Network Core Node KANS Emergency Maintenance Completed



Affected:Core Node KANS and Its Connectors, Participants, and Peers



Start Time:Saturday, September 29, 2007, 10:00 AM (1000) UTC



End Time:Saturday, September 29, 2007, 10:03 AM (1003) UTC


Description:   Internet2 engineers performed software upgrades on the
Internet2 core nodes.  Maintenance has been completed.



  







Ticket No.:11026:45



Subject:Internet2 IP Network Core Node HOUS Emergency Maintenance Completed



Affected:Core Node HOUS and Its Connectors, Participants, and Peers



Start Time:Saturday, September 29, 2007, 10:20 AM (1020) UTC



End Time:Saturday, September 29, 2007, 10:22 AM (1022) UTC


Description:   Internet2 engineers performed software upgrades on the
Internet2 core nodes.  Maintenance has been completed.



  







Ticket No.:11027:45



Subject:Internet2 IP Network Core Node SEAT Emergency Maintenance Completed



Affected:Core Node SEAT and Its Connectors, Participants, and Peers



Start Time:Saturday, September 29, 2007, 10:40 AM (1040) UTC



End Time:Saturday, September 29, 2007, 10:42 AM (1042) UTC


Description:   Internet2 engineers performed software upgrades on the
Internet2 core nodes.  Maintenance has been completed.



  







Ticket No.:11028:45



Subject:Internet2 IP Network Core Node SALT Emergency Maintenance Completed



Affected:Core Node SALT and Its Connectors, Participants, and Peers



Start Time:Saturday, September 29, 2007, 11:00 AM (1100) UTC



End Time:Saturday, September 29, 2007, 11:03 AM (1103) UTC


Description:   Internet2 engineers performed software
upgrades on the Internet2 core nodes. Maintenance has
been completed.



  







Ticket No.:11029:45



Subject:Internet2 IP Network Core Node LOSA Emergency Maintenance Completed



Affected:Core Node LOSA and Its Connectors, Participants, and Peers



Start Time:Saturday, September 29, 2007, 11:20 AM (1120) UTC



End Time:Saturday, September 29, 2007, 11:22 AM (1122) UTC


Description:   Internet2 engineers performed software upgrades on the
Internet2 core nodes.  Maintenance has been completed.



  










Downtime and Availability







Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability


** CHIC T6400 hr 0 min100.00000%100.00000%
DNVR T6400 hr 0 min100.00000%99.99962%
HSTN T6400 hr 0 min100.00000%99.99409%
LOSA T6400 hr 0 min100.00000%99.98741%
** NEWY T6400 hr 4 min99.96032%99.99657%
SNVA T6400 hr 0 min100.00000%99.99657%
** SEAT T6400 hr 2 min99.98016%99.99847%
** WASH T6400 hr 0 min100.00000%100.00000%
** HOUS T6400 hr 2 min99.98016%99.99962%
** LOSA T6400 hr 2 min99.98016%99.99962%
ATLA M50 hr 0 min100.00000%99.97844%
** ATLA T6400 hr 3 min99.97024%99.99618%
** SALT T6400 hr 3 min99.97024%99.99866%
** KANS T6400 hr 3 min99.97024%99.99847%
      
Aggregate Internet2 IP Core Nodes0 hr 19 min99.98654%99.99598%

** Less than 52 weeks of data available

Internet2 IP Backbone CircuitsDown TimeReporting Period Availability39 Week Availability




DNVR-SNVA OC-1920 hr 0 min100.00000%100.00000%
DENV-STTL OC-1920 hr 0 min100.00000%99.97100%
HSTN-LOSA OC-1920 hr 0 min100.00000%99.98016%
LOSA-SNVA OC-1920 hr 0 min100.00000%100.00000%
ATLA-HSTN OC-1920 hr 0 min100.00000%99.62327%
SNVA-STTL OC-1920 hr 0 min100.00000%99.78353%
** I2-CHIC-ATLA0 hr 4 min99.96032%99.10333%
** I2-CHIC-NEWY0 hr 8 min99.92063%99.49379%
** I2-CHIC-KANS0 hr 0 min100.00000%100.00000%
** I2-CHIC-WASH0 hr 0 min100.00000%100.00000%
** I2-NEWY-WASH0 hr 8 min99.92063%99.97583%
** I2-WASH-ATLA0 hr 0 min100.00000%99.98270%
** I2-ATLA-HOUS0 hr 0 min100.00000%99.91606%
** I2-KANS-ATLA0 hr 0 min100.00000%99.78149%
** I2-KANS-SALT0 hr 0 min100.00000%100.00000%
** I2-SALT-SEAT0 hr 0 min100.00000%100.00000%
** I2-HOUS-LOSA0 hr 0 min100.00000%99.95370%
** I2-LOSA-SALT0 hr 0 min100.00000%100.00000%
** I2-LOSA-SEAT0 hr 0 min100.00000%99.99949%
      
Aggregate All Internet2 IP Backbone Circuits0 hr 20 min99.98956%99.87181%






Upcoming Scheduled Maintenances





Ticket No.:10980:45



Subject:Internet2 IP Network Backbone Circuits HOUS-LOSA Maintenance



Affected:Backbone Circuit HOUS-LOSA



Start Time:Thursday, October 4, 2007, 5:00 AM (0500) UTC



End Time:Thursday, October 4, 2007, 7:00 AM (0700) UTC



Description:   Backbone circuit HOUS-LOSA will be briefly unavailable to
the Internet 2 IP Network community. Level(3)
Communications will be performing fiber maintenance in
Austin TX. No impact is expected. However, the entire
maintenance window will be observed.



  







Ticket No.:11002:45



Subject:Internet2 IP Network Backbone Circuit HOUS-LOSA Maintenance



Affected:IP Backbone Circuit HOUS-LOSA
Grooming Backbone Circuit ELPA-HOUS
Grooming Backbone Circuit HOUS-TULS



Start Time:Thursday, October 4, 2007, 5:00 AM (0500) UTC



End Time:Thursday, October 4, 2007, 11:00 AM (1100) UTC



Description:   Level(3) Communications Technicians will be relocating
fiber optic
cables for a highway right-of-way project in Houston,
TX. The cable has already been relocated, and now
traffic will need to be hot cut over to it. An outage
of about 1 hour is expected; however, the entire window
is reserved.