Internet2 IP Network Operations Report
07/14/2014 - 07/20/2014

Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None

New Services

None



Unscheduled Outages Summary







Ticket Number
Customer Impact
Network Impact
Title
Outage Type
Source Of Impact
Start Time (UTC)

End Time (UTC)





234472-High
2-High

Outage Resolved - I2 IP Backbone Various Backbones
HardwareVendor
07/14/2014 4:33 AM
07/14/2014 5:58 AM


234492-High
2-High

Brief Outage Resolved - I2 IP Backbone HOUS-LOSA
PowerVendor
07/14/2014 5:38 PM
07/14/2014 5:42 PM






Unscheduled Outages Detail






Ticket No.:23447:45



Subject:Outage Resolved - I2 IP Backbone Various Backbones



Source Of Impact:Vendor



Affected:Backbone LOSA-WASH
Backbone NEWY32AOA-WASH
Backbone CLEV-WASH
Backbone ATLA-WASH
Backbone CHIC-WASH
Connector OARnet
Connector 3ROX
Connector CAAREN
Connector SOX
Connector MAX GigaPOP
Connector MARIA
Peer SINET
Peer RedCLARA
Peer RNP
Peer CENIT
Peer CKLN
Peer GEANT
Participant ARE-ON
Peer Amazon
Participant University of Pennsylvania



Start Time:Monday, July 14, 2014, 4:33 AM (0433) UTC



End Time:Monday, July 14, 2014, 5:58 AM (0558) UTC


Description:The peers listed above are now available.  Internet2 Engineers report
that the reason for outage was a management module failure in the AL2S
network which
stopped pushing openflow rules to the switch. No further outages are
expected.  Backbone traffic re-routed during this outage event.







Ticket No.:23449:45



Subject:Brief Outage Resolved - I2 IP Backbone HOUS-LOSA



Source Of Impact:Vendor



Affected:Backbone HOUS-LOSA



Start Time:Monday, July 14, 2014, 5:38 PM (1738) UTC



End Time:Monday, July 14, 2014, 5:42 PM (1742) UTC


Description:Backbone HOUS-LOSA is now available following a brief outage. Internet2
Fiber Provider Engineers report the reason for outage was due to a power event that has been resolved.  IP traffic will
re-route.







Scheduled Maintenances Summary





Ticket Number
Customer Impact
Network Impact
Title

Maintenance Type
Source Of Impact
Start Time (UTC)
End Time (UTC)





23425
3-Elevated
1-Critical

Maintenance Completed - I2 IP Backbone ATLA-HOUS
Software
Vendor07/16/2014 5:01 AM
07/16/2014 8:12 AM


23429
3-Elevated
1-Critical

Maintenance Completed - I2 IP Backbone HOUS-KANS
SoftwareVendor
07/16/2014 6:10 AM
07/16/2014 7:31 AM


23401
3-Elevated
2-High

Maintenance Completed - I2 IP Backbones ATLA-CHIC
Hardware
Vendor07/17/2014 7:16 AM
07/17/2014 7:19 AM




Scheduled Maintenances Detail





Ticket No.:23425:45



Subject:Maintenance Completed - I2 IP Backbone ATLA-HOUS



Source Of Impact:Vendor



Affected:Backbone ATLA-HOUS



Start Time:Wednesday, July 16, 2014, 5:01 AM (0501) UTC



End Time:Wednesday, July 16, 2014, 8:12 AM (0812) UTC


Description:   Backbone ATLA-HOUS was unavailable while Internet2 Engineers
performed a Software Upgrade.  he maintenance has been
completed. The upgrade has been reverted; notification will be
provided for a new maintenance window.
IP traffic was re-routed during this maintenance.



  







Ticket No.:23429:45



Subject:Maintenance Completed - I2 IP Backbone HOUS-KANS



Source Of Impact:Vendor



Affected:Backbone HOUS-KANS



Start Time:Wednesday, July 16, 2014, 6:10 AM (0610) UTC



End Time:Wednesday, July 16, 2014, 7:31 AM (0731) UTC


Description:   Backbone HOUS-KANS was unavailable while Internet2 Engineers
performed a Software Upgrade.  The upgrade was reverted; notifcation
will be provided for a new maintenance window.
IP traffic was re-routed during this maintenance.



  







Ticket No.:23401:45



Subject:Maintenance Completed - I2 IP Backbones ATLA-CHIC



Source Of Impact:Vendor



Affected:Backbone ATLA-CHIC
ION Backbone ATLA-CHIC
Connector KyRON (CHIC)*
Connector Indiana Gigapop
Connector SOX



Start Time:Thursday, July 17, 2014, 7:16 AM (0716) UTC



End Time:Thursday, July 17, 2014, 7:19 AM (0719) UTC


Description:   Backbone ATLA-CHIC was briefly unavailable while Internet2's fiber
vendor performed hardware maintenance in Louisville, KY.
Maintenance has been completed. No further outages are expected. IP
traffic re-routed during this maintenance.



  










Downtime and Availability







Internet2 IP Core Nodes
Down TimeReporting Period Availability52 Week Availability


ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99355%  99.99867%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98596%  100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99089%  99.99924%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99734%  100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99639%  100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99184%  100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943%  100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99317%  99.99981%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99639%  100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98843%  100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
      
Aggregate Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%99.99445%99.99981%

** Less than 52 weeks of data available
 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  
	    

	    
Internet2 IP Backbone CircuitsDown Time
Reporting Period Availability52 Week Availability


R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181
               0 hr 3 min0 hr 0 min
               99.97024%100.00000%
               99.76624%99.95560%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182
               3 hr 11 min0 hr 0 min
               98.10516%100.00000%
               99.64443%99.89488%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179
               1 hr 25 min1 hr 25 min
               99.15675%99.15675%
               99.77136%99.82525%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.58978%99.74860%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.92126%99.97723%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97230%99.99924%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178
               1 hr 25 min1 hr 25 min
               99.15675%99.15675%
               99.84479%99.96566%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184
               1 hr 21 min0 hr 0 min
               99.19643%100.00000%
               99.62337%99.71748%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.58428%99.68333%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.93131%99.97002%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.94592%100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.95465%99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.94991%99.99905%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176
               1 hr 25 min1 hr 25 min
               99.15675%99.15675%
               99.92866%99.93985%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98710%99.99924%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98729%99.99943%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318
               1 hr 25 min1 hr 25 min
               99.15675%99.15675%
               99.72924%99.81349%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419
               0 hr 3 min0 hr 0 min
               99.97024%100.00000%
               99.89507%99.96774%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.34047%99.73399%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.84707%99.85030%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.84707%99.85030%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.74632%99.76852%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.95048%99.98710%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.95048%99.98710%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96053%99.99981%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.82886%99.83891%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.64879%99.75011%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559
               0 hr 4 min0 hr 4 min
               99.96032%99.96032%
               99.51882%99.60705%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.84384%99.90930%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.84366%99.90912%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.74992%99.80666%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96243%99.99772%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.95921%99.99772%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.78825%99.91177%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98956%99.99943%
      
Aggregate All Internet2 IP Backbone Circuits
                   10 hr 22 min4 hr 38 min
                   99.83323%99.90777%
                   99.83251%99.90162%


 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  
*  Internet2 IP 100G Backbone Circuits Ride over AL2S for connectivity.

Upcoming Scheduled Maintenances





Ticket No.:23458:45



Subject:Maintenance - I2 IP Backbones KANS-SALT



Affected:Backbone KANS-SALT
ION Backbone KANS-SALT



Start Time:Wednesday, July 23, 2014, 6:00 AM (0600) UTC



End Time:Wednesday, July 23, 2014, 10:00 AM (1000) UTC



Description:   Backbone KANS-SALT will be unavailable while Internet2's fiber provider performs hardware maintenance in Bethune, CO. Outages are expected; the entire window is reserved.  IP traffic
will re-route during this maintenance.



  







Ticket No.:23475:45



Subject:Maintenance - I2 IP Backbone KANS-SALT



Affected:Backbone KANS-SALT
ION Backbone KANS-SALT



Start Time:Friday, July 25, 2014, 10:00 AM (1000) UTC



End Time:Friday, July 25, 2014, 12:00 PM (1200) UTC



Description:   The above listed will become available while vendor engineers perform
maintenance to replace a faulty CFP in Denver, CO.  Outages are
anticipated and the entire maintenance window has been reserved.



  







Ticket No.:23478:45



Subject:Maintenance 1 of 2 - I2 IP Backbone ATLA-CHIC



Affected:Backbone ATLA-CHIC
Connector MissION
Connector KyRON



Start Time:Friday, August 1, 2014, 4:00 AM (0400) UTC



End Time:Friday, August 1, 2014, 10:00 AM (1000) UTC



Description:   Backbone ATLA-CHIC will be unavailable while fiber provider Engineers
perform maintenance.  An outage  is expected, and the entire window is
reserved.  IP traffic will re-route during this maintenance. This is
the first of two maintenance windows.



  







Ticket No.:23479:45



Subject:Maintenance 2 of 2 - I2 IP Backbone ATLA-CHIC



Affected:Backbone ATLA-CHIC
Connector MissION
Connector KyRON



Start Time:Saturday, August 2, 2014, 4:00 AM (0400) UTC



End Time:Saturday, August 2, 2014, 10:00 AM (1000) UTC



Description:   Backbone ATLA-CHIC will be unavailable while fiber provider Engineers
perform maintenance.  An outage  is expected, and the entire window is
reserved.  IP traffic will re-route during this maintenance. This is
the second of two maintenance windows.



  







Ticket No.:23465:45



Subject:Maintenance *Tracking* - I2 IP Node oob.kans & oob.chic



Affected:



Start Time:Thursday, August 7, 2014, 5:00 AM (0500) UTC



End Time:Thursday, August 7, 2014, 9:00 AM (0900) UTC



Description:   Node oob.kans may be unavailable while Level 3 Engineers perform 
maintenance.  Level 3 has indicated a possible outage of up to 30 minutes, 
but Internet2 does not expect this to be service-impacting.