Generated Report for Internet2 IP

Internet2 IP Network Operations Report
04/07/2014 - 04/13/2014

Internet2 IP Network Operations Center
at Indiana University

New Connections

None

Upgrades

None

New Services

None

Unscheduled Outages Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Outage Type	*Start Time (UTC)*	*End Time (UTC)*

22754	2-High	2-High	Outage Resolved - I2 IP Backbone HOUS-KANS	Hardware	04/09/2014 3:59 PM	04/10/2014 5:37 AM
22767	3-Elevated	3-Elevated	Degradation Resolved - I2 IP Various Backbones (CHIC)	Hardware	04/10/2014 4:38 PM	04/10/2014 5:32 PM
22786	4-Normal	2-High	Outage Resolved - I2 IP Backbone HOUS-LOSA	Unannounced Maintenance	04/12/2014 5:38 AM	04/12/2014 6:19 AM

Unscheduled Outages Detail

Ticket No.:	22754:45
Subject:	Outage Resolved - I2 IP Backbone HOUS-KANS
Affected:	Backbone HOUS-KANS
Start Time:	Wednesday, April 9, 2014, 3:59 PM (1559) UTC
End Time:	Thursday, April 10, 2014, 5:37 AM (0537) UTC
Description:	Backbone HOUS-KANS is now available. Level(3) Communications has determined the problem to be a bad OCLD in opt.tuls. During the maintenance, the part was replaced and services have been restored. Maintenance has been completed

Ticket No.:	22767:45
Subject:	Degradation Resolved - I2 IP Various Backbones (CHIC)
Affected:	AL2S Core Node (CHIC) and its Connector(s) and Participants Backbones ATLA-CHIC, CHIC-CLEV, CHIC-KANS
Start Time:	Thursday, April 10, 2014, 4:38 PM (1638) UTC
End Time:	Thursday, April 10, 2014, 5:32 PM (1732) UTC
Description:	The above are now available following a degraded condition. Vendor engineers indicated that there was a hardware issue that required a reboot to restore service. No further outages are expected.

Ticket No.:	22786:45
Subject:	Outage Resolved - I2 IP Backbone HOUS-LOSA
Affected:	Backbone HOUS-LOSA
Start Time:	Saturday, April 12, 2014, 5:38 AM (0538) UTC
End Time:	Saturday, April 12, 2014, 6:19 AM (0619) UTC
Description:	The above listed is now available. Level(3) Engineers were replacing a card in Phoenix, AZ that was not expected to be service impacting. Work has been completed and no further outages are expected.

Scheduled Maintenances Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Maintenance Type	*Start Time (UTC)*	*End Time (UTC)*

22768	3-Elevated	1-Critical	Emergency Maintenance Completed - I2 IP Various Backbones (CHIC)	Hardware	04/10/2014 5:32 PM	04/10/2014 5:36 PM
22757	3-Elevated	2-High	Emergency Maintenance Completed - I2 IP Backbones ATLA-CHIC and ATLA-WASH	Hardware	04/11/2014 5:05 AM	04/11/2014 5:25 AM
22669	3-Elevated	2-High	Maintenance Completed - I2 IP Backbone HOUS-LOSA	Hardware	04/11/2014 7:10 AM	04/11/2014 7:26 AM
22773	3-Elevated	2-High	Emergency Maintenance Completed - I2 IP Backbone HOUS-LOSA	Hardware	04/12/2014 11:02 AM	04/12/2014 11:53 AM

Scheduled Maintenances Detail

Ticket No.:	22768:45
Subject:	Emergency Maintenance Completed - I2 IP Various Backbones (CHIC)
Affected:	AL2S Core Node (CHIC) and its Connector(s) and Participants Backbones ATLA-CHIC, CHIC-CLEV, CHIC-KANS
Start Time:	Thursday, April 10, 2014, 5:32 PM (1732) UTC
End Time:	Thursday, April 10, 2014, 5:36 PM (1736) UTC
Description:	The above Backbone circuits were briefly unavailable while Internet2 Engineers performed hardware maintenance on AL2S Core Node (CHIC). Traffic was re-routed.

Ticket No.:	22757:45
Subject:	Emergency Maintenance Completed - I2 IP Backbones ATLA-CHIC and ATLA-WASH
Affected:	Backbone ATLA-CHIC Backbone ATLA-WASH
Start Time:	Friday, April 11, 2014, 5:05 AM (0505) UTC
End Time:	Friday, April 11, 2014, 5:25 AM (0525) UTC
Description:	Backbones ATLA-CHIC and ATLA-WASH were unavailable while Level3 Engineers performed emergency maintenance. An outage of less than 30 minutes was observed. Maintenance has been completed. IP traffic was re-routed during this maintenance.

Ticket No.:	22669:45
Subject:	Maintenance Completed - I2 IP Backbone HOUS-LOSA
Affected:	Backbone HOUS-LOSA
Start Time:	Friday, April 11, 2014, 7:10 AM (0710) UTC
End Time:	Friday, April 11, 2014, 7:26 AM (0726) UTC
Description:	Backbone HOUS-LOSA was unavailable while Level3 Engineers performed maintenance to swap a card in an attempt to clear suspect pluggable alarms on opt.phoe. The maintenance was not successful in clearing these alarms. The maintenance has been completed. IP traffic was re-routed during this maintenance.

Ticket No.:	22773:45
Subject:	Emergency Maintenance Completed - I2 IP Backbone HOUS-LOSA
Affected:	Backbone HOUS-LOSA
Start Time:	Saturday, April 12, 2014, 11:02 AM (1102) UTC
End Time:	Saturday, April 12, 2014, 11:53 AM (1153) UTC
Description:	Backbone HOUS-LOSA was unavailable while Internet2 Engineers performed maintenance to install a new Layer 2 core node in Tucson, AZ. Maintenance has been completed . IP traffic re-routed during this maintenance.

Downtime and Availability

Internet2 IP Core Nodes	Down Time		Reporting Period Availability		52 Week Availability
ATLA MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99488%	100.00000%
CHIC MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98520%	99.99905%
CLEV MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99146%	99.99924%
HOUS MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.97685%	100.00000%
KANS MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99791%	100.00000%
LOSA MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99279%	100.00000%
NEWY MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
SALT MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99317%	100.00000%
SEAT MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99734%	100.00000%
WASH MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.94175%	99.99962%

Aggregate Internet2 IP Core Nodes	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98713%	99.99979%

** Less than 52 weeks of data available

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 IP Backbone Circuits	Down Time		Reporting Period Availability		52 Week Availability
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181	1 hr 18 min	0 hr 54 min	99.22619%	99.46429%	99.92695%	99.98843%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98615%	100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.78863%	99.84138%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180	0 hr 58 min	0 hr 54 min	99.42460%	99.46429%	99.79774%	99.85409%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183	0 hr 58 min	0 hr 54 min	99.42460%	99.46429%	99.95332%	99.98975%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98786%	99.99924%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.95522%	99.98179%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184	13 hr 38 min	13 hr 38 min	91.88492%	91.88492%	99.73873%	99.77364%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189	1 hr 48 min	0 hr 41 min	98.92857%	99.59325%	99.72089%	99.79660%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.93947%	99.97002%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.97780%	100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.96395%	99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.96395%	99.99905%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.95124%	99.95636%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98881%	99.99924%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98900%	99.99943%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.72374%	99.81538%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05638	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.57290%	99.66587%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.64917%	99.98122%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05562	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.64803%	99.98141%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05133	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.61787%	99.71141%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.61919%	99.71141%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.84821%	99.85030%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-6366	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.96680%	99.98786%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08892	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.96338%	99.98786%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.97970%	99.99981%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.97970%	99.99981%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08893	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.94858%	99.99829%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.95560%	99.99905%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.82886%	99.88464%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08890	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.80798%	99.88464%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.60288%	99.66625%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05561	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.60288%	99.66625%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.49757%	99.60686%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05581	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.49359%	99.60667%
Legacy 10G Backbone: I2-KANS-CHIC-10GE-05781	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05138	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.81557%	99.90930%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.81557%	99.90930%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.81861%	99.90912%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.78180%	99.80666%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05571	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.78503%	99.80666%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.80400%	99.87117%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.80077%	99.87117%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.73076%	99.91101%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07613	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.73076%	99.91101%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99127%	99.99943%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05573	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99127%	99.99943%

Aggregate All Internet2 IP Backbone Circuits	18 hr 40 min	1 hr 39 min	99.77778%	99.79742%	99.84004%	99.90115%

* Internet2 IP 100G Backbone Circuits Ride over AL2S for connectivity.

Upcoming Scheduled Maintenances

Ticket No.:	22741:45
Subject:	Maintenance 1 of 2 - I2 IP Backbone HOUS-KANS
Affected:	Backbone HOUS-KANS Connector University of Arizona (HOUS)* Connector MissiON (HOUS)* Connector LONI
Start Time:	Tuesday, April 15, 2014, 5:00 AM (0500) UTC
End Time:	Tuesday, April 15, 2014, 11:00 AM (1100) UTC
Description:	Backbone HOUS-KANS will be unavailable while Level(3) Communications relocate cable in Dallas, TX. An outage of 1 hour is expected; however the entire window is reserved. IP traffic will re-route during this maintenance. An alternate window has been reserved for 4/16 at 0500 UTC, should it be required. *Denotes the connector is multi-homed and therefore will remain reachable

Ticket No.:	22711:45
Subject:	Maintenance - I2 IP Backbone ATLA-HOUS
Affected:	Backbone ATLA-HOUS
Start Time:	Wednesday, April 16, 2014, 4:00 AM (0400) UTC
End Time:	Wednesday, April 16, 2014, 6:00 AM (0600) UTC
Description:	Backbone ATLA-HOUS will be unavailable while I2 Engineers perform maintenance. An outage is expected; however, the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:	22742:45
Subject:	Maintenance 2 of 2 - I2 IP Backbone HOUS-KANS
Affected:	Backbone HOUS-KANS Connector University of Arizona (HOUS)* Connector MissiON (HOUS)* Connector LONI
Start Time:	Wednesday, April 16, 2014, 5:00 AM (0500) UTC
End Time:	Wednesday, April 16, 2014, 11:00 AM (1100) UTC
Description:	Backbone HOUS-KANS will be unavailable while Level(3) Communications relocate cable in Dallas, TX. An outage of 1 hour is expected; however the entire window is reserved. IP traffic will re-route during this maintenance. This is Maintenance window 2 of 2. *Denotes the connector is multi-homed and therefore will remain reachable

Ticket No.:	22733:45
Subject:	Maintenance 1 of 2 - I2 IP Backbones ATLA-CHIC and ATLA-WASH
Affected:	Backbones ATLA-CHIC Backbone ATLA-WASH Connector Indiana Gigapop Connector KyRON Connector MissiON
Start Time:	Wednesday, April 16, 2014, 5:00 AM (0500) UTC
End Time:	Wednesday, April 16, 2014, 11:00 AM (1100) UTC
Description:	This is the first of two maintenance windows Backbones ATLA-CHIC and ATLA-WASH will be unavailable while fiber provider Engineers perform maintenance. Outages are expected, and the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:	22734:45
Subject:	Maintenance 2 of 2 - I2 IP Backbones ATLA-CHIC and ATLA-WASH
Affected:	Backbones ATLA-CHIC Backbone ATLA-WASH Connector Indiana Gigapop Connector KyRON Connector MissiON
Start Time:	Thursday, April 17, 2014, 5:00 AM (0500) UTC
End Time:	Thursday, April 17, 2014, 11:00 AM (1100) UTC
Description:	This is the second of two maintenance windows Backbones ATLA-CHIC and ATLA-WASH will be unavailable while fiber provider Engineers perform maintenance. Outages are expected, and the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:	22708:45
Subject:	Maintenance - I2 IP OOB (ATLA)
Affected:	oob.atla.net.internet2.edu
Start Time:	Thursday, April 17, 2014, 7:00 AM (0700) UTC
End Time:	Thursday, April 17, 2014, 10:00 AM (1000) UTC
Description:	OOB monitoring will become unavailable, while Level(3) Engineers perform maintenance in Atlanta, GA. No network impact is expected.

Ticket No.:	22792:45
Subject:	Maintenance 1 of 2 - I2 IP Backbone CINC-INDI
Affected:	I2-ATLA-CHIC-VLAN-09218 ATLA-CHI BB I2-ATLA-CHIC-VLAN-09219 ATLA-CHIC BB (ION) I2-ATLA-CHIC-VLAN-09320 ATLA-CHIC BB I2-CHIC-NASH-10GE-05845 I2-CHIC-NASH-10GE-06055 (SOX) I2-INDI-ATLA-10GE-04182 (InGP)
Start Time:	Thursday, April 24, 2014, 4:00 AM (0400) UTC
End Time:	Thursday, April 24, 2014, 10:00 AM (1000) UTC
Description:	The above listed will be unavailable while Level(3) Communications relocate cable due to mandated bridge installation project between Greensburg, IN and Indianapolis, IN. An outage of 30 minutes is expected; however the entire window is reserved. This is the first of two maintenance windows

Ticket No.:	22774:45
Subject:	Maintenance - I2 IP Backbone CHIC-KANS
Affected:	Backbone CHIC-KANS
Start Time:	Thursday, April 24, 2014, 7:00 AM (0700) UTC
End Time:	Thursday, April 24, 2014, 11:00 AM (1100) UTC
Description:	Backbone CHIC-KANS will be unavailable while Level(3) Engineers perform maintenance. An outage of 30min is expected; however, the entire window is reserved. IP traffic will re-route during this maintenance.