Generated Report for Internet2 IP

Internet2 IP Network Operations Report
05/05/2014 - 05/11/2014

Internet2 IP Network Operations Center
at Indiana University

New Connections

None

Upgrades

None

New Services

None

Unscheduled Outages Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Outage Type	Source Of Impact	*Start Time (UTC)*	*End Time (UTC)*

22997	2-High	1-Critical	Brief Outage Resolved - I2 IP Core Node ATLA	Hardware	Internal	05/05/2014 7:04 PM	05/05/2014 7:11 PM
23002	2-High	2-High	Outage Resolved - I2 IP Backbone CLEV-WASH	Hardware	Undetermined	05/06/2014 4:02 AM	05/06/2014 8:17 AM
23013	2-High	2-High	Brief Outage - I2 IP Backbone HOUS-KANS	Undetermined	Undetermined	05/06/2014 6:10 PM	05/06/2014 6:12 PM
23048	2-High	2-High	Brief Outage - I2 IP Backbone NEWY-WASH	Undetermined	Vendor	05/09/2014 2:02 PM	05/09/2014 2:03 PM
23064	2-High	2-High	Outage Update - I2 IP Backbone ATLA-WASH	Undetermined	Undetermined	05/11/2014 7:33 PM	(No Data Available)

Unscheduled Outages Detail

Ticket No.:	22997:45
Subject:	Brief Outage Resolved - I2 IP Core Node ATLA
Source Of Impact:	Internal
Affected:	Core Node ATLA and its Connector(s) and Participants.
Start Time:	Monday, May 5, 2014, 7:04 PM (1904) UTC
End Time:	Monday, May 5, 2014, 7:11 PM (1911) UTC
Description:	Core Node ATLA was briefly unavailable. Internet2 Engineers have advised that this is related to a hardware failure. Redundant systems recovered the service. Internet2 Engineering replaced the faulty component and it is resolved.. All connections to the Core Node were affected.

Ticket No.:	23002:45
Subject:	Outage Resolved - I2 IP Backbone CLEV-WASH
Source Of Impact:	Undetermined
Affected:	Backbone CLEV-WASH Connector MREN
Start Time:	Tuesday, May 6, 2014, 4:02 AM (0402) UTC
End Time:	Tuesday, May 6, 2014, 8:17 AM (0817) UTC
Description:	Backbone CLEV-WASH is now available. Level(3) Communications Technicians and CENIC Engineers have reported that the reason for the outage was due to a failed OCLD card. The Card has been replaced and services have been restored. No further outages are expected.

Ticket No.:	23013:45
Subject:	Brief Outage - I2 IP Backbone HOUS-KANS
Source Of Impact:	Undetermined
Affected:	Backbone HOUS-KANS
Start Time:	Tuesday, May 6, 2014, 6:10 PM (1810) UTC
End Time:	Tuesday, May 6, 2014, 6:12 PM (1812) UTC
Description:	Backbone HOUS-KANS was briefly unavailable. Internet2's Vendor Engineers have advised that the root cause of this issue is undetermined. IP traffic re-routed during the outage.

Ticket No.:	23048:45
Subject:	Brief Outage - I2 IP Backbone NEWY-WASH
Source Of Impact:	Vendor
Affected:	Backbone NEWY-WASH ION Backbone NEWY-WASH Connector MAGPI Connector MAX* Connector OARnet*
Start Time:	Friday, May 9, 2014, 2:02 PM (1402) UTC
End Time:	Friday, May 9, 2014, 2:03 PM (1403) UTC
Description:	The above listed are currently available to the community after a brief outage. Internet2 technicians are currently investigating the reason for the outage. * Denotes multihomed entities which should have remained available.

Ticket No.:	23064:45
Subject:	Outage Update - I2 IP Backbone ATLA-WASH
Source Of Impact:	Undetermined
Affected:	Backbone ATLA-WASH
Start Time:	Sunday, May 11, 2014, 7:33 PM (1933) UTC
End Time:	(No Data Available)
Description:	Backbone ATLA-WASH is unavailable. Internet2 Engineers are investigating the cause. The reason for the outage is a suspected hardware failure.Internet2 Engineers and Level(3) Engineers have isolated the issue to a failure on the Juniper gear in WASH. Additional work is being scheduled to resolve the issue. Updates will be provided as they become available.

Scheduled Maintenances Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Maintenance Type	Source Of Impact	*Start Time (UTC)*	*End Time (UTC)*

22925	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (LOSA)	Software	Internal	05/06/2014 11:01 AM	05/06/2014 11:06 AM
22927	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (SEAT)	Software	Internal	05/06/2014 11:30 AM	05/06/2014 11:35 AM
22926	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (SALT)	Software	Internal	05/07/2014 10:00 AM	05/07/2014 10:05 AM
22928	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (KANS)	Software	Internal	05/07/2014 10:30 AM	05/07/2014 10:35 AM
22951	3-Elevated	2-High	Maintenance 1 of 2 Completed - I2 IP Backbone CLEV-NEWY32AOA	Circuit	Vendor	05/08/2014 4:32 AM	05/08/2014 4:34 AM
22930	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (CHIC)	Software	Internal	05/08/2014 10:05 AM	05/08/2014 10:06 AM
22929	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (HOUS)	Software	Internal	05/08/2014 10:35 AM	05/08/2014 10:36 AM
22934	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (CLEV)	Software	Internal	05/09/2014 8:01 AM	05/09/2014 8:04 AM
22932	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (NEWY32AOA)	Software	Internal	05/09/2014 8:31 AM	05/09/2014 8:34 AM
22933	3-Elevated	1-Critical	Maintenance Completed - I2 IP Core Node (WASH)	Software	Internal	05/09/2014 9:02 AM	05/09/2014 9:06 AM

Scheduled Maintenances Detail

Ticket No.:	22925:45
Subject:	Maintenance Completed - I2 IP Core Node (LOSA)
Source Of Impact:	Internal
Affected:	Core Node (LOSA) and its Connector(s) and Participant(s)
Start Time:	Tuesday, May 6, 2014, 11:01 AM (1101) UTC
End Time:	Tuesday, May 6, 2014, 11:06 AM (1106) UTC
Description:	Core Node (LOSA) was briefly unavailable while Internet2 Engineers performed core upgrade maintenance. Maintenance has been completed.

Ticket No.:	22927:45
Subject:	Maintenance Completed - I2 IP Core Node (SEAT)
Source Of Impact:	Internal
Affected:	Core Node (SEAT) and its Connector(s) and Participants
Start Time:	Tuesday, May 6, 2014, 11:30 AM (1130) UTC
End Time:	Tuesday, May 6, 2014, 11:35 AM (1135) UTC
Description:	Core Node (SEAT) was briefly unavailable while Internet2 Engineers performed core upgrade maintenance. Maintenance has been completed.

Ticket No.:	22926:45
Subject:	Maintenance Completed - I2 IP Core Node (SALT)
Source Of Impact:	Internal
Affected:	Core Node (SALT) and its Connector(s) and Participant(s)
Start Time:	Wednesday, May 7, 2014, 10:00 AM (1000) UTC
End Time:	Wednesday, May 7, 2014, 10:05 AM (1005) UTC
Description:	Core Node (SALT) was briefly unavailable while Internet2 Engineers performed core upgrade maintenance. Maintenance has been completed.

Ticket No.:	22928:45
Subject:	Maintenance Completed - I2 IP Core Node (KANS)
Source Of Impact:	Internal
Affected:	Core Node (KANS) and its Connector(s) and Participant(s)
Start Time:	Wednesday, May 7, 2014, 10:30 AM (1030) UTC
End Time:	Wednesday, May 7, 2014, 10:35 AM (1035) UTC
Description:	Core Node (KANS) was briefly unavailable while Internet2 Engineers performed code upgrade maintenance. Maintenance has been completed.

Ticket No.:	22951:45
Subject:	Maintenance 1 of 2 Completed - I2 IP Backbone CLEV-NEWY32AOA
Source Of Impact:	Vendor
Affected:	Backbone CLEV-NEWY32AOA Connector NOX
Start Time:	Thursday, May 8, 2014, 4:32 AM (0432) UTC
End Time:	Thursday, May 8, 2014, 4:34 AM (0434) UTC
Description:	Backbone CLEV-NEWY32AOA was briefly unavailable while Level(3) Communications performed maintenance to repair a damaged cable in Buffalo, NY. The maintenance has been completed, and no further outages are expected. IP traffic was re-routed during this maintenance. *This is the 1st of 2 maintenance windows

Ticket No.:	22930:45
Subject:	Maintenance Completed - I2 IP Core Node (CHIC)
Source Of Impact:	Internal
Affected:	Core Node (CHIC) and its Connector(s) and Participants
Start Time:	Thursday, May 8, 2014, 10:05 AM (1005) UTC
End Time:	Thursday, May 8, 2014, 10:06 AM (1006) UTC
Description:	Core Node (CHIC) was briefly unavailable while Internet2 Engineers performed code upgrade maintenance to swaps Routing Engine mastership. Maintenance has been successfully completed.

Ticket No.:	22929:45
Subject:	Maintenance Completed - I2 IP Core Node (HOUS)
Source Of Impact:	Internal
Affected:	Core Node (HOUS) and its Connector(s) and Participants
Start Time:	Thursday, May 8, 2014, 10:35 AM (1035) UTC
End Time:	Thursday, May 8, 2014, 10:36 AM (1036) UTC
Description:	Core Node (HOUS) was briefly unavailable while Internet2 Engineers performed code upgrade maintenance. The upgrade was successful. Maintenance has been completed.

Ticket No.:	22934:45
Subject:	Maintenance Completed - I2 IP Core Node (CLEV)
Source Of Impact:	Internal
Affected:	Core Node (CLEV) and its Connector(s) and Participants
Start Time:	Friday, May 9, 2014, 8:01 AM (0801) UTC
End Time:	Friday, May 9, 2014, 8:04 AM (0804) UTC
Description:	Core Node (CLEV) was briefly unavailable while Internet2 Engineers performed code upgrade maintenance. Maintenance has been successfully completed.

Ticket No.:	22932:45
Subject:	Maintenance Completed - I2 IP Core Node (NEWY32AOA)
Source Of Impact:	Internal
Affected:	Core Node (NEWY32AOA) and its Connector(s) and Participants
Start Time:	Friday, May 9, 2014, 8:31 AM (0831) UTC
End Time:	Friday, May 9, 2014, 8:34 AM (0834) UTC
Description:	Core Node (NEWY32AOA) was briefly unavailable while Internet2 Engineers performed code upgrade maintenance. Maintenance has been completed successfully.

Ticket No.:	22933:45
Subject:	Maintenance Completed - I2 IP Core Node (WASH)
Source Of Impact:	Internal
Affected:	Core Node (WASH) and its Connector(s) and Participant(s)
Start Time:	Friday, May 9, 2014, 9:02 AM (0902) UTC
End Time:	Friday, May 9, 2014, 9:06 AM (0906) UTC
Description:	Core Node (WASH) was briefly unavailable while Internet2 Engineers performed code upgrade maintenance. Maintenance has been completed. No further outages are anticipated.

Downtime and Availability

Internet2 IP Core Nodes	Down Time		Reporting Period Availability		52 Week Availability
ATLA MX960	0 hr 7 min	0 hr 7 min	99.93056%	99.93056%	99.99355%	99.99867%
CHIC MX960	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.98501%	99.99905%
CLEV MX960	0 hr 3 min	0 hr 0 min	99.97024%	100.00000%	99.99089%	99.99924%
HOUS MX960	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.97571%	100.00000%
KANS MX960	0 hr 5 min	0 hr 0 min	99.95040%	100.00000%	99.99639%	100.00000%
LOSA MX960	0 hr 5 min	0 hr 0 min	99.95040%	100.00000%	99.99184%	100.00000%
NEWY MX960	0 hr 3 min	0 hr 0 min	99.97024%	100.00000%	99.99943%	100.00000%
SALT MX960	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.99241%	100.00000%
SEAT MX960	0 hr 5 min	0 hr 0 min	99.95040%	100.00000%	99.99639%	100.00000%
WASH MX960	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.98843%	100.00000%

Aggregate Internet2 IP Core Nodes	0 hr 38 min	0 hr 38 min	99.96230%	99.99306%	99.99101%	99.99970%

** Less than 52 weeks of data available

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 IP Backbone Circuits	Down Time		Reporting Period Availability		52 Week Availability
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.87363%	99.95693%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.74177%	99.89545%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.78787%	99.84138%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.79698%	99.85409%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183	0 hr 6 min	0 hr 0 min	99.94048%	100.00000%	99.95219%	99.98975%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177	0 hr 8 min	0 hr 0 min	99.92063%	100.00000%	99.98634%	99.99924%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178	0 hr 7 min	0 hr 0 min	99.93056%	100.00000%	99.95389%	99.98179%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184	0 hr 8 min	0 hr 2 min	99.92063%	99.98016%	99.73057%	99.77326%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189	0 hr 6 min	0 hr 0 min	99.94048%	100.00000%	99.71881%	99.79660%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.93777%	99.97002%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.97609%	100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187	0 hr 10 min	0 hr 0 min	99.90079%	100.00000%	99.96205%	99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328	0 hr 10 min	0 hr 0 min	99.90079%	100.00000%	99.96205%	99.99905%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176	0 hr 8 min	0 hr 1 min	99.92063%	99.99008%	99.94972%	99.95617%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.98710%	99.99924%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.98729%	99.99943%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.84100%	99.91310%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05638	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.69016%	99.76359%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.54349%	99.87667%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05562	0 hr 1 min	0 hr 0 min	99.99008%	100.00000%	99.54235%	99.87686%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05133	4 hr 30 min	4 hr 26 min	97.32143%	97.36111%	99.61464%	99.66189%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.66644%	99.71236%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.84745%	99.85030%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-6366	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.96604%	99.98786%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08892	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.96262%	99.98786%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781	0 hr 6 min	0 hr 0 min	99.94048%	100.00000%	99.97856%	99.99981%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976	0 hr 6 min	0 hr 0 min	99.94048%	100.00000%	99.97856%	99.99981%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08893	0 hr 8 min	0 hr 0 min	99.92063%	100.00000%	99.95807%	99.99829%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894	0 hr 8 min	0 hr 0 min	99.92063%	100.00000%	99.96509%	99.99905%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889	4 hr 22 min	4 hr 15 min	97.40079%	97.47024%	99.82620%	99.83626%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08890	4 hr 22 min	4 hr 15 min	97.40079%	97.47024%	99.80533%	99.83626%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560	0 hr 8 min	0 hr 2 min	99.92063%	99.98016%	99.68067%	99.75030%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05561	0 hr 8 min	0 hr 2 min	99.92063%	99.98016%	99.68067%	99.75030%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559	0 hr 6 min	0 hr 0 min	99.94048%	100.00000%	99.52357%	99.60686%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05581	0 hr 6 min	0 hr 0 min	99.94048%	100.00000%	99.51958%	99.60667%
Legacy 10G Backbone: I2-KANS-CHIC-10GE-05781	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05138	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.81709%	99.90930%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.81709%	99.90930%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.81690%	99.90912%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.78009%	99.80666%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05571	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.78332%	99.80666%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564	0 hr 10 min	0 hr 0 min	99.90079%	100.00000%	99.93416%	99.97249%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572	0 hr 10 min	0 hr 0 min	99.90079%	100.00000%	99.93094%	99.97249%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612	0 hr 8 min	0 hr 1 min	99.92063%	99.99008%	99.78825%	99.91177%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07613	0 hr 8 min	0 hr 1 min	99.92063%	99.99008%	99.78825%	99.91177%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.98956%	99.99943%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05573	0 hr 9 min	0 hr 0 min	99.91071%	100.00000%	99.98956%	99.99943%

Aggregate All Internet2 IP Backbone Circuits	17 hr 58 min	4 hr 53 min	99.78611%	99.84425%	99.84660%	99.90268%

* Internet2 IP 100G Backbone Circuits Ride over AL2S for connectivity.

Upcoming Scheduled Maintenances

Ticket No.:	23014:45
Subject:	Maintenance 1 of 2 - I2 IP Backbone LOSA-SALT
Affected:	Backbone LOSA-SALT
Start Time:	Wednesday, May 14, 2014, 7:00 AM (0700) UTC
End Time:	Wednesday, May 14, 2014, 1:00 PM (1300) UTC
Description:	Backbone LOSA-SALT will be unavailable while Level(3) Engineers perform maintenance. An outage is expected. The entire window is reserved. IP traffic will re-route during this maintenance. This is maintenance window 1 of 2.

Ticket No.:	23015:45
Subject:	Maintenance 2 of 2 - I2 IP Backbone LOSA-SALT
Affected:	Backbone LOSA-SALT
Start Time:	Friday, May 16, 2014, 7:00 AM (0700) UTC
End Time:	Friday, May 16, 2014, 1:00 PM (1300) UTC
Description:	Backbone LOSA-SALT will be unavailable while Level(3) Engineers perform maintenance. An outage is expected. The entire window is reserved. IP traffic will re-route during this maintenance. This is maintenance window 2 of 2.