Unified - Internet2 IP Network Operations Report
08/24/2015 - 08/30/2015

Unified - Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Source Of Impact Start Time (UTC) End Time (UTC)

52784-Normal Brief Outage - I2 AL2S Backbone CINC-LOUI InvestigatingUndetermined 08/25/2015 4:43 AM 08/25/2015 4:44 AM
53073-Elevated 1-Critical Brief Outage - I2 Optical DWDM System BOST-HART2 InvestigatingVendor 08/26/2015 4:47 AM 08/26/2015 4:48 AM
53132-High Outage Resolved - I2 AL2S Backbone PHOE-TUCS HardwareVendor 08/26/2015 9:39 AM 08/26/2015 2:05 PM
53372-High Outage Resolved - I2 IP Backbone ATLA-ATLA HardwareVendor 08/27/2015 3:04 PM 08/27/2015 3:18 PM
08/27/2015 3:24 PM 08/27/2015 4:32 PM
53544-Normal 1-Critical Outages Resolved - I2 Optical DWDM System ASHB-WASH Unannounced MaintenanceVendor 08/28/2015 4:42 AM 08/28/2015 7:02 AM
08/28/2015 8:43 AM 08/28/2015 8:44 AM

Unscheduled Outages Detail


Ticket No.:5278:144
Subject:Brief Outage - I2 AL2S Backbone CINC-LOUI
Source Of Impact:Undetermined
Affected:AL2S Backbone CINC-LOUI
IP Backbone ATLA-CHIC
Start Time:Tuesday, August 25, 2015, 4:43 AM (0443) UTC
End Time:Tuesday, August 25, 2015, 4:44 AM (0444) UTC
Description:The above listed asset is available after a brief outage. Level 3 Engineers are investigating the cause. The reason for the outage is pending. Workgroup administrators receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:5307:144
Subject:Brief Outage - I2 Optical DWDM System BOST-HART2
Source Of Impact:Vendor
Affected:DWDM System BOST-HART
AL2S Backbone BOST-HART2
IP Backbone CHIC-NEWY32AOA
IP Backbone CLEV-NEWY32AOA
AL2S Participant NOX
IP Connector NOX (NEWY32AoA) and its participants
OARnet (WASH)
Start Time:Wednesday, August 26, 2015, 4:47 AM (0447) UTC
End Time:Wednesday, August 26, 2015, 4:48 AM (0448) UTC
Description:The above listed assets are now available to the community following a brief outage. Level (3) Communications Engineers are currently investigating the reason for the outage. Updates will be provided as they are received.

Ticket No.:5313:144
Subject:Outage Resolved - I2 AL2S Backbone PHOE-TUCS
Source Of Impact:Vendor
Affected:AL2S Backbone PHOE-TUCS
IP Backbone HOUS-LOSA
Start Time:Wednesday, August 26, 2015, 9:39 AM (0939) UTC
End Time:Wednesday, August 26, 2015, 2:05 PM (1405) UTC
Description:The above listed assets are now available. Internet2 engineers have investigate the cause of the outage and determined that it was caused by failing hardware. Maintenance has been completed to replace the failing hardware. No further outages are expected.

Ticket No.:5337:144
Subject:Outage Resolved - I2 IP Backbone ATLA-ATLA
Source Of Impact:Vendor
Affected:IP Backbones ATLA-ATLA, ATLA-HOUS, ATLA-CLEV, ATLA-NEWY32AOA
Peer RedCLARA
Peer Peer MCNC
Peer Vanderbuilt
Peer FLR
Peer Indiana Gigapop
Start Time:Thursday, August 27, 2015, 3:04 PM (1504) UTC
End Time:Thursday, August 27, 2015, 3:18 PM (1518) UTC
Start Time:Thursday, August 27, 2015, 3:24 PM (1524) UTC
End Time:Thursday, August 27, 2015, 4:32 PM (1632) UTC
Description:The above listed assets are now available. Internet2 engineers restarted a line card to resolve the outage. IP Backbone traffic re-routed to the available paths.

Ticket No.:5354:144
Subject:Outages Resolved - I2 Optical DWDM System ASHB-WASH
Source Of Impact:Vendor
Affected:DWDM System ASHB-WASH
AL2S Backbone(s) ASHB-WASH and CLEV-WASH
IP Backbone(s) ATLA-CLEV, CHIC-WASH and CLEV-WASH
IP Connector 3ROX and its Participants
Start Time:Friday, August 28, 2015, 4:42 AM (0442) UTC
End Time:Friday, August 28, 2015, 7:02 AM (0702) UTC
Start Time:Friday, August 28, 2015, 8:43 AM (0843) UTC
End Time:Friday, August 28, 2015, 8:44 AM (0844) UTC
Description:The above listed assets are currently available to the community. Level (3) Communications Engineers have confirmed that the outage is due to an unannounced maintenance to decommission a site in Reston, VA. The maintenance has been completed and no further outages are expected.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Start Time (UTC) End Time (UTC)

5289 3-Elevated Emergency Maintenance Completed - I2 IP Core Node WASH Hardware Internal08/26/2015 4:01 AM 08/26/2015 4:06 AM
5192 3-Elevated 1-Critical Maintenance Completed - I2 Optical DWDM System CHIC-STAR CircuitVendor 08/26/2015 5:07 AM 08/26/2015 7:09 AM
5321 3-Elevated 1-Critical Emergency Maintenance Completed - I2 Optical DWDM System BOST-HART2 Circuit Vendor08/27/2015 8:11 AM 08/27/2015 8:57 AM

Scheduled Maintenances Detail

Ticket No.:5289:144
Subject:Emergency Maintenance Completed - I2 IP Core Node WASH
Source Of Impact:Internal
Affected:IP Core Node WASH and its Connector(s) and Participants
IP Backbone ATLA-WASH
IP Backbone CHIC-WASH
IP Backbone CLEV-WASH
IP Backbone LOSA-WASH
IP Backbone NEWY32AOA-WASH
AL2S-IP interconnect
Start Time:Wednesday, August 26, 2015, 4:01 AM (0401) UTC
End Time:Wednesday, August 26, 2015, 4:06 AM (0406) UTC
Description: The above listed assets are now available after Internet2 Engineers performed core node maintenance. All connections to the Core Node were affected. The maintenance has been completed. No further outage is expected.

Ticket No.:5192:144
Subject:Maintenance Completed - I2 Optical DWDM System CHIC-STAR
Source Of Impact:Vendor
Affected:DWDM System CHIC-STAR
AL2S Backbone CHIC-STAR
AL2S Backbone EQCH-STAR
IP Backbone CHIC-NEWY32AOA
IP Backbone CHIC-CLEV
AL2S Core Node STAR and Connectors
Net+ Provider Code42 (CHIC)
Start Time:Wednesday, August 26, 2015, 5:07 AM (0507) UTC
End Time:Wednesday, August 26, 2015, 7:09 AM (0709) UTC
Description: The above listed assets were unavailable while Internet2 Fiber Vendor Engineers replaced the Optical Core Node opt.star which had sustained physical damage.This maintenance has been completed.

Ticket No.:5321:144
Subject:Emergency Maintenance Completed - I2 Optical DWDM System BOST-HART2
Source Of Impact:Vendor
Affected:DWDM System BOST-HART2
AL2S Backbone(s) BOST-HART2
IP Backbone CHIC-NEWY32AOA
IP BAckbone CLEV-NEWY32AOA
Connector(s)/Peer(s)
Start Time:Thursday, August 27, 2015, 8:11 AM (0811) UTC
End Time:Thursday, August 27, 2015, 8:57 AM (0857) UTC
Description: The above listed assets are now available after Level(3) Communications performed emergency maintenance on DWDM System BOST- HART2. Maintenance has been completed; no further outages are expected.


Downtime and Availability



Unified - Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability
ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%

Aggregate Unified - Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%100.00000%100.00000%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Unified - Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
I2-ATLA-CLEV-VLAN-14832 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-CLEV-VLAN-14837 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14833 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14838 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14831 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14834 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 0 hr 1 min0 hr 1 min 99.99008%99.99008% 99.44938%99.79831%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 1 hr 22 min1 hr 22 min 99.18651%99.18651% 99.51977%99.96376%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.72488%99.84271%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 2 hr 2 min0 hr 0 min 98.78968%100.00000% 99.86244%99.99905%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.90323%99.99564%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 0 hr 47 min0 hr 1 min 99.53373%99.99008% 99.92790%99.99469%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 2 hr 26 min2 hr 21 min 98.55159%98.60119% 99.84802%99.88654%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.61578%99.81235%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 4 hr 26 min4 hr 26 min 97.36111%97.36111% 99.28241%99.41693%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84973%99.97002%
R&E 10G Backbone: I2-LOSA-PAIX-VLAN-23105 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99184%100.00000%
Commodity 10G Backbone: I2-LOSA-PAIX-VLAN-23106 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99184%100.00000%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95959%99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80077%99.88521%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80077%99.88521%
R&E 100G Backbone: I2-LOSA-WILC-VLAN-23103 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98918%99.99602%
Commodity 100G Backbone: I2-LOSA-WILC-VLAN-23104 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98918%99.99602%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.89659%99.95959%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96471%99.99962%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96471%99.99962%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.72488%99.84271%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.65771%99.90912%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.36779%99.72355%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93871%100.00000%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.93245%99.99829%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95105%99.99981%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97647%99.99962%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96395%99.99981%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 2 hr 26 min2 hr 21 min 98.55159%98.60119% 99.93378%99.93815%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.77212%99.87306%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.74822%99.82734%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93245%99.99905%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93720%99.99924%
Legacy 10G Backbone: I2-LOSA-PAIX-10GE-05853 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98805%100.00000%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98103%100.00000%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.83493%99.85732%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981%99.99981%
Legacy 100G Backbone: I2-LOSA-WILC-100GE-23893 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.99317%99.99810%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99658%99.99981%

Aggregate All Unified - Internet2 IP Backbone Circuits 13 hr 55 min3 hr 23 min 99.83094%99.87204% 99.87680%99.94623%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.



Upcoming Scheduled Maintenances

Ticket No.:5302:144
Subject:Maintenance - I2 Optical Member ESNet
Affected:
Start Time:Monday, August 31, 2015, 12:00 PM (1200) UTC
End Time:Friday, September 4, 2015, 10:00 PM (2200) UTC
Description: Member ESNet's above listed optical circuits will be unavailable while CenturyLink performs maintenance to relocate fiber. Multiple 10 minute outages are expected during the window. The entire maintenance window has been reserved.

Ticket No.:5374:144
Subject:Maintenance - I2 Optical Member CENIC
Affected:Optical Member CENIC
Start Time:Thursday, September 3, 2015, 7:00 AM (0700) UTC
End Time:Thursday, September 3, 2015, 1:00 PM (1300) UTC
Description: The above listed assset will be unavailable while CENIC Engineers perform maintenance to upgrade software. An outage is expected, however the entire window is reserved.

Ticket No.:5366:144
Subject:Maintenance 2 of 2 - I2 Optical Circuits I2-COLU4-KANS-100GE-11579 and I2-COLU4-SAIN-100GE-11580
Affected:AL2S Backbone COLU4-KANS and COLU4-SAIN
Start Time:Friday, September 4, 2015, 5:01 AM (0501) UTC
End Time:Friday, September 4, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while MOREnet Engineers perform maintenance. An outage is expected and the entire window is reserved.

Ticket No.:5361:144
Subject:Maintenance - I2 AL2S Backbone CHIC-COLU4
Affected:AL2S Backbone CHIC-COLU4
IP Backbone CHIC-COLU4
Start Time:Saturday, September 5, 2015, 7:00 AM (0700) UTC
End Time:Saturday, September 5, 2015, 9:00 AM (0900) UTC
Description: The above listed asset will be unavailable while Level(3) Engineers perform maintenance. An outage is expected. The entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:5387:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:
Start Time:Monday, September 7, 2015, 1:00 PM (1300) UTC
End Time:Monday, September 7, 2015, 2:00 PM (1400) UTC
Description: Member ESnet's Optical circuit ESNET-DENV-LASV-OTU3-11574 will be unavailable while ESnet Engineers perform maintenance. A brief outage is expected, however the entire maintenance window has been reserved.

Ticket No.:5400:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:
Start Time:Tuesday, September 8, 2015, 1:00 PM (1300) UTC
End Time:Tuesday, September 8, 2015, 2:00 PM (1400) UTC
Description: Member ESnet's above optical circuits will be unavailable while ESnet Engineers perform maintenance. A brief outage is expected, however the entire maintenance window has been reserved.

Ticket No.:5362:144
Subject:Maintenance - I2 Optical Various 100G Circuits (DENV/PORT)
Affected:
Start Time:Thursday, September 10, 2015, 4:00 AM (0400) UTC
End Time:Thursday, September 10, 2015, 10:00 AM (1000) UTC
Description: The above listed circuits will be unavailable while Level(s) Engineers perform software maintenance. An outage is expected and the entire maintenance window has been reserved.

Ticket No.:5342:144
Subject:Maintenance - I2 IP SIP Provider MiTel (KANS)
Affected:Internet2 SIP Provider MiTel (KANS)
Start Time:Thursday, September 10, 2015, 5:00 AM (0500) UTC
End Time:Thursday, September 10, 2015, 8:00 AM (0800) UTC
Description: The above listed asset will be unavailable while Level(3) Engineers perform maintenance. An outage of 30 minutes is expected, however the entire maintenance window has been reserved. The Internet2 SIP service provided by MiTel will remain available via other assets not impacted by this event.

Ticket No.:5378:144
Subject:Maintenance 1 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
AL2S Backbones ATLA-JCSN, RALE-SALT
Start Time:Tuesday, September 22, 2015, 5:00 AM (0500) UTC
End Time:Tuesday, September 22, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo performs maintenance to replace a splice case. An outage of up to 30 minutes is expected. The entire window is reserved.

Ticket No.:5379:144
Subject:Maintenance 2 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
AL2S Backbones ATLA-JCSN, RALE-SALT
Start Time:Wednesday, September 23, 2015, 5:00 AM (0500) UTC
End Time:Wednesday, September 23, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo performs maintenance to replace a splice case. An outage of up to 30 minutes is expected. The entire window is reserved.

Ticket No.:5380:144
Subject:Maintenance 3 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
AL2S Backbones ATLA-JCSN, RALE-SALT
Start Time:Thursday, September 24, 2015, 5:00 AM (0500) UTC
End Time:Thursday, September 24, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo performs maintenance to replace a splice case. An outage of up to 30 minutes is expected. The entire window is reserved.

Ticket No.:5381:144
Subject:Maintenance 4 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
AL2S Backbones ATLA-JCSN, RALE-SALT
Start Time:Friday, September 25, 2015, 5:00 AM (0500) UTC
End Time:Friday, September 25, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo performs maintenance to replace a splice case. An outage of up to 30 minutes is expected. The entire window is reserved.