Unified - Internet2 IP Network Operations Report
10/24/2016 - 10/30/2016

Unified - Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Source Of Impact Start Time (UTC) End Time (UTC)

107534-Normal 2-High Outage Resolved - I2 IP Backbone ASHB-ATLA SoftwareVendor 10/21/2016 5:24 AM 10/25/2016 12:14 AM
108814-Normal 1-Critical Outage Resolved - I2 Optical Circuit I2-HOUH-HOUS-100GE-09363 SoftwareVendor 10/28/2016 5:27 PM 10/28/2016 5:52 PM
108834-Normal 1-Critical Brief Outage Resolved - I2 Optical DWDM System BATO-JACK PowerVendor 10/28/2016 6:27 PM 10/28/2016 6:30 PM
108924-Normal 1-Critical Brief Outage Resolved - I2 Various Optical Circuits Human ErrorVendor 10/29/2016 5:03 PM 10/29/2016 5:04 PM

Unscheduled Outages Detail


Ticket No.:10753:144
Subject:Outage Resolved - I2 IP Backbone ASHB-ATLA
Source Of Impact:Vendor
Affected:IP Backbone ASHB-ATLA
Start Time:Friday, October 21, 2016, 5:24 AM (0524) UTC
End Time:Tuesday, October 25, 2016, 12:14 AM (0014) UTC
Description:IP Backbone ASHB-ATLA is currently available. Internet2 engineers working with Juniper TAC discovered a software limitation which caused the outage. A solution was devised and implemented and service has restored. No further outages are expected.

Ticket No.:10881:144
Subject:Outage Resolved - I2 Optical Circuit I2-HOUH-HOUS-100GE-09363
Source Of Impact:Vendor
Affected:Optical HOUH-HOUS AL2S/AL3S Interconnect
IP Backbone ATLA-HOUS
IP Backbone HOUS-LOSA
Start Time:Friday, October 28, 2016, 5:27 PM (1727) UTC
End Time:Friday, October 28, 2016, 5:52 PM (1752) UTC
Description:The above listed asset is now available. The reason for the outage is a hardware failure. Workgroup administrators receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:10883:144
Subject:Brief Outage Resolved - I2 Optical DWDM System BATO-JACK
Source Of Impact:Vendor
Affected:DWDM System BATO-JACK
AL2S Backbone BATO-JACK
IP Backbone ATLA-HOUS
Member NOAA N-Wave
Start Time:Friday, October 28, 2016, 6:27 PM (1827) UTC
End Time:Friday, October 28, 2016, 6:30 PM (1830) UTC
Description:The above listed assets are now available after a brief outage. Level (3) Communications Engineers have investigated and have determined that the outage was due to a power outage of undetermined origin. No further outages are anticipated.

Ticket No.:10892:144
Subject:Brief Outage Resolved - I2 Various Optical Circuits
Source Of Impact:Vendor
Affected:IP Backbone ATLA-CHIC
Member SoX
Member ESnet
Member Indiana Gigapop
Member University of Memphis
Member NOAA N-Wave
Start Time:Saturday, October 29, 2016, 5:03 PM (1703) UTC
End Time:Saturday, October 29, 2016, 5:04 PM (1704) UTC
Description:The above listed circuits are available following a brief outage. Internet2 Fiber Vendor Engineers have investigated the cause. The reason for the outage was a fiber bump between Rexville, IN and Cincinnati, OH. IP backbone traffic re-routed.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Start Time (UTC) End Time (UTC)

10782 3-Elevated 2-High Emergency Maintenance Completed- I2 AL2S Backbone ATLA-WASH Hardware Vendor10/24/2016 4:09 AM 10/24/2016 4:20 AM
10733 3-Elevated 1-Critical Maintenance Completed - I2 IP Core Node HOUS HardwareInternal 10/27/2016 5:02 AM 10/27/2016 6:00 AM

Scheduled Maintenances Detail

Ticket No.:10782:144
Subject:Emergency Maintenance Completed- I2 AL2S Backbone ATLA-WASH
Source Of Impact:Vendor
Affected:AL2S Backbone ATLA-WASH
IP Backbone ATLA-WASH
Start Time:Monday, October 24, 2016, 4:09 AM (0409) UTC
End Time:Monday, October 24, 2016, 4:20 AM (0420) UTC
Description: The above listed asset is now available after Internet2 Engineers performed an emergency maintenance to replace an FPC in McLean, VA. Maintenance is completed. No further outages are expected.

Ticket No.:10733:144
Subject:Maintenance Completed - I2 IP Core Node HOUS
Source Of Impact:Internal
Affected:IP Core Node HOUS Connectors
IP Backbone ATLA-HOUS, HOUS-KANS & HOUS-LOSA
Net+ Provider Aastra Clearspan
IP Connectors:
OneNet
Arkansas Research and Education Optical Network
LoneStar Education and Research Network
Center for Education Network Initiatives in California
Sun Corridor
Corporacion Universitaria para el Desarrollo de Internet
MissiON
Louisiana Optical Network Initiative
Global Environment for Network Innovations
Tulane University
Start Time:Thursday, October 27, 2016, 5:02 AM (0502) UTC
End Time:Thursday, October 27, 2016, 6:00 AM (0600) UTC
Description: The above listed assets are now available after Internet2 Engineers performed a maintenance to consolidate hardware in Houston, TX. Maintenance has been completed. No further outages are expected.


Downtime and Availability



Unified - Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability
ASHB MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%

Aggregate Unified - Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%100.00000%100.00000%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Unified - Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
Layer3 AL3S Backbone ASHB-ATLA R&E VLAN 55199 24 hr 14 min24 hr 14 min 85.57540%85.57540% 98.46219%98.74167%
Layer3 AL3S Backbone ASHB-ATLA Commodity VLAN 55204 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.42268%99.61701%
Layer3 AL3S Backbone ASHB-CHIC R&E VLAN 55198 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Layer3 AL3S Backbone ASHB-CHIC Commodity VLAN 55203 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.68987%99.81551%
Layer3 AL3S Backbone ASHB-CLEV R&E VLAN 55197 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.81381%99.98392%
Layer3 AL3S Backbone ASHB-CLEV Commodity VLAN 55202 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98808%100.00000%
Layer3 AL3S Backbone ASHB-NEWY R&E VLAN 55201 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.01132%99.29534%
Layer3 AL3S Backbone ASHB-NEWY Commodity VLAN 55206 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.60055%99.71806%
Layer3 AL3S Backbone ASHB-WASH R&E VLAN 55200 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.85714%99.92658%
Layer3 AL3S Backbone ASHB-WASH Commodity VLAN 55205 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.83954%99.92072%
I2-CHIC-WASH-VLAN-12131 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.37065%99.61361%
I2-CHIC-WASH-VLAN-12132 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.35191%99.62913%
I2-ATLA-CLEV-VLAN-14832 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-CLEV-VLAN-14837 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14833 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14838 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14831 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14834 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.60566%99.87682%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 1 hr 26 min0 hr 28 min 99.14683%99.72222% 98.64309%98.90478%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.62345%99.81040%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.58031%99.87360%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.79091%100.00000%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.57236%99.66508%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.48078%99.88590%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 0 hr 58 min0 hr 0 min 99.42460%100.00000% 98.98085%99.32542%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 1 hr 23 min0 hr 25 min 99.17659%99.75198% 99.49667%99.81570%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.47907%99.79564%
R&E 10G Backbone: I2-LOSA-PAIX-VLAN-23105 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93472%99.99622%
Commodity 10G Backbone: I2-LOSA-PAIX-VLAN-23106 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93472%99.99622%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.63707%99.90236%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.71087%99.95515%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.71087%99.95515%
R&E 100G Backbone: I2-LOSA-WILC-VLAN-23103 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97408%100.00000%
Commodity 100G Backbone: I2-LOSA-WILC-VLAN-23104 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97408%100.00000%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.43820%99.57368%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.79167%99.99981%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.79167%99.99981%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.62345%99.81040%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 0 hr 1 min0 hr 1 min 99.99008%99.99008% 99.56347%99.87587%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 1 hr 1 min0 hr 3 min 99.39484%99.97024% 98.64328%98.93222%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.72733%99.84295%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80283%99.88949%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80945%100.00000%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.86187%99.98259%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 58 min0 hr 0 min 99.42460%100.00000% 99.12352%99.32978%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 58 min0 hr 0 min 99.42460%100.00000% 99.60415%99.84749%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.65921%99.86773%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84370%99.87190%
Legacy 10G Backbone: I2-LOSA-PAIX-10GE-05853 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93472%99.99622%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92185%100.00000%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.67624%99.87038%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 100G Backbone: I2-LOSA-WILC-100GE-23893 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.76082%99.87852%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89309%99.99981%

Aggregate All Unified - Internet2 IP Backbone Circuits 31 hr 21 min6 hr 10 min 99.69409%99.75426% 99.68865%99.82932%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.



Upcoming Scheduled Maintenances

Ticket No.:11061:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:Member ESnet
Start Time:Wednesday, November 16, 2016, 2:00 PM (1400) UTC
End Time:Wednesday, November 16, 2016, 3:00 PM (1500) UTC
Description: Member ESnet's Optical circuit ESNET-SNLL-SACR-10GE-07949 and ESNET- SNLL-SUNN-10GE-07950 will be unavailable while ESnet Engineers perform maintenance. A brief outage is expected, however the entire maintenance window has been reserved.

Ticket No.:11063:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:Member ESnet
Start Time:Wednesday, November 16, 2016, 3:00 PM (1500) UTC
End Time:Wednesday, November 16, 2016, 4:00 PM (1600) UTC
Description: Member ESnet's Optical circuit ESNET-SNLL-SACR-10GE-07947 and ESNET-SNLL-SUNN-10GE-07948 will be unavailable while ESnet Engineers perform maintenance. A brief outage is expected, however the entire maintenance window has been reserved.

Ticket No.:11013:144
Subject:Maintenance 1 of 2 - I2 Optical DWDM System CHAR-RALE
Affected:DWDM System CHAR-RALE
AL2S Backbone(s) ATLA-WASH & CHAR-RALE
IP Backbone(s) ATLA-WASH, ATLA-CLEV, ATLA-NEWY32AOA, & ASHB-ATLA
Member ESnet
Member NOAA N-Wave
Start Time:Saturday, November 19, 2016, 5:00 AM (0500) UTC
End Time:Saturday, November 19, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Level(3) Communications relocate fiber in Greensboro, NC. An outage is expected. The entire window is reserved. This is maintenance window 1 of 2.

Ticket No.:10914:144
Subject:Maintenance - I2 AL2S Circuit I2-LOSA-SING-100GE-55449
Affected:Internet2 Singapore
Start Time:Saturday, November 19, 2016, 1:00 PM (1300) UTC
End Time:Saturday, November 19, 2016, 8:00 PM (2000) UTC
Description: The above listed asset will be unavailable while Level(3) Engineers perform maintenance. An outage of 7 hours is expected. The entire maintenance window is reserved.

Ticket No.:10943:144
Subject:Maintenance - I2 AL2S Circuit I2-LOSA-SING-100GE-55449
Affected:Internet2 Singapore
Start Time:Saturday, November 19, 2016, 1:00 PM (1300) UTC
End Time:Saturday, November 19, 2016, 8:00 PM (2000) UTC
Description: The above listed asset will be unavailable while Level(3) Engineers perform maintenance. An outage is expected. The entire maintenance window is reserved.

Ticket No.:10862:144
Subject:Maintenance - I2 Optical Member Center for Education Network Initiatives in California
Affected:Member Center for Education Network Initiatives
Start Time:Sunday, November 20, 2016, 4:00 AM (0400) UTC
End Time:Sunday, November 20, 2016, 8:00 AM (0800) UTC
Description: Member Center for Education Network Initiatives in California's Optical circuit I2- SEAT-SUNN-100GE-07910 will be unavailable while PNWGP Engineers perform maintenance. A outage of 240 minutes is expected, however the entire maintenance window has been reserved.

Ticket No.:11016:144
Subject:Maintenance 2 of 2 - I2 Optical DWDM System CHAR-RALE
Affected:DWDM System CHAR-RALE
AL2S Backbone(s) ATLA-WASH & CHAR-RALE
IP Backbone(s) ATLA-WASH, ATLA-CLEV, ATLA-NEWY32AOA, & ASHB-ATLA
Member ESnet
Member NOAA N-Wave
Start Time:Sunday, November 20, 2016, 5:00 AM (0500) UTC
End Time:Sunday, November 20, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Level(3) Communications relocate fiber in Greensboro, NC. An outage is expected. The entire window is reserved. This is maintenance window 2 of 2.

Ticket No.:10915:144
Subject:Maintenance - I2 AL2S Backbone MINN-MISS2
Affected:AL2S Backbone MINN-MISS2
IP Net+ Provider Code42
Start Time:Sunday, November 20, 2016, 6:00 AM (0600) UTC
End Time:Sunday, November 20, 2016, 11:00 AM (1100) UTC
Description: The above listed asset will be unavailable while Zayo Engineers perform maintenance. An outage is expected. The entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:10880:144
Subject:Maintenance Rescheduled - I2 Optical Node opt2.newy32aoa
Affected:DWDM System NEWY1118TH-NEWY32AOA
AL2S Backbones HART2-NEWY32AOA, NEWY32AOA-PHIL,
IP Backbones ASHB-NEWY32AOA
Start Time:Tuesday, November 22, 2016, 4:00 AM (0400) UTC
End Time:Tuesday, November 22, 2016, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Level(3) Communications performs maintenance to replace a faulty chassis in New York. An outage is expected. The entire window is reserved.

Ticket No.:11038:144
Subject:Maintenance 1 of 2 - I2 Optical DWDM System MEMP-NASH
Affected:DWDM System MEMP-NASH
AL2S Backbone ATLA-JCSN
Member NOAA N-Wave
Member Univeristy of Memphis
Member MissiON
Member ESnet
Start Time:Thursday, December 1, 2016, 6:00 AM (0600) UTC
End Time:Thursday, December 1, 2016, 12:00 PM (1200) UTC
Description: The above listed assets will be unavailable while Level(3) Communications perform cable relocation maintenance in Mount Pleasant, TN. An outage is expected. The entire window is reserved.

Ticket No.:11039:144
Subject:Maintenance 2 of 2 - I2 Optical DWDM System MEMP-NASH
Affected:DWDM System MEMP-NASH
AL2S Backbone ATLA-JCSN
Member NOAA N-Wave
Member Univeristy of Memphis
Member MissiON
Member ESnet
Start Time:Friday, December 2, 2016, 6:00 AM (0600) UTC
End Time:Friday, December 2, 2016, 12:00 PM (1200) UTC
Description: The above listed assets will be unavailable while Level(3) Communications perform cable relocation maintenance in Mount Pleasant, TN. An outage is expected. The entire window is reserved.