Unified - Internet2 IP Network Operations Report
10/20/2014 - 10/26/2014

Unified - Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None

New Services

None



Unscheduled Outages Summary







Ticket Number
Customer Impact
Network Impact
Title
Outage Type
Source Of Impact
Start Time (UTC)

End Time (UTC)





11031-Critical
1-Critical

Outage Resolved - I2 Optical DWDM System PHOE-SAND
Cut or Damaged FiberVendor
10/20/2014 3:10 PM
10/21/2014 6:10 AM


11621-Critical
1-Critical

Brief Outage Resolved - I2 Optical Various Circuits
Human ErrorVendor
10/23/2014 5:33 AM
10/23/2014 5:34 AM


11661-Critical


Brief Outage - I2 Optical DWDM BOST-HART2
InvestigatingUndetermined
10/23/2014 6:13 PM
10/23/2014 6:14 PM


11922-High


Brief Outage - I2 AL2S Backbone BOST-HART2 
HardwareVendor
10/25/2014 4:19 AM
10/25/2014 4:20 AM


11973-Elevated
3-Elevated

Brief Outage - I2 Optical Various Circuits
InvestigatingUndetermined
10/25/2014 6:27 PM
10/25/2014 6:28 PM






Unscheduled Outages Detail






Ticket No.:1103:144



Subject:Outage Resolved - I2 Optical DWDM System PHOE-SAND



Source Of Impact:Vendor



Affected:DWDM System PHOE-SAND
AL2S Backbone LOSA-PHOE
IP Backbone HOUS-LOSA
IP Connector Sun Corridor (LOSA)*
IP Connector CENIC (HOUS)*



Start Time:Monday, October 20, 2014, 3:10 PM (1510) UTC



End Time:Tuesday, October 21, 2014, 6:10 AM (0610) UTC


Description:The above-listed was unavailable to the community.  Level 3
has indicated that the reason for the outage was fiber damage caused by
construction. The fiber has been repaired and no future outages are expected.

* indicates entity is multihomed and therefore remained reachable.







Ticket No.:1162:144



Subject:Brief Outage Resolved - I2 Optical Various Circuits



Source Of Impact:Vendor



Affected:AL2S Backbone PORT-SEAT
IP Backbone LOSA-SEAT



Start Time:Thursday, October 23, 2014, 5:33 AM (0533) UTC



End Time:Thursday, October 23, 2014, 5:34 AM (0534) UTC


Description:The above listed Optical circuit was briefly unavailable. Level(3) Communications has 
been contacted. The reason for the outage is was due to preparation work being done for 
the emergency maintenance being tracked in ticket 1154:144. This maintenance is 
currently active and extends to 13:00 UTC.







Ticket No.:1166:144



Subject:Brief Outage - I2 Optical DWDM BOST-HART2



Source Of Impact:Undetermined



Affected:AL2S Backbone BOST-HART2
IP Backbone CLEV-NEWY32AOA



Start Time:Thursday, October 23, 2014, 6:13 PM (1813) UTC



End Time:Thursday, October 23, 2014, 6:14 PM (1814) UTC


Description:The above listed assets were briefly unavailable.  Level(3) Engineers
have been engaged.  No RFO is available at this time.  Layer2 and
Layer3 Backbones re-routed during the event. Workgroup administrators
receive notifications from OESS regarding circuits re-routed on the
backbone.







Ticket No.:1192:144



Subject:Brief Outage - I2 AL2S Backbone BOST-HART2 



Source Of Impact:Vendor



Affected:AL2S Backbone BOST-HART2
IP Backbone CLEV-NEWY32AOA



Start Time:Saturday, October 25, 2014, 4:19 AM (0419) UTC



End Time:Saturday, October 25, 2014, 4:20 AM (0420) UTC


Description:Backbone BOST-HART2 was briefly unavailable.  Internet2 Engineers have indicated 
that the reason for the outage was a hardware failure, resolved by hardware 
replacement.  Layer 2 and 3 backbone traffic re-routed during the outage.  Workgroup 
administrators receive notifications from OESS regarding circuits re-routed on the 
backbone.







Ticket No.:1197:144



Subject:Brief Outage - I2 Optical Various Circuits



Source Of Impact:Undetermined



Affected:Optical Various Circuits
IP Backbone ATLA-HOUS
IP Connector FLR/SOX (ATLA)



Start Time:Saturday, October 25, 2014, 6:27 PM (1827) UTC



End Time:Saturday, October 25, 2014, 6:28 PM (1828) UTC


Description:The above listed was briefly unavailable.  Level 3 and FLR/SOX Engineers have been 
contacted, however the reason for the outage is pending.  Layer 3 traffic re-routed 
during the outage.







Scheduled Maintenances Summary





Ticket Number
Customer Impact
Network Impact
Title

Maintenance Type
Source Of Impact
Start Time (UTC)
End Time (UTC)





690
3-Elevated
1-Critical

Maintenance 1 of 2 Completed - I2 Optical DWDM System LASV-LOSA
Circuit
Vendor10/20/2014 7:42 AM
10/20/2014 8:09 AM


1053
4-Normal


Maintenance Completed - I2 AL2S Core Node WASH (Interfaces 7/1 and et-3/0/0)
CircuitInternal
10/22/2014 4:42 AM
10/22/2014 5:03 AM


1046
3-Elevated


Maintenance Completed - I2 Optical Circuit I2-BOST-HART2-100GE-12807
Hardware
Vendor10/23/2014 4:46 AM
10/23/2014 5:09 AM


1154
3-Elevated


Emergency Maintenance Completed - I2 Optical DWDM System PORT-SEAT
CircuitVendor
10/23/2014 8:46 AM
10/23/2014 9:28 AM




Scheduled Maintenances Detail





Ticket No.:690:144



Subject:Maintenance 1 of 2 Completed - I2 Optical DWDM System LASV-LOSA



Source Of Impact:Vendor



Affected:DWDM System LASV-LOSA
AL2S Backbone LOSA-SALT
IP Backbone LOSA-SALT



Start Time:Monday, October 20, 2014, 7:42 AM (0742) UTC



End Time:Monday, October 20, 2014, 8:09 AM (0809) UTC


Description:   DWDM System  LASV-LOSA was unavailable while Level(3) Communications 
relocated
backbone cable in San Bernardino, CA. An outage was experienced. This window of 
the maintenance has been completed. 
*This is maintenance window 1 of 2*



  







Ticket No.:1053:144



Subject:Maintenance Completed - I2 AL2S Core Node WASH (Interfaces 7/1 and et-3/0/0)



Source Of Impact:Internal



Affected:AL2S Interconnect to IP (WASH)
AL2S NET+ Provider Blackboard
IP Backbone NEWY32AoA-WASH
IP Backbone CLEV-WASH
IP Peer RNP
IP NET+ Provider Amazon
IP NET+ Provider Blackboard
IP Connector Mid-Atlantic Crossroads (WASH)*
IP Connector CAAREN (WASH)*
IP Connector NIH (WASH)*
IP Connector 3ROX (WASH)*
IP Connector Smithsonian (WASH)*
IP Connector MARIA (WASH)*
IP Connector UPENN (WASH)*



Start Time:Wednesday, October 22, 2014, 4:42 AM (0442) UTC



End Time:Wednesday, October 22, 2014, 5:03 AM (0503) UTC


Description:   The above listed were unavailable while Internet2 Engineers with
the assistance of Level(3) Communications performed fiber cleaning maintenance. Layer 2 and 3 backbone traffic was re-routed during this maintenance. Maintenance 
has been completed.

* Indicates entity is multihomed and therefore will remain available.



  







Ticket No.:1046:144



Subject:Maintenance Completed - I2 Optical Circuit I2-BOST-HART2-100GE-12807



Source Of Impact:Vendor



Affected:AL2S Backbone BOST-HART2
IP Backbone CLEV-NEWY32AOA



Start Time:Thursday, October 23, 2014, 4:46 AM (0446) UTC



End Time:Thursday, October 23, 2014, 5:09 AM (0509) UTC


Description:   The above listed assets were unavailable while Level(3) Communications performed 
hardware replacement in Hartford, CT. Maintenance has been completed. No further 
outages are expected.  IP and AL2S Backbones re-routed during the event. Workgroup 
administrators receive notifications from OESS regarding circuits re-routed on the 
backbone.



  







Ticket No.:1154:144



Subject:Emergency Maintenance Completed - I2 Optical DWDM System PORT-SEAT



Source Of Impact:Vendor



Affected:Optical DWDM System LACY-SEAT
AL2S Backbone PORT-SEAT
IP Backbone LOSA-SEAT
NDDI Backbone LOSA-SEAT



Start Time:Thursday, October 23, 2014, 8:46 AM (0846) UTC



End Time:Thursday, October 23, 2014, 9:28 AM (0928) UTC


Description:   The above listed are now available after Level 3 Communications
performed fiber
maintenance.  Level(3) has confirmed they are hands off at the
worksite.  No further outages are expected.  Layer 2 and 3 backbone
traffic will re-route during this maintenance.  Workgroup
administrators will receive notifications from OESS regarding circuits
re-routed on the backbone.



  










Downtime and Availability







Unified - Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability


ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
      
Aggregate Unified - Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%100.00000%100.00000%

 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  
	    

	    
Unified - Internet2 IP Backbone CircuitsDown Time
Reporting Period Availability52 Week Availability


R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.82544%99.97268%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.75979%99.99924%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99753%100.00000%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96110%99.99924%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99469%99.99962%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177
               0 hr 25 min0 hr 2 min
               99.75198%99.98016%
               99.95313%99.99772%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178
               0 hr 21 min0 hr 0 min
               99.79167%100.00000%
               99.97306%99.99507%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.93871%99.99981%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189
               15 hr 0 min15 hr 0 min
               91.07143%91.07143%
               99.76415%99.76415%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.91272%99.98975%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188
               0 hr 27 min0 hr 0 min
               99.73214%100.00000%
               99.99488%100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187
               0 hr 43 min0 hr 1 min
               99.57341%99.99008%
               99.98368%99.99943%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328
               0 hr 43 min0 hr 1 min
               99.57341%99.99008%
               99.98368%99.99943%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176
               0 hr 21 min0 hr 0 min
               99.79167%100.00000%
               99.99374%99.99886%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99753%100.00000%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.82677%99.97306%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423
               0 hr 1 min0 hr 1 min
               99.99008%99.99008%
               99.60155%99.83796%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99621%100.00000%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99621%100.00000%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96452%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96376%99.99962%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.94346%99.99753%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559
               15 hr 0 min15 hr 0 min
               91.07143%91.07143%
               99.80609%99.82772%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.93150%99.99924%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.93150%99.99924%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563
               0 hr 27 min0 hr 0 min
               99.73214%100.00000%
               99.99488%100.00000%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564
               0 hr 42 min0 hr 0 min
               99.58333%100.00000%
               99.97875%99.99962%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99431%100.00000%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99886%100.00000%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
      
Aggregate All Unified - Internet2 IP Backbone Circuits
                   34 hr 10 min4 hr 5 min
                   99.45034%99.51603%
                   99.94492%99.98241%


 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  

Upcoming Scheduled Maintenances





Ticket No.:387:144



Subject:Maintenance 2 of 2 Completed - I2 Optical Various Circuits (Norther Tier)



Affected:Optical Circuits EQCH-FARG and FARG-SEAT



Start Time:Tuesday, October 28, 2014, 6:00 AM (0600) UTC



End Time:Tuesday, October 28, 2014, 11:00 AM (1100) UTC



Description:   ---Note Updated Affected---
The above listed was unavailable while vendor Engineers performed node installation 
maintenance in Helena, Montana.  The maintenance is completed.



  







Ticket No.:1133:144



Subject:Maintenance *Tracking* - I2 IP OOB (ATLA)



Affected:IP OOB (ATLA)



Start Time:Thursday, October 30, 2014, 4:00 AM (0400) UTC



End Time:Thursday, October 30, 2014, 6:00 AM (0600) UTC



Description:   IP OOB (ATLA) will be unavailable while Level 3 Engineers perform hardware upgrade 
maintenance.



  







Ticket No.:1163:144



Subject:Maintenance - I2 Optical Member ESnet (SUNN)



Affected:Optical Member ESnet (SUNN)



Start Time:Thursday, October 30, 2014, 4:30 PM (1630) UTC



End Time:Thursday, October 30, 2014, 7:30 PM (1930) UTC



Description:   Member ESnet (SUNN)'s Optical circuits listed above will be unavailable while ESnet 
Engineers upgrade the operating system on their router SUNN-CR5. A outage is 
expected and the entire maintenance window has been reserved.



  







Ticket No.:1218:144



Subject:Maintenance - I2 IP Connector LEARN (HOUS)



Affected:Connector LEARN (HOUS) and its Participants
AL2S Connector LEARN (DALL)



Start Time:Monday, November 3, 2014, 6:00 AM (0600) UTC



End Time:Monday, November 3, 2014, 12:00 PM (1200) UTC



Description:   Connector LEARN will be unavailable while LEARN's vendor perform a maintenance to 
relocate fiber due to a road widening project. A brief outage of 1 to 2 minutes is 
expected; however, the entire window is reserved. Connector LEARN is multihomed and 
therefore should remain available.



  







Ticket No.:1216:144



Subject:Maintenance - I2 Optical DWDM System ATLA-CHAR 



Affected:IP Backbone ATLA-WASH
AL2S Backbone ATLA-CHAR
AL2S Backbone ATLA-JACK
Connector MCNC  (ATLA)*



Start Time:Tuesday, November 4, 2014, 4:00 AM (0400) UTC



End Time:Tuesday, November 4, 2014, 6:00 AM (0600) UTC



Description:   DWDM System ATLA-CHAR should remain available while
Level(3) Communications perform a maintenance on the Managed Ciena Network to
verify that static routes are being redistributed into OSPF  in  Atlanta, GA.   An
outage  is not expected; however the entire
window is reserved. Workgroup administrators receive notifications from OESS
regarding circuits re-routed on the backbone. Backbone traffic will re-route during the window.
* Represents that the affected is multihomed and should remain available.



  







Ticket No.:1155:144



Subject:Maintenance *Tracking* - I2 Optical Various Circuits (Northern Tier)



Affected:Optical Various Circuits (Northern Tier)



Start Time:Wednesday, November 5, 2014, 7:00 AM (0700) UTC



End Time:Tuesday, November 11, 2014, 11:00 AM (1100) UTC



Description:   The above listed will remain available while Zayo Engineers perform power 
maintenance.  No outage is expected; however, the entire window is reserved.



  







Ticket No.:1194:144



Subject:Maintenance - I2 Optical Circuit I2-MINN-MISS2-100GE-09036 (Northern Tier)



Affected:AL2S Backbone MINN-MISS2
Net+ Provider Code42



Start Time:Friday, November 7, 2014, 8:00 AM (0800) UTC



End Time:Friday, November 7, 2014, 10:00 AM (1000) UTC



Description:   The above listed Optical circuit will be unavailable while our fiber provider's engineers 
perform circuit maintenance. An outage or 30 minutes is expected, however the entire 
maintenance window has been reserved.



  







Ticket No.:1240:144



Subject:Maintenance - I2 Optical DWDM System ATLA-CHAT



Affected:Optical DWDM System ATLA-CHAT



Start Time:Tuesday, November 11, 2014, 5:00 AM (0500) UTC



End Time:Tuesday, November 11, 2014, 11:00 AM (1100) UTC



Description:   The above listed will be unavailable while Level 3 Communications performs fiber 
relocation maintenance.  An outage is expected, and the entire window is reserved.  
Layer 2 and 3 backbone traffic will re-route during this maintenance.  Workgroup 
administrators will receive notifications from OESS regarding circuits re-routed on the 
backbone.