Indiana GigaPOP Network Operations Report
2018-11-30 - 2018-12-13

Indiana GigaPOP Network Operations Center

New Connections

ConnectorTypeDateSponsor


Upgrades

None


New Services


None


Other


None


Unscheduled Outages Summary

Incident Number Customer Impact Network Impact Title Outage Type Source of Impact Current State Start Time (UTC) End Time (UTC)
INC0025168 2 - High2 - High Outage Resolved - Indiana GigaPOP Backbone BLDC-LLCircuit - Damaged Fiber Internal Closed 2018-12-05 16:49:34 2018-12-06 09:08:18
INC0025707 3 - Moderate3 - Moderate Alarms - Indiana GigaPOP ns02.ip.wustl.edu HOST and DNS_SERVER_RESPONSEUndetermined Undetermined Closed 2018-12-11 11:26:00 2018-12-11 14:28:34

Unscheduled Outages Detail


Ticket No.:INC0025168
Subject:Outage Resolved - Indiana GigaPOP Backbone BLDC-LL
Source Of Impact: Internal
Affected:ING-BLDC-LL-LAG-03613
Backbone BLDC-LL
Current State:Closed
Start Time:2018-12-05 16:49:34
End Time:2018-12-06 09:08:18
DescriptionBackbone BLDC-LL is now available. I-Light Engineers have been contacted and have confirmed fiber damage near Indianapolis, Edgewood Ave. one (1) mile south of the 465 interchange. Splicers were dispatched and ran new cable to repair the damaged fiber. No further outages are expected.


Ticket No.:INC0025707
Subject:Alarms - Indiana GigaPOP ns02.ip.wustl.edu HOST and DNS_SERVER_RESPONSE
Source Of Impact: Undetermined
Affected:ns02.ip.wustl.edu
Member St. Louis Internet2 Access Consortium (WUSTL) (128.252.0.17)
Current State:Closed
Start Time:2018-12-11 11:26:00
End Time:2018-12-11 14:28:34
Description Alarms have since cleared, resolving. Pending RFO at this time.


Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Current State Start Time (UTC) End Time (UTC)

Scheduled Maintenances Detail


None during this report period

Downtime and Availability


Indiana GigaPOP NodesDown TimeReporting Period Availability52 Week Availability
sw3.ictc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
sw.bldc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
rtr.bldc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
rtr4.ictc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
rtr.ll.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
rtr2.chic.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
sw2.bldc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
rtr.ictc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
rtr4.bldc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
sw.eqch.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
rtr3.ictc.indiana.gigapop.net00 hr 00 min100.00000% 100.00000%
Aggregate Indiana GigaPOP Core Nodes0 days 0 hr 0 min 100.00000% 100.00000%

Indiana GigaPOP Backbone CircuitsDown TimeReporting Period Availability52 Week Availability
ING-BLDC-BLDC-LAG-0362500 hr 00 min 100.00000% 100.00000%
ING-LL-LL-10GE-0378100 hr 00 min 100.00000% 100.00000%
GLOBAL-IPIU-IPIU-10GE-2495500 hr 00 min 100.00000% 100.00000%
GLOBAL-ILN-ING-10GE-2398200 hr 00 min 100.00000% 100.00000%
GLOBAL-ILN-ING-10GE-0413700 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-10GE-0376100 hr 00 min 100.00000% 100.00000%
ING-BLDC-LL-LAG-0361316 hr 18 min 95.14517% 99.68072%
ING-ICTC-ICTC-10GE-0363300 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-LAG-0363400 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-10GE-0365700 hr 00 min 100.00000% 100.00000%
ING-LL-LL-LAG-0368700 hr 00 min 100.00000% 100.00000%
GLOBAL-ILN-ING-10GE-0413800 hr 00 min 100.00000% 100.00000%
ILN-IPIU-IPIU-GIGE-0410400 hr 00 min 100.00000% 100.00000%
ING-ICTC-IOT-10GE-0364600 hr 00 min 100.00000% 100.00000%
ING-CHIC-EQCH-10GE-0377100 hr 00 min 100.00000% 100.00000%
GLOBAL-ING-IPG-10GE-167900 hr 00 min 100.00000% 100.00000%
GLOBAL-CIC-ING-10GE-0168000 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-10GE-0366200 hr 00 min 100.00000% 100.00000%
ING-LL-NCAA-GIGE-0372700 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-10GE-0376200 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-10GE-0171200 hr 00 min 100.00000% 100.00000%
GLOBAL-ILN-ING-10GE-0371200 hr 00 min 100.00000% 100.00000%
ING-LL-LL-GIGE-0378400 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-10GE-0171100 hr 00 min 100.00000% 100.00000%
ILN-IPIU-IPIU-GIGE-0410500 hr 00 min 100.00000% 100.00000%
ING-BLDC-BLDC-10GE-0362300 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-LAG-0361800 hr 00 min 100.00000% 100.00000%
ING-BLDC-BLDC-10GE-0362400 hr 00 min 100.00000% 100.00000%
ING-PURDUE-LL-10GE-0164100 hr 00 min 100.00000% 100.00000%
ING-111N-UL-10GE-0380900 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-10GE-0366100 hr 00 min 100.00000% 100.00000%
GLOBAL-ILN-ING-LAG-0413400 hr 00 min 100.00000% 100.00000%
ING-CHIC-CHIC-10GE-0382200 hr 00 min 100.00000% 100.00000%
ING-INDYZOO-LL-GIGE-0369400 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-GIGE-0371800 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-LAG-0366600 hr 00 min 100.00000% 100.00000%
ING-LL-COGENT-10GE-0162700 hr 00 min 100.00000% 100.00000%
ILN-ICTC-ICTC-10GE-2393200 hr 00 min 100.00000% 100.00000%
ING-NCAA-ICTC-GE-0164000 hr 00 min 100.00000% 100.00000%
GLOBAL-ILN-ING-10GE-0371300 hr 00 min 100.00000% 100.00000%
ING-LL-LL-LAG-0378300 hr 00 min 100.00000% 100.00000%
ING-111N-111N-LAG-168400 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-10GE-0366500 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-GIGE-0377700 hr 00 min 100.00000% 100.00000%
ING-LL-LL-GIGE-0378900 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-10GE-0376000 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-10GE-0371900 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-LAG-0376700 hr 00 min 100.00000% 100.00000%
ING-BLDC-BLDC-GIGE-0377000 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-GIGE-0363700 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-LAG-0365800 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-LAG-0366000 hr 00 min 100.00000% 100.00000%
ILN-IPIU-IPIU-GIGE-0410600 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-10GE-0363200 hr 00 min 100.00000% 100.00000%
ING-BLDC-BLDC-GIGE-0369900 hr 00 min 100.00000% 100.00000%
ING-BLDC-BLDC-GIGE-0377400 hr 00 min 100.00000% 100.00000%
ING-LL-SAIN-10GE-0361200 hr 00 min 100.00000% 100.00000%
ING-BLDC-ICTC-LAG-0361600 hr 00 min 100.00000% 100.00000%
GLOBAL-FG-ING-10GE-150000 hr 00 min 100.00000% 100.00000%
ING-600W-ICTC-LAG-0358700 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-GIGE-0151700 hr 00 min 100.00000% 100.00000%
ING-BLDC-ICTC-10GE-0361900 hr 00 min 100.00000% 100.00000%
ING-LL-LL-10GE-0368100 hr 00 min 100.00000% 100.00000%
ING-PURDUE-LL-10GE-0366300 hr 00 min 100.00000% 100.00000%
ING-IOT-LL-10GE-0364700 hr 00 min 100.00000% 100.00000%
ING-ICTC-ICTC-10GE-0365600 hr 00 min 100.00000% 100.00000%
GLOBAL-ILN-ING-LAG-0405900 hr 00 min 100.00000% 100.00000%
ING-ICTC-LL-LAG-0171300 hr 00 min 100.00000% 100.00000%
Aggregate All Indiana GigaPOP Backbone Circuits0 days 16 hr 18 min 99.92861% 99.99530%


Upcoming Scheduled Maintenances

Ticket No.:CHG0034403
Subject:Maintenance Completed - Indiana GigaPOP Internal Configuration Clean-up
Source Of Impact: Internal
Affected:
Start Time:2018-12-27 15:30:00
End Time:2018-12-27 16:00:00
DescriptionIndiana GigaPOP Engineers cleaned up some unused interfaces, no outages or alarms were observed. No notification was required. Maintenance is completed.


Ticket No.:CHG0034246
Subject:Maintenance Completed - Indiana GigaPOP Member University of Notre Dame
Source Of Impact: Internal
Affected:Member University of Notre Dame
Start Time:2018-12-14 17:00:00
End Time:2018-12-14 22:00:00
DescriptionMember University of Notre Dame remained available while Peer I-Light Engineers performed maintenance. No outages were observed, and maintenance has been completed.


Ticket No.:CHG0034229
Subject:Maintenance Completed - Indiana GigaPOP Internal Update RIB Group Statements on rtr2.chic.indiana.gigapop.net
Source Of Impact: Internal
Affected:
Start Time:2018-12-18 09:00:00
End Time:2018-12-18 11:00:00
DescriptionIndiana GigaPop Engineers were updating routing policies on Indiana GigaPop router rtr2.chic. No outages occurred and the entire maintenance window was reserved.


Ticket No.:CHG0034510
Subject:Maintenance Completed - Indiana GigaPOP Control Plane Policing for Core Routers CHIC, ICTC and LL
Source Of Impact: Internal
Affected:
Start Time:2019-01-13 09:00:00
End Time:2019-01-13 11:00:00
DescriptionIndiana GigaPOP Engineers implemented a new control plane policing filter for Core Routers CHIC, ICTC and LL. The change did not cause any disruptions to Member connectivity. Maintenance has been completed.


Ticket No.:CHG0034440
Subject:Maintenance Completed - Indiana GigaPOP Various Backbones
Source Of Impact: Internal
Affected:rtr.ictc.indiana.gigapop.net
rtr.ll.indiana.gigapop.net
Backbone ICTC-LL
Backbone BLDC-ICTC
Backbone ICTC-IPIU
Backbone CHIC-LL
Backbone BLDC-LL
Fiber/Services Provider Wintek Corporation
Fiber/Services Provider Smithville Digital
Member NCAA
Member IU Research Network
Member Purdue University
Member Indiana Office of Technology
Member Indiana University
Member University of Chicago
Member University of Notre Dame
Member Mid West Tier 2 Admins
Member St. Louis Internet2 Access Consortium (WUSTL)
Member Indianapolis Zoo
Member CSPAN
Member FutureGrid
Peer Pennsylvania State University
Peer OARnet
Peer University of Minnesota - Twin Cities
Peer University of Illinois at Urbana-Champaign
CPS Peer Netflix Streaming Services
Start Time:2019-01-06 09:00:00
End Time:2019-01-06 11:00:00
DescriptionPeer I-Light Engineers have implemented a new control plane policing filter. No outages occurred and this maintenance has been completed.


Ticket No.:CHG0034391
Subject:Maintenance Completed - Indiana GigaPOP Core Nodes ICTC and LL (Routing Policy Changes)
Source Of Impact: Internal
Affected:Core Node ICTC
Core Node LL
Start Time:2019-01-02 09:30:00
End Time:2019-01-02 11:00:00
DescriptionIndiana GigaPOP Engineers implemented additional as-path filtering for Member eBGP Peers and anti-spoofing for Members' interfaces on Indiana GigaPOP Core Routers rtr.ictc and rtr.ll. No outages occurred and the entire window was reserved.


Ticket No.:CHG0034226
Subject:Maintenance Completed - Indiana GigaPOP Internal St. Louis Internet2 Access Consortium (WUSTL) - I2/ING Routing
Source Of Impact: Internal
Affected:
Start Time:2018-12-17 10:00:00
End Time:2018-12-17 11:00:00
DescriptionIndiana GigaPOP network engineers will updating the ING to I2 routing preferences. This will restore traffic to the normally preferred route.. Currently we're running on a less preferred route to skirt an issue that we recently resolved. Time to move things back.


Ticket No.:CHG0034388
Subject:Maintenance Completed - Indiana GigaPOP Unused BGP Sessions Cleanup
Source Of Impact: Internal
Affected:
Start Time:2018-12-26 13:30:00
End Time:2018-12-26 16:00:00
Description Indiana GigaPOP Engineers cleaned up configurations for BGP sessions on rtr.ictc and rtr.ll. No outages were observed. Maintenance completed


Ticket No.:CHG0034270
Subject:Maintenance Completed - Indiana GigaPOP Various Nodes Interface Description Clean Up
Source Of Impact: Internal
Affected:rtr.ll.indiana.gigapop.net
rtr.ictc.indiana.gigapop.net
rtr2.chic.indiana.gigapop.net
oob.chic.indiana.gigapop.net
rtr4.bldc.indiana.gigapop.net
Start Time:2018-12-17 15:00:00
End Time:2018-12-17 17:00:00
DescriptionThe above listed remained available while Indiana GigaPOP Network Engineers cleaned up some interface descriptions by removing [PENDING] tags from ports that are now active. No outages were observed. Maintenance is completed.


Ticket No.:CHG0034273
Subject:Maintenance Completed - Indiana GigaPOP Core Node ICTC
Source Of Impact: Internal
Affected:opt2.ictc.indiana.gigapop.net
Core Node ICTC
Start Time:2018-12-18 05:00:00
End Time:2018-12-18 11:00:00
DescriptionI-Light engineers will be will be swapping out power supplies on opt.ictc. No outage is expected.


Ticket No.:CHG0034242
Subject:Maintenance Completed - Indiana GigaPOP Core Node ICTC
Source Of Impact: Internal
Affected:Core Node ICTC
Member FutureGrid
Member I-Light
Member IU Research Network
Member Indiana Office of Technology (IOT) PP
Member Indiana University
Member Mid West Tier 2 Admins
Member NCAA
Member Netflix Streaming Services
Member OARnet
Member Pennsylvania State University
Member Purdue University
Member Smithville Digital
Member University of Chicago
Member University of Illinois at Urbana-Champaign
Member University of Minnesota - Twin Cities
Member University of Notre Dame
Start Time:2018-12-18 05:00:00
End Time:2018-12-18 11:00:00
DescriptionING engineers were swapping out the Backup-RE0 on rtr.ictc. "Loss of communication with backup RE" alarms occurred. There were no outages and the entire window was reserved.


Ticket No.:CHG0034374
Subject:Maintenance Completed - Indiana GigaPOP Member University of Notre Dame
Source Of Impact: Vendor
Affected:Member University of Notre Dame
Start Time:2019-01-11 05:01:00
End Time:2019-01-11 11:00:00
DescriptionMember University of Notre Dame remained available while Zayo Engineers performed Fiber Maintenance at the intersection of Clay Street South and Hoover Lane located in Markle Indiana. Maintenance has been completed.


Ticket No.:CHG0034121
Subject:Maintenance Completed - Indiana GigaPOP Routing Policy Changes
Source Of Impact: Internal
Affected:
Start Time:2018-12-26 10:00:00
End Time:2018-12-26 11:00:00
DescriptionIndiana GigaPOP Engineers were implementing as-path filtering for Member eBGP Peers and anti-spoofing on Members interfaces. Maintenance has been completed.


Ticket No.:CHG0034453
Subject:Maintenance Completed - Indiana GigaPOP Backbone 600W-ICTC
Source Of Impact: Internal
Affected:Backbone 600W-ICTC
Start Time:2019-01-11 06:00:00
End Time:2019-01-11 12:00:00
DescriptionIndiana GigaPOP Engineers were working on the Monon400 connections between 600W and ICTC. Maintenance has been completed.