Indiana GigaPOP Network Operations Report
2021-07-23 - 2021-08-05
Indiana GigaPOP Network Operations Center
| Incident Number | Customer Impact | Network Impact | Title | Outage Type | Source of Impact | Current State | Start Time (UTC) | End Time (UTC) | Duration |
|---|---|---|---|---|---|---|---|---|---|
| INC0088495 | 3 - Moderate | 3 - Moderate | Brief Routing Instability - Indiana GigaPOP Core Node ICTC | Investigating | Undetermined | Resolved | 2021-07-29 17:03:00 | 2021-07-29 17:18:00 | 0 days 0 hr 15 min |
| INC0088611 | 3 - Moderate | 3 - Moderate | Brief Routing Instability - Indiana GigaPOP Core Node ICTC | Investigating | Undetermined | Resolved | 2021-07-31 00:50:00 | 2021-07-31 01:42:00 | 0 days 0 hr 52 min |
| INC0088807 | 3 - Moderate | 3 - Moderate | Brief Outage - Indiana GigaPOP Backbone BLDC-ICTC | Undetermined | Undetermined | Resolved | 2021-08-03 16:49:50 | 2021-08-03 16:53:36 | 0 days 0 hr 3 min |
| Ticket No.: | INC0088495 |
| Subject: | Brief Routing Instability - Indiana GigaPOP Core Node ICTC |
| Source Of Impact: | Undetermined |
| Affected: | rtr.ictc.indiana.gigapop.net
Core Node ICTC Backbone ICTC-LL Backbone ICTC-ICTC Backbone CHIC-ICTC Backbone BLDC-ICTC Backbone ICTC-IPIU Backbone ILN-ING Peer FutureGrid Member IU Research Network Member Indiana Office of Technology Member Indiana University Peer Mid West Tier 2 Admins Member NCAA Peer Netflix Streaming Services Peer Pennsylvania State University Peer Purdue Research Member Purdue University Peer University of Chicago Member University of Notre Dame Peer University of Wisconsin at Madison Member Wabash College |
| Current State: | Resolved |
| Start Time: | 2021-07-29 17:03:00 |
| End Time: | 2021-07-29 17:18:00 |
| Outgoing Notification Text | Indiana GigaPOP Engineers installed a new MPC in rtr.ictc.indiana.gigapop.net at approximately 1:03 PM ET. At completion of the MPC booting, alarms were raised against the chassis fabric on Core Node ICTC. Initial observation indicated only the new MPC was impacted; however, alarms for connectivity were raised shortly after. The new MPC was offlined at 1:18 PM ET, which cleared the alarms for connectivity. Indiana GigaPOP Engineers are currently investigating with Vendor Juniper to determine the cause of the fabric impact. Re-creating may not be possible and attempts to install the new MPC will be done via scheduled maintenance. |
| Ticket No.: | INC0088611 |
| Subject: | Brief Routing Instability - Indiana GigaPOP Core Node ICTC |
| Source Of Impact: | Undetermined |
| Affected: | rtr.ictc.indiana.gigapop.net
Core Node ICTC Backbone ICTC-LL Backbone BLDC-ICTC IBackbone CTC-ICTC Backbone CHIC-ICTC Backbone ICTC-LL Member FutureGrid Member I-Light Member IU Research Network Member Indiana Office of Technology Member Indiana University Member Mid West Tier 2 Admins Member NCAA Peer Netflix Streaming Services Member Pennsylvania State University Member Purdue Research Member Purdue University Member Purdue University (PURDUE) Member University of Chicago Member University of Notre Dame (UND) Member University of Wisconsin at Madison Member Wabash College |
| Current State: | Resolved |
| Start Time: | 2021-07-31 00:50:00 |
| End Time: | 2021-07-31 01:42:00 |
| Outgoing Notification Text | Indiana GigaPOP's previously troublesome and "offlined" MPC in rtr.ictc.indiana.gigapop.net suddenly came back online this evening around 8:50 PM ET. At completion of the MPC booting, alarms were raised against the chassis fabric on Core Node ICTC. Initial observation indicated only the new MPC was impacted; however, alarms for connectivity were raised shortly after. The new MPC was again offlined at 9:42 PM ET, which cleared the alarms for connectivity. Shortly after this shutdown, this card was physically removed from the chassis. Indiana GigaPOP Engineers are continuing the investigation with Vendor Juniper. A fresh set of logs and update has been sent to Juniper for further investigation. |
| Ticket No.: | INC0088807 |
| Subject: | Brief Outage - Indiana GigaPOP Backbone BLDC-ICTC |
| Source Of Impact: | Undetermined |
| Affected: | ING-BLDC-ICTC-LAG-03616
Backbone BLDC-ICTC |
| Current State: | Resolved |
| Start Time: | 2021-08-03 16:49:50 |
| End Time: | 2021-08-03 16:53:36 |
| Outgoing Notification Text | Backbone BLDC-ICTC was briefly unavailable. Indiana GigaPOP Engineers are engaged and investigating the root cause for the outage. The reason for outage is undetermined at this time. Further updates to be provided as they are made available. |