SOX Network Operations Report
2018-10-01 - 2018-10-31
SOX Network Operations Center
| Connector | Type | Date | Sponsor |
|---|
None
None
None
| Incident Number | Customer Impact | Network Impact | Title | Outage Type | Source of Impact | Current State | Start Time (UTC) | End Time (UTC) |
|---|---|---|---|---|---|---|---|---|
| INC0022125 | 3 - Moderate | 3 - Moderate | Availability - SoX Participant Georgia Tech | Circuit - Cut Fiber | Vendor | Closed | 2018-10-27 11:53:00 | 2018-10-28 05:44:41 |
| INC0021500 | 3 - Moderate | 3 - Moderate | Availability - SoX Participant Emory | Undetermined | Undetermined | Resolved | 2018-10-18 06:35:49 | 2018-10-18 13:57:19 |
| INC0022190 | 3 - Moderate | 3 - Moderate | Availability - SoX Core Node 56MAR (OOB) | Undetermined | Undetermined | Resolved | 2018-10-29 05:23:13 | 2018-10-29 09:14:03 |
| INC0021597 | 3 - Moderate | 3 - Moderate | Availability - SoX Participant Morehouse (MSM) | Investigating | Undetermined | Closed | 2018-10-19 15:34:00 | 2018-10-21 16:22:00 |
| INC0020861 | 3 - Moderate | 3 - Moderate | Outage Resolved - SoX Various Participants | Unannounced Maintenance | Vendor | Closed | 2018-10-10 08:52:51 | 2018-10-10 12:05:47 |
| INC0021918 | 3 - Moderate | 3 - Moderate | Availability - Sox Peer Comcast | Investigating | Undetermined | Resolved | 2018-10-24 14:11:07 | 2018-10-24 22:44:27 |
| INC0021286 | 3 - Moderate | 3 - Moderate | Availability - SoX Core Node 56MAR (OOB) | Undetermined | Undetermined | Resolved | 2018-10-16 05:54:10 | 2018-10-16 07:09:41 |
| INC0020778 | 3 - Moderate | 3 - Moderate | Outage Resolved - SoX Participant CLIGHT | Investigating | Undetermined | Closed | 2018-10-09 04:51:16 | 2018-10-09 09:44:15 |
| INC0022152 | 3 - Moderate | 3 - Moderate | Availability - SoX Node 365DC | Software | Internal | Resolved | 2018-10-28 09:26:38 | 2018-10-29 12:23:55 |
| INC0022368 | 3 - Moderate | 3 - Moderate | Availability - SoX Peer GTT Communications | Undetermined | Undetermined | Resolved | 2018-10-31 13:21:14 | 2018-10-31 15:36:20 |
| INC0020424 | 3 - Moderate | 3 - Moderate | Outage Resolved - SoX Participant Vanderbilt University | Unannounced Maintenance | Internal | Closed | 2018-10-04 10:53:29 | 2018-10-09 14:45:28 |
| Ticket No.: | INC0022125 |
| Subject: | Availability - SoX Participant Georgia Tech |
| Source Of Impact: | Vendor |
| Affected: | SOX-SIDCO-SIDCO-10GE-02060
Participant Georgia Tech |
| Current State: | Closed |
| Start Time: | 2018-10-27 11:53:00 |
| End Time: | 2018-10-28 05:44:41 |
| Description | Participant Georgia Tech is now available. CenturyLink and Georgia Tech Engineers have been contacted and the reason for outage is a fiber cut. No ETR is yet available. |
| Ticket No.: | INC0021500 |
| Subject: | Availability - SoX Participant Emory |
| Source Of Impact: | Undetermined |
| Affected: | SOX-56MAR-56MAR-VLAN-01790
SoX-56MAR-56MAR-GIGE-01789 Participant Emory |
| Current State: | Resolved |
| Start Time: | 2018-10-18 06:35:49 |
| End Time: | 2018-10-18 13:57:19 |
| Description | Participant Emory is now available. Emory University Engineers have been contacted; however, the reason for the outage is pending. |
| Ticket No.: | INC0022190 |
| Subject: | Availability - SoX Core Node 56MAR (OOB) |
| Source Of Impact: | Undetermined |
| Affected: | oob.56mar.sox.net
Core Node 56MAR (OOB) |
| Current State: | Resolved |
| Start Time: | 2018-10-29 05:23:13 |
| End Time: | 2018-10-29 09:14:03 |
| Description | The 56 Marietta OOB is now available. SoX engineerining is currently investigating; however, the reason for the outage is pending. |
| Ticket No.: | INC0021597 |
| Subject: | Availability - SoX Participant Morehouse (MSM) |
| Source Of Impact: | Undetermined |
| Affected: | SOX-56MAR-56MAR-GIGE-01836
Participant Morehouse |
| Current State: | Closed |
| Start Time: | 2018-10-19 15:34:00 |
| End Time: | 2018-10-21 16:22:00 |
| Description | Participant Morehouse (MSM) is now available. Morehouse Engineers have been contacted; however, the reason for the outage is pending. |
| Ticket No.: | INC0020861 |
| Subject: | Outage Resolved - SoX Various Participants |
| Source Of Impact: | Vendor |
| Affected: | SOX-SIDCO-SIDCO-10GE-02082
SOX-56MAR-56MAR-10GE-01658 SOX-56MAR-56MAR-100GE-02013 SOX-SIDCO-SIDCO-100GE-02048 Participant UAB Participant UAH Participant Vanderbilt Participant University of Alabama System |
| Current State: | Closed |
| Start Time: | 2018-10-10 08:52:51 |
| End Time: | 2018-10-10 12:05:47 |
| Description | The below listed participants are now available. Participant Engineers reported the reason for the outage was emergency fiber relocation maintenance between Huntsville and Birmingham, AL. Maintenance has been completed and no further outages are expected. |
| Ticket No.: | INC0021918 |
| Subject: | Availability - Sox Peer Comcast |
| Source Of Impact: | Undetermined |
| Affected: | SOX-GTRCH-GTRCH-GIGE-01975
Peer Comcast |
| Current State: | Resolved |
| Start Time: | 2018-10-24 14:11:07 |
| End Time: | 2018-10-24 22:44:27 |
| Description | Peer Comcast is now available. Comcast Engineers have been contacted; however, the reason for the outage is pending. |
| Ticket No.: | INC0021286 |
| Subject: | Availability - SoX Core Node 56MAR (OOB) |
| Source Of Impact: | Undetermined |
| Affected: | oob.56mar.sox.net
Core Node 56MAR (OOB) |
| Current State: | Resolved |
| Start Time: | 2018-10-16 05:54:10 |
| End Time: | 2018-10-16 07:09:41 |
| Description | The OOB for Core Node 56 Marietta St is now available. SoX Engineers have been contacted to investiage; however the reason for outage is pending. |
| Ticket No.: | INC0020778 |
| Subject: | Outage Resolved - SoX Participant CLIGHT |
| Source Of Impact: | Undetermined |
| Affected: | SOX-56MAR-56MAR-10GE-01608
Participant CLIGHT |
| Current State: | Closed |
| Start Time: | 2018-10-09 04:51:16 |
| End Time: | 2018-10-09 09:44:15 |
| Description | Participant CLIGHT is now available. CLIGHT Engineers reported the reason for the outage was maintenance. |
| Ticket No.: | INC0022152 |
| Subject: | Availability - SoX Node 365DC |
| Source Of Impact: | Internal |
| Affected: | rtr.365dc.sox.net
Node 365 Data Center |
| Current State: | Resolved |
| Start Time: | 2018-10-28 09:26:38 |
| End Time: | 2018-10-29 12:23:55 |
| Description | The 365 Data Center Node is now available. SoX Engineers were contacted, and engaged Vanderbilt Engineers to dispatch to the site. Vanderbilt Engineers determined that the filesystem was full, and found the switch in Aboot mode. The tech deleted the old support files, and rebooted the switch. No further outages are expected. |
| Ticket No.: | INC0022368 |
| Subject: | Availability - SoX Peer GTT Communications |
| Source Of Impact: | Undetermined |
| Affected: | SoX-56MAR-56MAR-VLAN-02182
Peer GTT Communications |
| Current State: | Resolved |
| Start Time: | 2018-10-31 13:21:14 |
| End Time: | 2018-10-31 15:36:20 |
| Description | Peer GTT Communications is now available. GTT Communications Engineers have been engaged, however, the reason for the outage is pending. |
| Ticket No.: | INC0020424 |
| Subject: | Outage Resolved - SoX Participant Vanderbilt University |
| Source Of Impact: | Internal |
| Affected: | SOX-MPLEX-MPLEX-10GE-01523
Participant Vanderbilt University |
| Current State: | Closed |
| Start Time: | 2018-10-04 10:53:29 |
| End Time: | 2018-10-09 14:45:28 |
| Description | Vanderbilt University's 10G connection is currently unavailable. SoX Engineers have confirmed the reason for outage is a planned shutdown of the circuit, as a separate 100G connection has been turned up. No estimated time of alarm clearance is available. |
| Ticket Number | Customer Impact | Network Impact | Title | Maintenance Type | Source Of Impact | Current State | Start Time (UTC) | End Time (UTC) |
|---|---|---|---|---|---|---|---|---|
| CHG0033248 | 3 - Moderate | 3 - Moderate | Maintenance Completed - SoX Backbone AKARD-IMART | Hardware | Internal | Closed | 2018-10-09 05:32:16 | 2018-10-09 06:02:28 |
| Ticket No.: | CHG0033248 |
| Subject: | Maintenance Completed - SoX Backbone AKARD-IMART |
| Source Of Impact: | Internal |
| Affected: | SOX-AKARD-IMART-10GE-01547
SOX-IMART-56MAR-VLAN-01562 SOX-IMART-IMART-VLAN-01550 Backbone AKARD-IMART Peer Apple |
| Current State: | Closed |
| Start Time: | 2018-10-09 05:32:16 |
| End Time: | 2018-10-09 06:02:28 |
| Description | The backbone between AKARD and IMART were unavailable while Internet2 Engineers performed maintenance on their core node in Dallas, TX. The entire window was reserved. |
| SOX Nodes | Down Time | Reporting Period Availability | 52 Week Availability |
| rtr.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw2.gtrch.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw1.55mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.97310% | 100.00000% |
| dcp.sidco.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw1.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| rtr.imart.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.99571% | 100.00000% |
| dcp.court.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| bbsw.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| rtr.365dc.sox.net | 1 days 2 hr 57 min | 1 days 2 hr 57 min | 96.37705% | 96.37705% | 99.69314% | 100.00000% |
| rtr.mplex.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| dcp.mplex.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| mgmtsw.court.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw1.gtrch.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| rtr.sidco.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| mgmtsw.mplex.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| mgmtsw.sidco.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw2.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.77091% | 100.00000% |
| Aggregate SOX Core Nodes | 1 days 2 hr 57 min | 1 days 2 hr 57 min | 99.78689% | 99.78689% | 99.96664% | 100.00000% |
NOTE: Numbers in italics represent downtime with scheduled maintenance excluded.
| SOX Backbone Circuits | Down Time | Reporting Period Availability | 52 Week Availability |
| SOX-56MAR-56MAR-FE-01935 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01828 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01938 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-MPLEX-MPLEX-FE-01953 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01946 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01941 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-VLAN-01665 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-MPLEX-VLAN-01731 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-COURT-COURT-GIGE-01950 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-55MAR-56MAR-LAG-01892 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01934 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01718 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01944 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-55MAR-55MAR-LAG-01894 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01959 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-COURT-COURT-GIGE-01952 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01948 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01775 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-MPLEX-MPLEX-FE-01954 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01947 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01719 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01937 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-LAG-01862 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01930 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01929 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-365DC-56MAR-VLAN-01506 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01962 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01939 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-MPLEX-MPLEX-FE-01955 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01940 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01961 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-MPLEX-MPLEX-FE-01956 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01958 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01945 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01743 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-100GE-01666 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.99571% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01927 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01936 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01933 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-GTRCH-GIGE-01931 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01943 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01960 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-MPLEX-MPLEX-FE-01957 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-COURT-COURT-GIGE-01951 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-MPLEX-56MAR-VLAN-01522 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-55MAR-55MAR-LAG-01893 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-55MAR-10GE-01730 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01932 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01949 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| Aggregate All SOX Backbone Circuits | 0 days 0 hr 0 min |0 days 0 hr 0 min | 100.00000% | 99.78689% | 99.99991% | 100.00000% |
NOTE: Numbers in italics represent downtime with scheduled maintenance excluded.
| Ticket No.: | CHG0033572 |
| Subject: | Maintenance - SoX Peer Netflix (OCA) |
| Source Of Impact: | Customer |
| Affected: | |
| Start Time: | 2018-11-05 13:00:00 |
| End Time: | 2018-11-05 16:00:00 |
| Description | Peer Netflix will be unavailable while their Engineers perform maintenance. An outage of 30 hours is expected; however, the entire window is reserved. |
| Ticket No.: | CHG0033500 |
| Subject: | Decommission - SoX POP MPLEX |
| Source Of Impact: | Internal |
| Affected: | oob.mplex.sox.net
rtr.mplex.sox.net mgmtsw.mplex.sox.net dcp.mplex.sox.net POP MPLEX |
| Start Time: | 2018-11-06 16:00:00 |
| End Time: | 2018-11-06 21:00:00 |
| Description | SoX Engineers will be decommissioning their co-location with CenturyLink at the 460 Metroplex, Nashville, TN POP. During this window, Engineers will be removing all the equipment at the site (oob.mplex, rtr.mplex, mgmtsw.mplex, dcp.mplex). The entire window is reserved. |
| Ticket No.: | CHG0033604 |
| Subject: | Maintenance - SoX Peer Internet2 (Backup) |
| Source Of Impact: | Internal |
| Affected: | SOX-MPLEX-SIDCO-VLAN-01513
Peer Internet2 (Backup) |
| Start Time: | 2018-11-02 03:00:00 |
| End Time: | 2018-11-02 04:00:00 |
| Description | Peer Internet2's backup connection will be unavailable while SoX Engineers move the peering from rtr.mplex to rtr.sidco in preparation of the decommissioning of rtr.mplex. No service impact is expected, as traffic is routed out of the the primary 100G interface to the peer. The entire window is reserved. |
| Ticket No.: | CHG0033698 |
| Subject: | Maintenance - Sox Provider Cogent (365DC) |
| Source Of Impact: | Customer |
| Affected: | SoX-365DC-365DC-10GE-02187
Provider Cogent |
| Start Time: | 2018-11-10 05:01:00 |
| End Time: | 2018-11-10 11:00:00 |
| Description | Provider Cogent will be unavailable while their engineers perform software upgrade maintenance on their edge router in Nashville, TN. Two brief outages are expected; however, the entire window is reserved. |
| Task Number | Task Title | Reason not in Report |
| CHG0033549 | Maintenance Completed - SoX Peer |