SOX Network Operations Report
2019-02-01 - 2019-02-08
SOX Network Operations Center
| Connector | Type | Date | Sponsor |
|---|
None
None
None
| Incident Number | Customer Impact | Network Impact | Title | Outage Type | Source of Impact | Current State | Start Time (UTC) | End Time (UTC) | Duration |
|---|---|---|---|---|---|---|---|---|---|
| INC0027988 | 3 - Moderate | 3 - Moderate | Availability - SoX Participant CDC | Investigating | Undetermined | Closed | 2019-02-04 05:45:01 | 2019-02-04 08:29:20 | 0 days 2 hr 44 min |
| INC0028106 | 3 - Moderate | 3 - Moderate | Availability - SoX Core Node 365DC MGMTSW | Undetermined | Undetermined | Closed | 2019-02-06 06:00:15
2019-02-06 08:02:07 |
2019-02-06 06:57:06
2019-02-06 08:54:48 |
0 days 0 hr 56 min
0 days 0 hr 52 min |
| Ticket No.: | INC0027988 |
| Subject: | Availability - SoX Participant CDC |
| Source Of Impact: | Undetermined |
| Affected: | SOX-56MAR-56MAR-10GE-01715
Participant CDC |
| Current State: | Closed |
| Start Time: | 2019-02-04 05:45:01 |
| End Time: | 2019-02-04 08:29:20 |
| Description | Participant CDC is now available. CDC Engineers have been contacted; however, the reason for the outage is pending. |
| Ticket No.: | INC0028106 |
| Subject: | Availability - SoX Core Node 365DC MGMTSW |
| Source Of Impact: | Undetermined |
| Affected: | mgmtsw.365dc.sox.net
Core Node 365DC MGMTSW |
| Current State: | Closed |
| Start Time: | 2019-02-06 06:00:15 |
| End Time: | 2019-02-06 06:57:06 |
| Description | SoX Core Node 365DC MGMTSW is now available. SoX Engineering was engaged; however, the reason for outage was undetermined. |
| Ticket Number | Customer Impact | Network Impact | Title | Maintenance Type | Source Of Impact | Current State | Start Time (UTC) | End Time (UTC) | Duration |
|---|---|---|---|---|---|---|---|---|---|
| SOX Nodes | Down Time | Reporting Period Availability | 52 Week Availability |
| rtr.365dc.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.69314% | 99.69314% |
| dcp.365dc.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| rtr.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| mgmtsw.court.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw2.gtrch.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw1.55mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.97310% | 99.97310% |
| dcp.sidco.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw1.gtrch.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| sw1.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| rtr.sidco.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| dcp.court.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| bbsw.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| mgmtsw.365dc.sox.net | 1 hr 49 min | 1 hr 49 min | 99.04919% | 99.04919% | 99.90509% | 99.90509% |
| sw2.56mar.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.77091% | 99.77091% |
| mgmtsw.sidco.sox.net | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| Aggregate SOX Core Nodes | 0 days 1 hr 49 min | 0 days 1 hr 49 min | 99.93661% | 99.93661% | 99.95615% | 99.95615% |
NOTE: Numbers in italics represent downtime with scheduled maintenance excluded.
| SOX Backbone Circuits | Down Time | Reporting Period Availability | 52 Week Availability |
| SOX-56MAR-56MAR-FE-01935 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.96516% | 99.96516% |
| SOX-56MAR-56MAR-GIGE-01828 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01938 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01946 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01941 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-VLAN-01665 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-COURT-COURT-GIGE-01950 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-55MAR-56MAR-LAG-01892 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01934 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.96516% | 99.96516% |
| SOX-56MAR-56MAR-10GE-01718 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-365DC-365DC-FE-01954 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01944 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-55MAR-55MAR-LAG-01894 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01959 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-COURT-COURT-GIGE-01952 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01948 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01775 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-365DC-365DC-FE-01955 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01947 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01719 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01937 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-365DC-365DC-FE-01956 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-LAG-01862 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-365DC-365DC-FE-01953 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01930 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01929 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-365DC-56MAR-VLAN-01506 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01962 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01939 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01940 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01961 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01958 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01945 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01927 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-10GE-01743 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-100GE-01666 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.99571% | 99.99571% |
| SOX-56MAR-56MAR-10GE-01936 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.96516% | 99.96516% |
| SOX-56MAR-56MAR-GIGE-01933 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 99.96516% | 99.96516% |
| SOX-56MAR-GTRCH-GIGE-01931 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-FE-01943 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-SIDCO-SIDCO-FE-01960 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-COURT-COURT-GIGE-01951 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-55MAR-55MAR-LAG-01893 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-55MAR-10GE-01730 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-COURT-GIGE-01932 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| SOX-56MAR-56MAR-GIGE-01949 | 00 hr 00 min | 00 hr 00 min | 100.00000% | 100.00000% | 100.00000% | 100.00000% |
| Aggregate All SOX Backbone Circuits | 0 days 0 hr 0 min |0 days 0 hr 0 min | 100.00000% | 100.00000% | 99.99688% | 99.99688% |
NOTE: Numbers in italics represent downtime with scheduled maintenance excluded.
| Ticket No.: | CHG0035387 |
| Subject: | Maintenance - SoX Monitoring Tunnel via Indiana GigaPOP |
| Source Of Impact: | Other |
| Affected: | SOX-56MAR-56MAR-VLAN-01762
Monitoring Tunnel via Indiana GigaPOP |
| Start Time: | 2019-03-05 15:00:00 |
| End Time: | 2019-03-05 19:00:00 |
| Description | The SoX Monitoring tunnel via Indiana GigaPOP will be unavailable while their engineers perform software updates on rtr4.ictc.indiana.gigapop.net and rtr4.bldc.indiana.gigapop.net. Engineers will be rebooting one VPN aggregator at a time during the maintenance window. No network impact is expected, though monitoring will be briefly down. The entire maintenance window is reserved. |
| Ticket No.: | CHG0035129 |
| Subject: | Maintenance 1 of 2 Completed - SoX Peer Internet2 (ATLA) |
| Source Of Impact: | Vendor |
| Affected: | I2-ATLA-BATO-VLAN-183915
I2-DALL-JACK-VLAN-148528 I2-ATLA-BATO-VLAN-183914 I2-ATLA-BATO-VLAN-183916 Peer Internet2 (ATLA) |
| Start Time: | 2019-02-21 06:01:00 |
| End Time: | 2019-02-21 12:00:00 |
| Description | Peer Internet2 (ATLA) was unavailable while Peer Internet2 and CenturyLink Engineers performed maintenance to repair cable. This maintenance has been completed. This was maintenance window 1 of 2. |
| Ticket No.: | CHG0035137 |
| Subject: | Maintenance Completed - SoX Monitoring Tunnel via Indiana GigaPOP |
| Source Of Impact: | Other |
| Affected: | SOX-56MAR-56MAR-VLAN-01762
Monitoring Tunnel via Indiana GigaPOP |
| Start Time: | 2019-02-15 05:00:00 |
| End Time: | 2019-02-15 11:00:00 |
| Description | The SoX Monitoring tunnel via Indiana GigaPOP was unavailable while their engineers performed code upgrades in Bloomington. Maintenance has been completed. |
| Ticket No.: | CHG0035130 |
| Subject: | Maintenance 2 of 2 Completed - SoX Peer Internet2 (ATLA) |
| Source Of Impact: | Vendor |
| Affected: | I2-ATLA-BATO-VLAN-183915
I2-DALL-JACK-VLAN-148528 I2-ATLA-BATO-VLAN-183914 I2-ATLA-BATO-VLAN-183916 Peer Internet2 (ATLA) |
| Start Time: | 2019-02-22 06:01:00 |
| End Time: | 2019-02-22 12:00:00 |
| Description | Peer Internet2 (ATLA) was unavailable while Peer Internet2 and CenturyLink Engineers performed maintenance to repair cable. This maintenance has been completed and was maintenance window 2 of 2. |
| Ticket No.: | CHG0035178 |
| Subject: | Maintenance Completed - SoX Peer Internet2 (ATLA) |
| Source Of Impact: | Other |
| Affected: | I2-DALL-STAR-VLAN-183463
I2-DALL-STAR-VLAN-185359 I2-ATLA-TULS-VLAN-183898 Peer Internet2 (ATLA) |
| Start Time: | 2019-02-19 06:01:00 |
| End Time: | 2019-02-19 11:00:00 |
| Description | Peer Internet2 (ATLA) remained available while Internet2 Engineers performed software upgrade maintenance. Maintenance has been completed. |
| Ticket No.: | CHG0035067 |
| Subject: | Emergency Maintenance Completed - SoX Backbone 56MAR-56MAR |
| Source Of Impact: | Internal |
| Affected: | SOX-56MAR-56MAR-100GE-01666
SOX-365DC-56MAR-VLAN-01506 SOX-56MAR-56MAR-VLAN-01667 SOX-56MAR-56MAR-VLAN-01668 SOX-56MAR-56MAR-VLAN-01669 SOX-56MAR-56MAR-VLAN-01670 SOX-56MAR-56MAR-VLAN-01671 SOX-56MAR-56MAR-VLAN-01672 SOX-56MAR-56MAR-VLAN-01673 SOX-56MAR-56MAR-VLAN-01676 SOX-56MAR-56MAR-VLAN-01677 SOX-56MAR-56MAR-VLAN-01678 SOX-56MAR-56MAR-VLAN-01679 SOX-56MAR-56MAR-VLAN-01686 SOX-56MAR-56MAR-VLAN-01690 SOX-56MAR-56MAR-VLAN-01694 SOX-56MAR-56MAR-VLAN-01695 SOX-56MAR-56MAR-VLAN-01696 SOX-56MAR-56MAR-VLAN-01697 SOX-56MAR-56MAR-VLAN-01698 SOX-56MAR-56MAR-VLAN-01699 SOX-56MAR-56MAR-VLAN-01702 SOX-56MAR-56MAR-VLAN-01704 SOX-56MAR-56MAR-VLAN-01710 SOX-56MAR-56MAR-VLAN-01711 SOX-GENI-MP-VLAN-01603 SOX-XSEDE-MP-VLAN-01602 SOX-56MAR-56MAR-VLAN-01740 SOX-56MAR-56MAR-VLAN-01761 SOX-56MAR-56MAR-VLAN-01976 SoX-56MAR-56MAR-VLAN-01995 SoX-56MAR-56MAR-VLAN-02001 SoX-56MAR-56MAR-VLAN-02002 SoX-56MAR-56MAR-VLAN-02017 SoX-56MAR-56MAR-VLAN-02018 SoX-56MAR-56MAR-VLAN-02023 SoX-56MAR-56MAR-VLAN-02104 SoX-56MAR-56MAR-VLAN-02111 SoX-56MAR-56MAR-VLAN-02112 SoX-56MAR-56MAR-VLAN-02113 SoX-56MAR-56MAR-VLAN-02116 SoX-56MAR-56MAR-VLAN-02117 SoX-56MAR-56MAR-VLAN-02119 SOX-56MAR-56MAR-VLAN-02121 SoX-56MAR-56MAR-VLAN-02122 SoX-56MAR-56MAR-VLAN-02125 SoX-56MAR-56MAR-VLAN-02126 SoX-56MAR-56MAR-VLAN-02127 SoX-56MAR-56MAR-VLAN-02128 SoX-56MAR-56MAR-VLAN-02129 SoX-56MAR-56MAR-VLAN-02130 SoX-56MAR-56MAR-VLAN-02167 SoX-56MAR-56MAR-VLAN-02168 SoX-56MAR-56MAR-VLAN-02169 SoX-56MAR-56MAR-VLAN-02170 SoX-56MAR-56MAR-VLAN-02174 SoX-56MAR-56MAR-VLAN-02178 SoX-56MAR-56MAR-VLAN-02193 SoX-56MAR-56MAR-VLAN-02207 SoX-56MAR-56MAR-VLAN-02208 SoX-56MAR-56MAR-VLAN-02209 SoX-56MAR-56MAR-VLAN-02210 SoX-56MAR-56MAR-VLAN-02211 SoX-56MAR-56MAR-VLAN-02212 SoX-56MAR-56MAR-VLAN-02230 SOX-56MAR-SIDCO-VLAN-01506 SoX-56MAR-SIDCO-VLAN-01998 SoX-56MAR-SIDCO-VLAN-02070 SOX-SOX-56MAR-56MAR-02128 Backbone 56MAR-56MAR Peer OneNet Peer Red Clara Peer TeliaSonera Participant FLR Participant MARIA |
| Start Time: | 2019-02-18 05:00:00 |
| End Time: | 2019-02-18 10:00:00 |
| Description | The 100GE Backhaul to Internet2 AL2S was unavailable while Internet2 Engineers upgraded Jun0S versions on their Atlanta router. Blended ISP peerings remained available via Jacksonville during the maintenance window. Maintenance has been completed and no further outages are expected. |