SOX Network Operations Report
2018-11-01 - 2018-11-30

SOX Network Operations Center

New Connections

ConnectorTypeDateSponsor


Upgrades

None


New Services


None


Other


None


Unscheduled Outages Summary

Incident Number Customer Impact Network Impact Title Outage Type Source of Impact Current State Start Time (UTC) End Time (UTC)

INC0022594 3 - Moderate3 - Moderate Availability - SoX Participant University of Alabama in HuntsvilleUndetermined Undetermined Resolved 2018-11-03 12:32:00 2018-11-03 21:26:06
INC0024247 3 - Moderate3 - Moderate Availability - SoX Participant EmoryUndetermined Undetermined Closed 2018-11-26 00:14:14 2018-11-26 00:52:00
INC0024318 3 - Moderate3 - Moderate Availability - SoX Node 365DC (Management)Undetermined Undetermined Resolved 2018-11-26 17:59:40 2018-11-26 20:02:17
INC0024212 3 - Moderate3 - Moderate Availability - SoX Participant Emory UniversityUndetermined Undetermined Resolved 2018-11-25 12:50:58 2018-11-25 21:54:46
INC0024267 3 - Moderate3 - Moderate Availability - SoX Peer GoogleUndetermined Undetermined Resolved 2018-11-26 07:08:47 2018-11-26 07:19:20
INC0024507 3 - Moderate3 - Moderate Brief Outage - SoX Participant CDCUndetermined Undetermined Closed 2018-11-28 15:07:36 2018-11-28 15:28:38
INC0024270 3 - Moderate3 - Moderate Availability - SoX Node 365DC (Management)Undetermined Undetermined Resolved 2018-11-26 08:14:29 2018-11-26 10:28:13
INC0024361 3 - Moderate3 - Moderate Availability - SoX Core Node 56MAR OOBUndetermined Undetermined Resolved 2018-11-27 05:43:00 2018-11-27 06:12:52
INC0023226 3 - Moderate3 - Moderate Availability - SoX Core Node 56MAR OOBUndetermined Undetermined Resolved 2018-11-12 06:20:37 2018-11-12 06:38:50
INC0024590 3 - Moderate3 - Moderate Availability - SoX Core Node 56MAR OOBOther Internal Resolved 2018-11-29 07:33:56 2018-11-29 08:03:33
INC0023458 3 - Moderate3 - Moderate Availability - SoX Node 365DC (Management)Undetermined Undetermined Resolved 2018-11-14 09:51:27 2018-11-14 10:39:00
INC0022750 3 - Moderate3 - Moderate Outage Resolved - SoX Participant CLIGHTUnannounced Maintenance Undetermined Resolved 2018-11-06 05:26:42 2018-11-06 06:41:51
INC0024571 3 - Moderate3 - Moderate Availability - SoX Participant AuburnUndetermined Undetermined Resolved 2018-11-29 01:17:00 2018-11-29 06:39:00
INC0023844 3 - Moderate3 - Moderate Availability - SoX Participant Oak Ridge National LabsUndetermined Undetermined Resolved 2018-11-19 11:49:04 2018-11-19 17:53:00

Unscheduled Outages Detail


Ticket No.:INC0022594
Subject:Availability - SoX Participant University of Alabama in Huntsville
Source Of Impact: Undetermined
Affected:SOX-56MAR-56MAR-10GE-01658
Participant University of Alabama in Huntsville
Current State:Resolved
Start Time:2018-11-03 12:32:00
End Time:2018-11-03 21:26:06
DescriptionParticipant University of Alabama in Huntsville is now available. University of Alabama in Huntsville Engineers have been contacted; however, the reason for the outage is pending.


Ticket No.:INC0024247
Subject:Availability - SoX Participant Emory
Source Of Impact: Undetermined
Affected:SoX-56MAR-56MAR-GIGE-01789
Participant Emory
Current State:Closed
Start Time:2018-11-26 00:14:14
End Time:2018-11-26 00:52:00
DescriptionSoX Participant Emory is now available. Emory Engineers have been contacted; however the reason for outage is pending.


Ticket No.:INC0024318
Subject:Availability - SoX Node 365DC (Management)
Source Of Impact: Undetermined
Affected:mgmtsw.365dc.sox.net
oob.365dc.sox.net
Node 365DC (Management)
Current State:Resolved
Start Time:2018-11-26 17:59:40
End Time:2018-11-26 20:02:17
DescriptionThe 365 Data Center OOB is now available. 365 Data Center Engineers have been contacted; however the reason for outage is pending.


Ticket No.:INC0024212
Subject:Availability - SoX Participant Emory University
Source Of Impact: Undetermined
Affected:SoX-56MAR-56MAR-GIGE-01789
Participant Emory University
Current State:Resolved
Start Time:2018-11-25 12:50:58
End Time:2018-11-25 21:54:46
DescriptionParticipant Emory University is Available. Emory University Engineers have been contacted however; the reason for the outage is pending.


Ticket No.:INC0024267
Subject:Availability - SoX Peer Google
Source Of Impact: Undetermined
Affected:SOX-56MAR-56MAR-LAG-01922
Peer Google
Current State:Resolved
Start Time:2018-11-26 07:08:47
End Time:2018-11-26 07:19:20
DescriptionSoX Peer Google is now available. Google Engineers have been contacted; however the reason for outage is pending.


Ticket No.:INC0024507
Subject:Brief Outage - SoX Participant CDC
Source Of Impact: Undetermined
Affected:SOX-56MAR-56MAR-10GE-01715
Participant CDC
Current State:Closed
Start Time:2018-11-28 15:07:36
End Time:2018-11-28 15:28:38
DescriptionParticipant CDC was briefly unavailable. Center for Disease Control Engineers have been engaged; however, the reason for the outage is pending.


Ticket No.:INC0024270
Subject:Availability - SoX Node 365DC (Management)
Source Of Impact: Undetermined
Affected:oob.365dc.sox.net
mgmtsw.365dc.sox.net
Node 365DC
Current State:Resolved
Start Time:2018-11-26 08:14:29
End Time:2018-11-26 10:28:13
DescriptionThe 365 Data Center Management Nodes are now available. 365DC and SoX Engineers have been contacted; however the reason for outage is pending.


Ticket No.:INC0024361
Subject:Availability - SoX Core Node 56MAR OOB
Source Of Impact: Undetermined
Affected:oob.56mar.sox.net
Core Node 56MAR OOB
Current State:Resolved
Start Time:2018-11-27 05:43:00
End Time:2018-11-27 06:12:52
DescriptionThe OOB for Core Node 56 Marietta St is Available. SoX Engineers have been contacted to investigate; however the reason for outage is pending.


Ticket No.:INC0023226
Subject:Availability - SoX Core Node 56MAR OOB
Source Of Impact: Undetermined
Affected:oob.56mar.sox.net
Core Node 56MAR OOB
Current State:Resolved
Start Time:2018-11-12 06:20:37
End Time:2018-11-12 06:38:50
DescriptionThe OOB device at 56 Marietta St is now available. SoX Engineers have been contacted to investiage; however the reason for outage is pending.


Ticket No.:INC0024590
Subject:Availability - SoX Core Node 56MAR OOB
Source Of Impact: Internal
Affected:oob.56mar.sox.net
Core Node 56MAR OOB
Current State:Resolved
Start Time:2018-11-29 07:33:56
End Time:2018-11-29 08:03:33
DescriptionThe 56 Marietta OOB is now available. SoX Engineers investigated and determined the reason for outage was a temporary drop in LTE connectivity. The cause of the LTE drop is undetermined. No further outages are expected.


Ticket No.:INC0023458
Subject:Availability - SoX Node 365DC (Management)
Source Of Impact: Undetermined
Affected:mgmtsw.365dc.sox.net
oob.365dc.sox.net
Node 365DC (Management)
Current State:Resolved
Start Time:2018-11-14 09:51:27
End Time:2018-11-14 10:39:00
DescriptionVarious management nodes at the 365 Data Center POP are now available. 365 Data Center and SoX Engineers have been contacted; however the reason for outage is pending. All alarms have cleared.


Ticket No.:INC0022750
Subject:Outage Resolved - SoX Participant CLIGHT
Source Of Impact: Undetermined
Affected:SOX-56MAR-56MAR-10GE-01608
Participant CLIGHT
Current State:Resolved
Start Time:2018-11-06 05:26:42
End Time:2018-11-06 06:41:51
DescriptionSoX Participant CLIGHT is Available. Pinged Internet2 and found that CLIGHT is doing internal maintenance affecting SoX connection.


Ticket No.:INC0024571
Subject:Availability - SoX Participant Auburn
Source Of Impact: Undetermined
Affected:SOX-56MAR-56MAR-10GE-01847
Participant Auburn
Current State:Resolved
Start Time:2018-11-29 01:17:00
End Time:2018-11-29 06:39:00
DescriptionParticipant Auburn is now available. Auburn Engineers have been contacted; however the reason for the outage is pending.


Ticket No.:INC0023844
Subject:Availability - SoX Participant Oak Ridge National Labs
Source Of Impact: Undetermined
Affected:SOX-56MAR-56MAR-10GE-01708
Participant Oak Ridge National Labs
Current State:Resolved
Start Time:2018-11-19 11:49:04
End Time:2018-11-19 17:53:00
DescriptionParticipant Oak Ridge National Labs is now unavailable. Oak Ridge National Labs Engineers have engaged their fiber vendor, TVA. TVA has dispatched a field tech with an estimated time of arrival of 14:15 UTC. The reason for the outage is pending.


Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Current State Start Time (UTC) End Time (UTC)

CHG00336043 - Moderate3 - Moderate Maintenance Completed - SoX Peer Internet2 (Backup) Circuit Internal Closed 2018-11-02 03:15:592018-11-02 04:00:00
CHG00338013 - Moderate3 - Moderate Decommission Completed - SoX POP IMART Other Internal Closed 2018-11-13 19:47:112018-11-14 18:00:00

Scheduled Maintenances Detail


Ticket No.:CHG0033604
Subject:Maintenance Completed - SoX Peer Internet2 (Backup)
Source Of Impact: Internal
Affected:SOX-MPLEX-SIDCO-VLAN-01513
Peer Internet2 (Backup)
Current State:Closed
Start Time:2018-11-02 03:15:59
End Time:2018-11-02 04:00:00
DescriptionPeer Internet2's backup connection was unavailable while SoX Engineers moved the peering from rtr.mplex to rtr.sidco in preparation for decommissioning rtr.mplex. No service impact was observed, as traffic continued to route out of the the primary 100G interface to the peer. Maintenance has been completed.


Ticket No.:CHG0033801
Subject:Decommission Completed - SoX POP IMART
Source Of Impact: Internal
Affected:rtr.imart.sox.net
POP IMART
Current State:Closed
Start Time:2018-11-13 19:47:11
End Time:2018-11-14 18:00:00
DescriptionSoX Engineers will be decommissioning the InfoMart Data Center co-location in Dallas, TX during the listed window. During the window, Engineers will remove all the equipment at the site. Decommission work is expected to be completed by 12:00AM CST 11/14; however, the entire window is reserved.


Downtime and Availability



SOX NodesDown TimeReporting Period Availability52 Week Availability
rtr.365dc.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 99.69314% | 100.00000%
dcp.365dc.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
rtr.56mar.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
mgmtsw.court.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
sw2.gtrch.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
sw1.55mar.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 99.97310% | 100.00000%
dcp.sidco.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
sw1.gtrch.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
sw1.56mar.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
rtr.sidco.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
dcp.court.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
bbsw.56mar.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
mgmtsw.365dc.sox.net5 hr 3 min | 5 hr 3 min 99.29653% | 99.29653% 99.94234% | 100.00000%
sw2.56mar.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 99.77091% | 100.00000%
mgmtsw.sidco.sox.net00 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%

Aggregate SOX Core Nodes0 days 5 hr 3 min | 0 days 5 hr 3 min 99.95310% | 99.95310% 99.95863% | 100.00000%



NOTE: Numbers in italics represent downtime with scheduled maintenance excluded.



SOX Backbone CircuitsDown TimeReporting Period Availability52 Week Availability
SOX-56MAR-56MAR-FE-0193500 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0182800 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-FE-0193800 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0194600 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-FE-0194100 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-VLAN-0166500 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-COURT-COURT-GIGE-0195000 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-55MAR-56MAR-LAG-0189200 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0193400 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-10GE-0171800 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-365DC-365DC-FE-0195400 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-FE-0194400 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-55MAR-55MAR-LAG-0189400 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-SIDCO-SIDCO-FE-0195900 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-COURT-COURT-GIGE-0195200 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0194800 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-10GE-0177500 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-365DC-365DC-FE-0195500 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0194700 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-10GE-0171900 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-FE-0193700 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-365DC-365DC-FE-0195600 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-LAG-0186200 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-365DC-365DC-FE-0195300 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-COURT-GIGE-0193000 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-COURT-GIGE-0192900 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-365DC-56MAR-VLAN-0150600 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-SIDCO-SIDCO-FE-0196200 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0193900 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0194000 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-SIDCO-SIDCO-FE-0196100 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-SIDCO-SIDCO-FE-0195800 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0194500 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-COURT-GIGE-0192700 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-10GE-0174300 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-100GE-0166600 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 99.99571% | 100.00000%
SOX-56MAR-56MAR-10GE-0193600 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0193300 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-GTRCH-GIGE-0193100 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-FE-0194300 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-SIDCO-SIDCO-FE-0196000 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-COURT-COURT-GIGE-0195100 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-55MAR-55MAR-LAG-0189300 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-55MAR-10GE-0173000 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-COURT-GIGE-0193200 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%
SOX-56MAR-56MAR-GIGE-0194900 hr 00 min | 00 hr 00 min 100.00000% | 100.00000% 100.00000% | 100.00000%

Aggregate All SOX Backbone Circuits0 days 0 hr 0 min |0 days 0 hr 0 min 100.00000% | 99.95310%99.99991% | 100.00000%



NOTE: Numbers in italics represent downtime with scheduled maintenance excluded.





Upcoming Scheduled Maintenances




Task Records not Included in this report

Task NumberTask TitleReason not in Report
CHG0033572Maintenance Completed - SoX Peer Netflix (OCA)Not in report because source of impact was set to customer.
INC0024245Availability - SoX Peer Great AccessNo circuits, nodes, or other elements were listed as impacted in the report.
INC0023849Availability - SoX Peer AmazonNo circuits, nodes, or other elements were listed as impacted in the report.
INC0022004Outage Resolved - SoX Peer NationalNetNo circuits, nodes, or other elements were listed as impacted in the report.
INC0022579Brief Outage - SoX Participant CLIGHTNot in report because source of impact was set to customer.
No circuits, nodes, or other elements were listed as impacted in the report.
INC0022845Outage Resolved - SoX Peer OneNetNot in report because source of impact was set to customer.
No circuits, nodes, or other elements were listed as impacted in the report.
INC0023797Outage Resolved - SoX Peer ATLNX (Atlantic Nexus)Not in report because source of impact was set to customer.
INC0022423Availability - SoX Peer YahooNo circuits, nodes, or other elements were listed as impacted in the report.
INC0023459Availability - SoX Peer Georgia Public WebNo circuits, nodes, or other elements were listed as impacted in the report.
INC0022747Availability - SoX Peer YahooNo circuits, nodes, or other elements were listed as impacted in the report.