[Original Post ] We're aware of all servers in New York, US. being offline. We have contacted the data center about this and we're waiting on an update and resolution from them.
This does not affect Extreme Plus plans in New York (MCEP01-BUF-US) since this node is in a different datacenter.
[Update @ 10:30 AM - 25/10/2023 ET] The outage continues as of now. We've been told this is being caused due to a fiber cut on a Zayo dark fiber ring that is used to backhaul traffic for this location.
This incident is being tracked on: https://tranzact.zayo.com/#!/case-detail/5004z00001tVV9sAAG/TTN-0007089358/
The work on this should have finished by 10:30 AM ET. The last update on their end confirms that the repair work has been completed on the fiber itself, however no service restoration is being seen as originally expected, thus further investigation is being conducted.
We understand that an outage as long as this is frustrating, but unfortunately right now our hands are tied and there's not much we can do on our end other than waiting. This will be covered by our SLA and compensation will be available once the incident has been solved. We're also evaluating to move all customers in this location to a different datacenter in the upcoming weeks, as this kind of outage and service quality is unacceptable and it doesn't align with the level of service that we strieve to provide at HeavyNode.
We firmly believe that backhauling traffic from an entire location over a single provider (Zayo) even with two paths (full ring), without a backup solution in place is something that shouldn't have happened at all, and it's the reason why we're now looking at alternative providers to move customers in this location in the future.
Updates will follow as we receive them.
[Update @ 11:54 AM - 25/10/2023 ET] We've received word from the DC/provider that alternate transit is being brought online ASAP now.
[Update @ 14:42 PM - 25/10/2023 ET] Alternate transit has been brought online and all services are now reachable and operational. We're continuing to monitor the incident.
Compensation for this downtime is available as per our SLA. Please open a ticket with our billing department within the next 7 days to request this.
Friday, October 27, 2023
