Resolved – Cascading failures


Earlier this evening one of our fiber transport providers partially repaired one of our links that had been down since yesterday afternoon. During the repair, the link flapped on and off a number of times which caused a ripple effect across our network as traffic shifted between backhaul links, causing low speeds or no service at all for subscribers off and on. 

I’ve disabled the circuit being repaired until it can be more fully tested in order to stop the port and route flapping that has been causing issues for subscribers for the last 1.5 hours or so. Things should be back to normal now.

Thanks for your understanding.

– Jacob