This evening one of the access points serving subscribers in Redwood
Valley has failed. The vast majority of subscribers affected by this
failure have failed over gracefully to other equipment. Only a few
subscribers are currently without service. We will replace the failed
access point as soon as possible in the morning.
Other subscribers served by this same site may experience slower than normal service this evening with the added load placed on other equipment by this failure.
This afternoon, PG&E is likely to start a PSPS event which will
likely last until some time on Tuesday. This event will affect
subscribers in the Blue Lakes area.
We have sufficient
battery/generator runtime at most sites in the Blue Lakes area to get
through this PSPS event without any service interruption to our
subscribers that also have a means of backup power. We will be
delivering some additional battery capacity to some of those sites this
afternoon, in anticipation of the event.
There are a small number of subscribers around Lower Blue Lake for which service may be down during this PSPS event.
Please let us know if you have any questions.
It appears the troubled link is back up and traffic is again flowing freely. I’m still waiting to hear from the transport provider what the cause of the outage was, but things appear good for now.
8:56PM – I’m still on hold with the transport fiber provider’s support department while they investigate. While on-hold I have manually moved the majority of subscriber traffic to a backup link for a partial service restoration. Unfortunately, this backup link has about 5% the required capacity to carry the evening traffic load. I’m working on ordering a temporary capacity increase on this circuit for the duration of this outage.
Most subscribers should have service restored, though speeds will be quite low for now.
More updates to follow.
A fiber path between Ukiah and Fremont appears to be down currently, interrupting service for subscribers. I have opened a ticket with the fiber provider and am awaiting a response from them.
More to follow.
Service was fully restored on the affected link between Ukiah and Fremont at 8:37 this evening. I’m still awaiting followup from the fiber provider on the final resolution, but the bits are flowing freely again!
I’ll post a full report on the outage later this weekend on the website.
Thank you for your patience during this unexpected (and really long) service interruption. The positive messages received from many subscribers during the outage are very much appreciated.
Have a great rest of the weekend.
7:45PM – Unfortunately this update isn’t much of an update, except to say that the very large international carrier is still trying to get spares on-site to replace the failed line card.
I appreciate how frustrating it is to not be receiving a firm estimated time of repair in these messages. Unfortunately I can only pass on what little information I can get each time I wait on-hold, for sometimes over an hour for an update from the provider.
Hopefully more positive updates to follow soon.
The fiber provider has found a failed line card in one of their fiber aggregation switches and is working on getting a replacement card on-site and installed. They hope to have another update around 6:15PM.
The source of the connectivity problem appears to have been found within one of our fiber provider’s networks and they have dispatched technicians to diagnose and repair. Unfortunately I have not yet received an estimated time of repair from the provider as they are still en route.
If you happen to have my personal cell phone number I would like to remind you that it *is not a support line*, and calling me while I am awaiting important calls from technicians attempting to restore service only slows the process down.
More updates to follow.
Starting at just before noon a connectivity issue became apparent between our Ukiah head-end and Fremont datacenter. I’m currently in the process of determining the source of the problem. This issue is currently affecting all of our subscribers.
More updates to follow as I have them.