Category: System Status

As of 6PM tonight, crews are on-site replacing and re-splicing the damaged sections of fiber.  I am told that this may take a few hours, at least, but has begun.  In the interim, the capacity upgrades to the aforementioned backup link have been completed.  Subscribers will still experience slower than normal service, but it should be functional at least for typical web browsing.  Streaming video will not work satisfactorily until the primary link is restored.
Thanks for your understanding.
– Jacob

Word from ATT is that the current service interruption is being caused by a fiber cut in Napa, caused during road construction.  We have not yet received an estimated time of repair though.
A capacity upgrade on the limited capacity backup link that traffic is currently flowing over was ordered around noon today.  That upgrade has been partially completed, and I’m hoping it will be finalized shortly.
Thanks for your understanding.
– Jacob

The problem transport circuit is still down as of noon and we are awaiting updates from the provider.  At 11AM I re-routed traffic over a different path to Fremont.  This backup path is a very low capacity circuit, but I have placed an upgrade order for it that will hopefully be completed shortly.  Subscribers should be up and running, albeit very slowly, as of about 11:15 this morning.  
I’ll send an additional update once we have an ETR from the provider of the primary circuit.
Thanks,
– Jacob

At 10:18 this AM a transport circuit between Ukiah and Fremont is having issues (again).  I have opened a ticket with the carrier and we are awaiting a response.
Thanks,
– Jacob

This evening one of the access points serving subscribers in Redwood Valley has failed.  The vast majority of subscribers affected by this failure have failed over gracefully to other equipment.  Only a few subscribers are currently without service.   We will replace the failed access point as soon as possible in the morning.

Other subscribers served by this same site may experience slower than normal service this evening with the added load placed on other equipment by this failure.

Thanks,

– Jacob

This afternoon, PG&E is likely to start a PSPS event which will likely last until some time on Tuesday.  This event will affect subscribers in the Blue Lakes area. 

We have sufficient battery/generator runtime at most sites in the Blue Lakes area to get through this PSPS event without any service interruption to our subscribers that also have a means of backup power.​  We will be delivering some additional battery capacity to some of those sites this afternoon, in anticipation of the event.

There are a small number of subscribers around Lower Blue Lake for which service may be down during this PSPS event.

Please let us know if you have any questions.

Thanks,

– Jacob

It appears the troubled link is back up and traffic is again flowing freely. I’m still waiting to hear from the transport provider what the cause of the outage was, but things appear good for now.
Thanks,
– Jacob

8:56PM – I’m still on hold with the transport fiber provider’s support department while they investigate.  While on-hold I have manually moved the majority of subscriber traffic to a backup link for a partial service restoration.  Unfortunately, this backup link has about 5% the required capacity to carry the evening traffic load.  I’m working on ordering a temporary capacity increase on this circuit for the duration of this outage.
Most subscribers should have service restored, though speeds will be quite low for now.
More updates to follow.
Thanks, 
– Jacob

A fiber path between Ukiah and Fremont appears to be down currently, interrupting service for subscribers.  I have opened a ticket with the fiber provider and am awaiting a response from them.
More to follow.
– Jacob

Service was fully restored on the affected link between Ukiah and Fremont at 8:37 this evening.  I’m still awaiting followup from the fiber provider on the final resolution, but the bits are flowing freely again!
I’ll post a full report on the outage later this weekend on the website.
Thank you for your patience during this unexpected (and really long) service interruption.  The positive messages received from many subscribers during the outage are very much appreciated.
Have a great rest of the weekend.
-Jacob