Category: System Status

Resolved – Cascading failures

2021-07-07

Earlier this evening one of our fiber transport providers partially repaired one of our links that had been down since yesterday afternoon. During the repair, the link flapped on and off a number of times which caused a ripple effect across our network as traffic shifted between backhaul links, causing low speeds or no service at all for subscribers off and on. 

I’ve disabled the circuit being repaired until it can be more fully tested in order to stop the port and route flapping that has been causing issues for subscribers for the last 1.5 hours or so. Things should be back to normal now.

Thanks for your understanding.

– Jacob

Resolved (for now) – Continued power outages – Willits and parts of Redwood Valley

One of our sites on Laughlin Peak between Willits and Redwood Valley has been without power since Tuesday evening.  It has been running on backup battery and minimal solar backup, but the batteries have finally gone flat this morning.  Grid power is expected to be restored by 10PM tonight.  I will be delivering a generator to the site later this morning though, if the access road is passable, to restore service sooner.

This outage affects some subscribers in Redwood Valley, Ridgewood Ranch, Western Willits, and Brooktrails.

Thanks,

– Jacob

Resolved – Power outage – N. Ukiah, Redwood Valley, Potter Valley, Calpella

This afternoon at a bit before 3PM we were able to deliver a generator to the affected mountaintop site and restore service for those subscribers that themselves had power.   As of ~7PM this evening grid power is also restored at the site and no further outages are expected. 

Thanks for your patience while we cut trees out of the way and slogged through the snow to get service restored (it was a fun outage to resolve).

– Jacob

Resolved (for now) – Power outage – N. Ukiah, Redwood Valley, Potter Valley, Calpella

After clearing a number of trees from a snowy mountain road, we were able to access the affected site and start a backup generator. PG&E power is still not restored and there isn’t yet an ETR being provided by PG&E. The generator/battery combo should hopefully keep things powered up until the morning when we can make another trip up to refuel if needed.

Thanks,

– Jacob

Power outage – N. Ukiah, Redwood Valley, Potter Valley, Calpella

​Last night at around 9:45PM PG&E power failed at one of our mountain top sites that serves many subscribers in Northern Ukiah, Redwood Valley, Potter Valley, Calpella, and a few in Willits.  The backup battery bank at this site was depleted at around 5:30 this morning and the equipment at the site shut down.  As of yet, PG&E does not have an estimated time of restoration for power in the area.   A convenient PG&E outage map is available here:   https://m.pge.com/#outages​

Once it is light out, I will assess the snow situation at the site and if possible, deliver a generator to the site to restore service until PG&E power is restored.  The earliest I anticipate that happening is 10AM this morning.  Depending on snow conditions and downed trees across the road, service restoration may be later than estimated.

Thanks for your understanding.

– Jacob

Resolved – Fiber transport issue

As of approximately 5 minutes ago, repairs appear to have been completed on the fiber break affecting subscribers today.   I have re-routed all traffic over primary paths and the bits are flowing freely again. 
I’ll send another update once all of the details are available on the outage, and continue to monitor this circuit for issues.  For now, enjoy full speeds again!
Thanks,
– Jacob