Resolved – Cascading failures

2021-07-07

Earlier this evening one of our fiber transport providers partially repaired one of our links that had been down since yesterday afternoon. During the repair, the link flapped on and off a number of times which caused a ripple effect across our network as traffic shifted between backhaul links, causing low speeds or no service at all for subscribers off and on. 

I’ve disabled the circuit being repaired until it can be more fully tested in order to stop the port and route flapping that has been causing issues for subscribers for the last 1.5 hours or so. Things should be back to normal now.

Thanks for your understanding.

– Jacob

Resolved (for now) – Continued power outages – Willits and parts of Redwood Valley

One of our sites on Laughlin Peak between Willits and Redwood Valley has been without power since Tuesday evening.  It has been running on backup battery and minimal solar backup, but the batteries have finally gone flat this morning.  Grid power is expected to be restored by 10PM tonight.  I will be delivering a generator to the site later this morning though, if the access road is passable, to restore service sooner.

This outage affects some subscribers in Redwood Valley, Ridgewood Ranch, Western Willits, and Brooktrails.

Thanks,

– Jacob

Resolved – Power outage – N. Ukiah, Redwood Valley, Potter Valley, Calpella

This afternoon at a bit before 3PM we were able to deliver a generator to the affected mountaintop site and restore service for those subscribers that themselves had power.   As of ~7PM this evening grid power is also restored at the site and no further outages are expected. 

Thanks for your patience while we cut trees out of the way and slogged through the snow to get service restored (it was a fun outage to resolve).

– Jacob

Resolved (for now) – Power outage – N. Ukiah, Redwood Valley, Potter Valley, Calpella

After clearing a number of trees from a snowy mountain road, we were able to access the affected site and start a backup generator. PG&E power is still not restored and there isn’t yet an ETR being provided by PG&E. The generator/battery combo should hopefully keep things powered up until the morning when we can make another trip up to refuel if needed.

Thanks,

– Jacob

Power outage – N. Ukiah, Redwood Valley, Potter Valley, Calpella

​Last night at around 9:45PM PG&E power failed at one of our mountain top sites that serves many subscribers in Northern Ukiah, Redwood Valley, Potter Valley, Calpella, and a few in Willits.  The backup battery bank at this site was depleted at around 5:30 this morning and the equipment at the site shut down.  As of yet, PG&E does not have an estimated time of restoration for power in the area.   A convenient PG&E outage map is available here:   https://m.pge.com/#outages

Once it is light out, I will assess the snow situation at the site and if possible, deliver a generator to the site to restore service until PG&E power is restored.  The earliest I anticipate that happening is 10AM this morning.  Depending on snow conditions and downed trees across the road, service restoration may be later than estimated.

Thanks for your understanding.

– Jacob

Access point failure affecting some Redwood Valley subscribers

This evening one of the access points serving subscribers in Redwood Valley has failed.  The vast majority of subscribers affected by this failure have failed over gracefully to other equipment.  Only a few subscribers are currently without service.   We will replace the failed access point as soon as possible in the morning.

Other subscribers served by this same site may experience slower than normal service this evening with the added load placed on other equipment by this failure.

Thanks,

– Jacob

10-25-2020 PSPS info

This afternoon, PG&E is likely to start a PSPS event which will likely last until some time on Tuesday.  This event will affect subscribers in the Blue Lakes area. 

We have sufficient battery/generator runtime at most sites in the Blue Lakes area to get through this PSPS event without any service interruption to our subscribers that also have a means of backup power.​  We will be delivering some additional battery capacity to some of those sites this afternoon, in anticipation of the event.

There are a small number of subscribers around Lower Blue Lake for which service may be down during this PSPS event.

Please let us know if you have any questions.

Thanks,

– Jacob