This morning as of 6:20 a relay serving about a dozen subscribers on Feliz Creek Road in Hopland has gone down. There seem to be battery problems at this site the last two mornings. We are looking in to the root cause presently.
At approximately 4:30PM today our link to Fort Bragg (and the rest of the North Coast) went down. The cause of the outage is unclear at this time and is possibly a fiber cut. There is also a power outage in the general location of the tower site, but our equipment is running on battery power so that is unlikely to be the cause.
More info to follow.
Update @ 6:23PM
The root cause of the current outage on the North Coast is still currently unknown. Power is certainly out at the tower site but it is unknown if the equipment is running on battery still. Requests to the fiber provider for updates on our circuit there have only resulted in opening of trouble tickets at this time. PGE has no current estimate for power restoration.
Due to current stormy weather and a steep, rutted tower access road, I have postponed an on-site trip until first light for safety reasons. I anticipate arriving around 7AM barring fallen trees and the like on the access road that need to be moved.
For fun, you can see a video of the weather conditions at the site shortly before the outage occurred at the following link (caution, the wind noise is quite loud):
More details to follow.
Update @ 10:50PM
With restoration of power to our Fort Bragg tower site, service is back up for subscribers on the North Coast. Tomorrow I will investigate why backup power did not last as long as anticipated.
We will be closed on Thursday, November 28th for the Holiday. Friday our office will not be staffed, but we will reply to support emails and voicemails as needed.
Normal business hours will resume on Monday December 2.
Have a great holiday.
If you have received your Mendocino County property tax bill and are wondering what type of internet service you receive from us, the correct answer is “*Fixed Wireless*”. We *do not* provide Satellite based internet services.
Please let us know if you have any questions.
Later this morning I’ll be replacing some equipment that serves some subscribers along Feliz Creek Road outside of Hopland. I expect customer impact to be less than 15 minutes and that maintenance will start at around 10:30AM.
Well in to the third full day without power I thought I’d send an email to update everyone on the state of things related to our network and answer a few common questions.
The vast majority of the network has been fully operational during this extended outage. A large number of our sites are solar powered and thus not affected. However, our head-end in Ukiah and a handful of critical macro sites that serve large geographical areas are grid powered. These grid powered sites have been running continuously on generator and backup batter power for the duration of the outage and will continue to do so.
Some smaller micro-relay sites that serve small number of subscribers have either been partially impacted by the power outage, or totally down during the power outage. The vast majority of subscribers served from these micro-relays would not have noticed though, as they are also without an alternative power source. The number of customers served by such sites that haven’t been continuously up, account for 13% of our total subscriber count. Unfortunately there are a large number of these small sites and it is unrealistic for us to move generators and fuel around to all of them for such a wide-spread and extended outage.
Of the remaining 87% of customers, almost 50% have been online during the power outage at some point. Either because they are already off-grid, or they have used a generator or other alternative power source to get online.
Areas currently affected by the power outage (on our end of things) include the following:
– Fort Bragg to Westport – all subscribers – Redwood Valley – Uva Drive, North State Street, and surrounding area (11 subscribers) – Downtown Ukiah – Between Stanley and Clay – Laytonville – all subscribers – Old River Road – From Ruddick Cunningham South – McNab Ranch – Subscribers served from McNab Ridge Winery – Hopland – about 20 subscribers – Blue Lakes – about 25 subscribers east of Irvine Ave
Affected areas above will resume normal operation when power is restored in those locations.
It’s always my hope to provide the most reliable service to all subscribers. These power shutoffs are a great test of plans that have been put in place, and expose many areas in which we can improve. Some very positive infrastructure changes in our network will result from this experience.
I want to send a special thank you to those on Orr Springs road that jumped at the opportunity to help with keeping things afloat this weekend when a call went out. It is truly appreciated.
If you have any questions, please don’t hesitate to call or email.
As of 8:45PM Monday evening PGE is anticipating disrupting power to large areas of Lake and Mendocino Counties beginning at 4AM on 10-9-2019. Current PGE PSPS info is available here:
I have anticipated these possible outages and have portable generators at critical grid-powered sites to minimize any service disruptions to our subscribers, particularly those that serve public safety customers. In addition to this, a majority of our sites are off-grid and aren’t affected by grid outages. We do however, have a number of small relays that serve small groups of people that only have small battery backups. These small relays will go offline in the event of an extended power outage.
What I can’t anticipate unfortunately, is how these large-area power outages may affect the networks of upstream fiber providers that we lease connectivity from between Ukiah and the Bay Area. I expect that their facilities should have adequate battery and generator runtime to get through the outages without issue. Unfortunately though, this is something we won’t know until we know.
During an extended power outage our office phones and computers will be offline. We will however be available via email at *firstname.lastname@example.org* if you have any questions.
This morning around 6AM our relay on Moose Road went offline for unknown reasons. I’ll be on-site around 8:30 to diagnose and repair.
Thanks for your understanding.
Friday morning beginning at 3AM a number of devices serving subscribers on McNab Ranch and in Hopland are scheduled to receive important software updates. The various updates are scheduled to happen automatically between 3 and 4AM and should complete on their own. Subscribers may experience a few brief service interruptions during this window while the updates take place.
As of 11:15 this morning I have implemented temporary repairs to remedy the damage caused by the fire that started on Moose Road yesterday. All affected subscribers should have service restored as of that time.