Category Archives: System Status

Feliz Creek Road outage

This morning as of 6:20 a relay serving about a dozen subscribers on Feliz Creek Road in Hopland has gone down.  There seem to be battery problems at this site the last two mornings.  We are looking in to the root cause presently.
Thanks,
– Jacob

North coast outage – Resolved

At approximately 4:30PM today our link to Fort Bragg (and the rest of the North Coast) went down. The cause of the outage is unclear at this time and is possibly a fiber cut. There is also a power outage in the general location of the tower site, but our equipment is running on battery power so that is unlikely to be the cause.

More info to follow.

Thanks,

Jacob

===============================================

Update @ 6:23PM

The root cause of the current outage on the North Coast is still currently unknown.  Power is certainly out at the tower site but it is unknown if the equipment is running on battery still.  Requests to the fiber provider for updates on our circuit there have only resulted in opening of trouble tickets at this time.  PGE has no current estimate for power restoration. 

Due to current stormy weather and a steep, rutted tower access road, I have postponed an on-site trip until first light for safety reasons.  I anticipate arriving around 7AM barring fallen trees and the like on the access road that need to be moved.

For fun, you can see a video of the weather conditions at the site shortly before the outage occurred at the following link (caution, the wind noise is quite loud):

https://www.youtube.com/watch?v=Bh8ZWHOQxRU

More details to follow.

Thanks,

– Jacob

=================================================

Update @ 10:50PM

​With restoration of power to our Fort Bragg tower site, service is back up for subscribers on the North Coast.  Tomorrow I will investigate why backup power did not last as long as anticipated.

Thanks,

Jacob

Thanksgiving Holiday Hours

We will be closed on Thursday, November 28th for the Holiday.  Friday our office will not be staffed, but we will reply to support emails and voicemails as needed. 
Normal business hours will resume on Monday December 2.
Have a great holiday.
– Jacob

PGE PSPS Update – State of the network

Well in to the third full day without power I thought I’d send an email to update everyone on the state of things related to our network and answer a few common questions.
The vast majority of the network has been fully operational during this extended outage.  A large number of our sites are solar powered and thus not affected.  However, our head-end in Ukiah and a handful of critical macro sites that serve large geographical areas are grid powered.  These grid powered sites have been running continuously on generator and backup batter power for the duration of the outage and will continue to do so.  
Some smaller micro-relay sites that serve small number of subscribers have either been partially impacted by the power outage, or totally down during the power outage.  The vast majority of subscribers served from these micro-relays would not have noticed though, as they are also without an alternative power source.  The number of customers served by such sites that haven’t been continuously up, account for 13% of our total subscriber count.  Unfortunately there are a large number of these small sites and it is unrealistic for us to move generators and fuel around to all of them for such a wide-spread and extended outage.
Of the remaining 87% of customers, almost 50% have been online during the power outage at some point.  Either because they are already off-grid, or they have used a generator or other alternative power source to get online.
Areas currently affected by the power outage (on our end of things) include the following:
– Fort Bragg  to Westport – all subscribers – Redwood Valley – Uva Drive, North State Street, and surrounding area (11 subscribers) – Downtown Ukiah – Between Stanley and Clay – Laytonville – all subscribers – Old River Road – From Ruddick Cunningham South – McNab Ranch – Subscribers served from McNab Ridge Winery – Hopland – about 20 subscribers – Blue Lakes – about 25 subscribers east of Irvine Ave
Affected areas above will resume normal operation when power is restored in those locations.
It’s always my hope to provide the most reliable service to all subscribers.  These power shutoffs are a great test of plans that have been put in place, and expose many areas in which we can improve.   Some very positive infrastructure changes in our network will result from this experience.
I want to send a special thank you to those on Orr Springs road that jumped at the opportunity to help with keeping things afloat this weekend when a call went out. It is truly appreciated.
If you have any questions, please don’t hesitate to call or email.
Thanks,
– Jacob

Anticipated PGE outages – how this may impact your service

As of 8:45PM Monday evening PGE is anticipating disrupting power to large areas of Lake and Mendocino Counties beginning at 4AM on 10-9-2019.  Current PGE PSPS info is available here:
www.pge.com/en_US/safe… <www.pge.com/en_US/safety/emergency-preparedness/natural-disaster/wildfires/public-safety-event.page?WT.mc_id=Vanity_pspsupdates>
I have anticipated these possible outages and have portable generators at critical grid-powered sites to minimize any service disruptions to our subscribers, particularly those that serve public safety customers.  In addition to this, a majority of our sites are off-grid and aren’t affected by grid outages.   We do however, have a number of small relays that serve small groups of people that only have small battery backups.  These small relays will go offline in the event of an extended power outage.  
What I can’t anticipate unfortunately, is how these large-area power outages may affect the networks of upstream fiber providers that we lease connectivity from between Ukiah and the Bay Area.   I expect that their facilities should have adequate battery and generator runtime to get through the outages without issue. Unfortunately though, this is something we won’t know until we know.
During an extended power outage our office phones and computers will be offline.  We will however be available via email at *support@seakay.org* if you have any questions.
Thank You,
– Jacob

Friday morning software updates -McNab Ranch and Hopland

Friday morning beginning at 3AM a number of devices serving subscribers on McNab Ranch and in Hopland are scheduled to receive important software updates.  The various updates are scheduled to happen automatically between 3 and 4AM and should complete on their own.  Subscribers may experience a few brief service interruptions during this window while the updates take place.
Thanks,
– Jacob