A network issue at our Nevada City office over the weekend took our Barracuda spam filtering server offline. Symptoms of the problem were that Spiral customers could log in to their mailboxes, but could not receive any new e-mail. This affected nearly all customers — including nccn.net addresses, spiralemail.com addresses, sierraemail.com addresses, and most hosted domains.
Unfortunately, that same network issue also caused our voicemail system to fail over to our backup — which was configured to e-mail us call recordings. This prevented us from seeing customer reports of the problem, and our backup voicemail service quickly ran out of voicemail space. Finally, the administrator on call whose monitoring would have caught the issue was unexpectedly out of town for family reasons. We’re sorry for our unresponsiveness. It was an unusual chain of failures escalating a minor problem into an unnecessary outage, and that’s not the standard we try to set.
The network issue has been addressed. Email delivery has resumed as of approximately 10 am Monday, March 11, and Spiral customers should begin receiving the weekend’s backlog over the next several hours.
As the first fierce storms of the season pass through the foothills, they bring with them high winds and downed tree branches that damage infrastructure. We’re seeing the results of that today (Nov. 30), as several customers in the Nevada City area have reported that both their DSL and landline telephone service are out. This has been confirmed as part of a local AT&T outage.
We do not yet have an ETR for the outage; however, customers with both telephone and DSL issues can call AT&T for information on repair estimates for telephone service, at which time DSL will almost certainly be restored as well.
If your DSL service and telephone are both down, please be patient as the outage is addressed. If you are having DSL issues but no landline phone issues, give us a call at (530) 478-9822 and we can go through our normal troubleshooting to help you address the problem.
Due to an unforeseen disk issue, the mail server which handles NCCN.net e-mail addresses will be taken offline after 5 p.m. Monday, November 12 in order to safely transfer all user data to another drive. This will prevent users from logging in (via mail clients or our webmail interface) for the duration of the maintenance window — any attempts will give an error message that the server is unreachable.
This ONLY affects customers with e-mail addresses ending in @nccn.net; other domains are unaffected.
Inbound mail will be accepted by our Barracuda spam filtering server, held, and queued for delivery after the maintenance window passes. As such, we do not anticipate losing or bouncing any e-mail, but if you do not receive expected letters within 8 hours after the maintenance window ends, call our office at (530) 478-9822 and our technical support can help you.
Estimated maintenance window is from 5 p.m. 11/12 to to 1 a.m. 11/13. We will update this post as better estimates become available.
Update, 11 p.m.: At the current rate of data transfer, service will be restored around 6 a.m.
Update, 7 a.m. 11/13: The data transfer has been verified, and mail service has been restarted and tested. Mail from the maintenance window has not yet been delivered. Our Barracuda spam filtering server has a backlog of several thousand messages, which will be delivered over the next several hours.
Update, 5 p.m. 11/13: The mail backlog was delivered by noon, and after monitoring the system all day we are resolving this maintenance incident.
As of Sunday morning, Feb. 18, most or all of our DSL customers are out of service. Symptoms include the “Broadband”/”DSL” light on the modem being solid green as if the line was active, but the “Internet”/”Service” light on the modem being red, and no data getting through to the Internet. We have reported the issue to our upstream provider and are attempting to learn further details.
DSL customers using competing providers such as AT&T also appear to be affected, perhaps throughout Nevada County. Social media reports suggest that AT&T is reporting a cut fiber line impacting Placer, Nevada and Sierra counties; we have not been able to directly confirm this with AT&T yet.
We do not currently have an Estimated Time for Repair (ETR). However, historically, issues of this magnitude are typically repaired within 48 hours, often same-day. This post will be updated as we learn more.
Update, 12:45pm: We have confirmed that customers are affected as far south as Lake of the Pines and as far north as Sierra City. Alta Sierra, Penn Valley, Grass Valley, Nevada City, and North San Juan are of course within that outage zone. Our Auburn customers appear unaffected. Still no word from AT&T on an ETR.
Update, 9pm: Our affected customers appear to be back online within the past hour or two.
Update, 2/19: All outage issues are resolved. Please call our office at (530) 478-9822 if you are still experiencing trouble.
As of this afternoon (Wednesday, Dec. 20), we are receiving reports that calling our main office number (530-478-9822) results in a busy signal. We have confirmed the issue and are working on the problem with our telephone provider.
In the meantime, local customers can also visit our office at 416 Broad St., Nevada City, for service/billing issues or sales inquiries, or simply go to our web contact form to let us know your issue. If you provide a contact number there, we will call you back promptly.
Edit, 2:40 PM: As the issue appears to be network-dependent (T-Mobile and some VOIP numbers can call us, but not Verizon or AT&T), it has been escalated to our upstream carrier. We do not currently have an ETR. In the meantime, please use our web form to contact us, as above.
At about 4:30 p.m. Wednesday, Dec. 13, a routine reboot of one of our two DNS servers caused the server to lock up. We are actively working on restoring the server.
In the meantime, Spiral customers may notice some slowdowns in Internet usage. DNS servers perform the function of translating domain names (like spiral.com) into the routing numbers that allow your computer to speak to those domains. Requests to the downed server will fall back to the working server, but may result in delays in that lookup process.
Edit, 5:40 p.m.: Full functionality has been restored.
Today, Nov. 6, we’ve been getting scattered customer reports of difficulties accessing Spiral services. We have been impacted internally as well, with poor quality on our Voice Over IP phone lines and reduced connectivity to our Barracuda spam filtering server for inbound e-mail.
Today’s connectivity issues have been linked to nationwide outage reports by Internet backbone company Level3:
Mostly, the outages appeared to affect customers using Comcast cable internet, as Level3 is a primary provider of their backhaul. However, due to the distributed nature of cloud computing services and Level3’s role as a major Internet fiber backbone, some issues (such as ours) affected even non-Comcast customers.
As of approximately noon PST, Nov. 6, most of the issues appeared to be resolved at Level3’s end, and all of our servers are operating normally. We will continue to monitor the situation.
There has been a major stir this week with the announcement of the “KRACK Attacks” — an exploit against the WPA2 standard which secures the connections to virtually all wireless networks worldwide. As this affects everyone who uses Wi-Fi for their Internet connection, including most of our customers, we have been looking into the issue.
The first rule, for now, is: don’t panic. While the vulnerability will affect you, so far no malicious code has been released, and vendors are already working on patches to prevent future abuse. The good news is that WPA2 itself remains secure — there’s just a trick which can force it to reuse encryption keys in ways that an attacker can exploit. Both computer vendors and router vendors are at work on patches that prevent that reuse; if either the router or the connected computer is patched, the connection is safe.
The most important thing you can do in order to keep your Wi-Fi secure is to install the security updates put out for the operating system of your computers/phones/tablets. (This is important because, even if you patch your own router, you want to make certain your devices are safe on other Wi-Fi networks.) Microsoft has already released their patch. Apple and Google are working on theirs and a fix should be released within a week or two.
Patching your wireless router is also a best practice moving forward — although the process of applying firmware updates from your router manufacturer is more difficult than just updating your computer’s OS. The biggest benefit to updating your router is that it will provide protection to “Internet of Things” devices which may not receive security updates; if you are only using computers/phones/tablets which are patched, router patches will provide almost no additional protection.
Spiral does not sell standalone wireless routers. However, some DSL modems that we sell do include wireless capabilities. We are currently speaking with our vendors about firmware updates for those modems, and if Spiral sold you a wireless modem/router, we will be happy to assist you with upgrading it at the time that those patches become available. Please call our office at (530) 478-9822, Ext. 1 for technical support, if you would like help with that process.
Due to a network connectivity issue at our Nevada City office, as of 11 AM Tuesday, Sept. 20, we are currently experiencing problems with inbound email for all Spiral customers. Incoming messages are unable to reach our Barracuda spam filtering system, and therefore are not being delivered to customer mailboxes.
E-mail networks are designed to retry messages which failed to send, so at the time connectivity is restored, most or all of those backlogged messages should be retried and correctly delivered. Some messages (such as password reset emails) should be retried/resent when service is restored.
The network issue is also interfering with our ability to answer phone calls to our (530) 478-9822 office number. Our voicemail system is working and we will respond to messages as soon as possible.
Our upstream provider has noted this as part of a regional outage and given us an ETA of 5 PM for repair.
Update, 1:30 PM: The network is currently up and e-mail messages are passing through, but as we have already seen service return and then drop out again, we should still be presumed to be under potential outage conditions.
Update, 3:30 PM: As the network has been stable for several hours, we are returning to presumptively normal operation.
Early on Tuesday, Sept. 5, a compromised website on our shared hosting server began sending out high levels of spam which caused that server to be temporarily listed on e-mail spam blacklists. The compromised website has been re-secured and we are in the process of removing the server from the blacklists. In the meantime, outbound mail may be disrupted for customers with SierraEmail.com, SpiralEmail.com, or custom domain e-mails.
Symptoms include messages not arriving at their destination (due to being queued for later sending after the blacklisting resolves), or sent mail being returned with an error stating that the message could not be delivered.
We anticipate full service being restored by the morning of Wednesday, Sept. 6. In the meantime, failed e-mails can be resent from alternate addresses.
Our apologies for the inconvenience. If you have further concerns, please give us a call at (530) 478-9822, Ext. 1 for customer support.
Edit, 1:00 PM Sept. 6: The Spamhaus/CBL blacklisting which caused the most problems has been removed, which should address most spam-blocks by the end of the day.