Major Internet-wide security hole

A critical security vulnerability was announced today, affecting a system shell used by the vast majority of Internet servers.  The vulnerability allows remote hackers to run code on a system without authorization (somewhat like having a virus, but without any malicious code being installed).  We are taking this very seriously and are in the process of applying security patches to all Spiral services.

According to many reports, Mac OS X is also vulnerable — Macintosh users need to keep an eye open for security updates from Apple, and install them immediately once a patch is released.

Even Android smartphones may require a system update to address this, though it’s not known yet the extent of that vulnerability.

We will update this post as systems are secured and more information about Macintosh patches is released.


 

Update, 9/25: All Spiral servers were secured last night before the first exploits were published, though a smaller secondary vulnerability was identified and we are waiting for further official patches before declaring the issue fully resolved.

The primary concern for Macintosh users would be if you are running external services on your machine, such as a web server.  Although some websites are suggesting an immediate recompile of the Mac OS X bash binary, it is likely that the average Mac user should wait for an official patch from Apple rather than risk damaging their own system.

Best available information is that smartphones are not vulnerable out of the box, but specific applications may install or use the bash shell that’s at the root of the problem.

The vulnerability does not appear to exist in the Pace DSL modems that we sell, but we are currently working to confirm that.

Posted in Uncategorized | Comments Off

Outbound e-mail interruption

This afternoon, as we completed the process of decommissioning our local name server, a misconfiguration in our NCCN.net e-mail server caused outbound messages to briefly fail and inbound messages to be halted and queued for later delivery. The issue has been fixed; outbound messages are being correctly sent as of approx. 5:15 p.m., and the spam filtering system is currently working on processing the queue of inbound messages.

All other aspects of the DNS transition went smoothly. If you have DSL or Fusion service with a static IP address and have not yet updated your nameservers, click here for the support article.

Posted in unplanned outage | Comments Off

DNS cloud transition

As part of our goal to offer better Internet access and services, we are transitioning our DNS services to “the cloud” — virtual servers located on the Internet backbone.  DNS is the service that translates domain names (such as spiral.com) into the numeric addresses used to send traffic back and forth, so this will make your Spiral internet service more responsive and reliable.

In most cases, you will not need to make any changes.  However, if you have DSL or Fusion service with us, and one of the following applies:

  • You are paying for a static IP address
  • You have manually configured your modem’s DNS settings

Then read on for information on updating your DNS servers.

Please feel free to call our office for technical support (530-478-9822 in Nevada County, 916-252-1619 Sacramento), if you have questions or would like help with these settings.

New Server IPs

Our new cloud DNS servers are:
198.61.239.165
204.232.206.25

On September 16, 2014, we will be decommissioning our old (Nevada City) DNS server at the IP address 12.165.58.2.  If that IP is in your nameserver list, it must be replaced by Sept. 16.

Changing Pace modems

If you bought a modem directly from us that looks like this:

Pace 4111N

Then here are step-by-step instructions to change the DNS server IPs.  For older modems or third-party modems, please call our office for support.

  1. Open this page on a computer that connects to the Internet through your Pace modem, then click on this link: http://192.168.1.254
  2. You are now looking at your Pace modem’s configuration page.  Click on the speedometer icon next to the “Broadband” information at the top.
    Pace main page
  3. Click “Link Configuration” once the Broadband page comes up.
    Pace Broadband
  4. Scroll down the page to the “Broadband IP Network” heading, and find the DNS section.

    If “Obtain DNS information automatically” is selected, STOP.  You’re done! You do not need to make any changes.

    If “Manually specify DNS information” is selected, find and select the entry that says “12.165.58.2″ (this should be the Primary Server).  Your screen should look like this (the “IP Addressing” settings will be different; do NOT change those):
    Broadband Connex

  5. Enter the new server IP, 198.61.239.165.
    Screen Shot 2014-09-03 at 11.55.30 AM
  6. Scroll to the bottom of the page and click the “Save” button.

That’s it, you’re done!

Posted in planned maintenance, service update | Comments Off

Heartbleed

An extremely serious vulnerability, the “Heartbleed” bug, has just been announced for the server software that runs the vast majority of secured Internet connections.

The good news is that none of the servers hosting Spiral customer content are running a version of the OpenSSL software that is vulnerable to this bug.

If you have concerns about the security of your e-mail, website, or Spiral-hosted data, please feel free to call our office at (530) 478-9822, Ext. 1 for tech support.

Posted in service update | Comments Off

Brief e-mail/hosting outage

Due to a power outage in Nevada City (and our Nevada City office) affecting network connectivity, most of our e-mail and web hosting services suffered approximately 15 minutes of downtime early Sunday morning, March 2.

Services have been fully restored.

Posted in unplanned outage | Comments Off

Pace modem firmware upgrade

We have been working with our DSL modem vendor, Pace, to identify and resolve an issue where high network traffic is causing difficulties with customer Internet access.  This issue has been tracked to a vulnerability in the modem firmware, leading to attacks where a large volume of spoofed traffic is sent to Pace modems in order to contribute to a denial of service attack against a third party.

We have coordinated with Pace to push out a remote firmware upgrade to all our deployed Pace modems by the end of the business day.  This upgrade requires no action on your part.  No downtime is anticipated and no modem reboot is required.  However, if you begin experiencing any difficulties with a Pace 4111N modem you purchased from Spiral, call our office at (530) 478-9822 or (916) 252-1619 and press Ext. 1 for technical support.

EDIT: After further conversation with Pace we will push the upgrade on Monday, Jan. 27.

However, if you are currently experiencing slow/erratic Internet issues and are using a Pace 4111N modem you purchased from Spiral, you can manually install the patch before the vendor push with the following instructions.  This is not necessary — it will simply allow you to install the fix early if you would prefer not to wait through the weekend.

1) Sit down at a computer that is using the Pace 4111N modem to get through to the Internet.

2) Download the patch file at this link: http://spiral.com/pacehotfix* If you right-click that link and choose “Save As…”/”Download Linked File As…”/”Save File To…” you can choose where on your computer to save it.  If you simply click that link it should go to your normal Downloads folder.

3) Go to this address: http://192.168.1.254/upgrade

4) Click “Upgrade.”

5) Click “Choose File”, and select the file named “pace-ntpd-hotfix.install.pkgstream” that you downloaded in Step 1.

6) Click “Upgrade.”  Wait for approximately 5 minutes while the modem installs the patch and reboots.

7) When the power light goes back to solid green and your DSL/Internet lights come back on, you are ready to get back on the Internet!


* The file’s md5 hash is e8f2860c5091b6e840b67340b96e4502.  If this means nothing to you, don’t worry; it’s a security verification.

Posted in planned maintenance, service update | Comments Off

NCCN mail server down (restored)

As of Tuesday, November 26, the incoming and outgoing e-mail server for nccn.net e-mail addresses is currently offline due to a disk failure. We are in the process of restoring from backup and hope to have that server working again by the end of the day.

Edit, Weds. 11/27: The process of restoring from backup is unfortunately slower than anticipated but the server should be live approx. 4. pm today, 11/27.

Edit, Weds. 11/27: Service was restored as of approx. 4:20 p.m. and we have been monitoring the server since in order to ensure smooth recovery.

Posted in unplanned outage | Comments Off

Services down

As of approximately 4 p.m. Thursday 9/12, most Spiral web hosting and e-mail services are temporarily inaccessible. There appears to be an issue with the T1 circuit serving our internal and hosted servers, and we are in the process of tracking down the problem with AT&T (the telecom over whose lines the T1 passes). Updates will be posted here.

5:15 p.m.: Our servers do appear to be currently reachable from the outside world, and broadly responsive; however, some traffic is still not passing through correctly.  Testing continues to determine the extent of what’s affected.

6 p.m.: Wider problems have returned and efforts to isolate the source continue.

7 p.m.: Timeline for completion of AT&T testing is approx. 7:30 p.m. At this time, no traffic is getting in or out of our network and thus all hosted services and e-mail are once again down.

9 p.m.: Services were restored approx. 7:30 p.m. and we are verifying the fix.  There will be a short final outage later tonight (anticipated 15 mins. or less) as we deploy new rules on our primary firewall to address the new circumstances.

10:15 p.m.: Normal service has been restored and we will continue to monitor our servers in case of further issues.

Posted in unplanned outage | Comments Off

Mail server outage 9/3

Some lingering power issues stemming from yesterday’s lightning strikes caused one of our battery backup units to drop out of service today, resulting in a few of our internal servers being temporarily out of service.

This primarily meant that phone calls did not reach our office for an hour or two on Tuesday afternoon, 9/3, but one of the affected servers handled inbound and outbound e-mail for our NCCN.net customers.

Phone service has been restored as of 4 p.m. and we are currently bringing our mail server back online now that the power issues have been addressed.

Edit: As of 4:30 p.m. mail service has been restored.

Posted in unplanned outage | Comments Off

Power outage

Nevada City lost power at approx. 5:15 p.m. today 7/3 — right before the big July 4 weekend. We kept Spiral’s servers running (with just a few hiccups), and are going to double-check them before heading home.

This is a great time to remind you to check your DSL modems and computers: power outages can lead to surges that damage electronics. Make sure you have surge protectors or battery backup units on your valuable devices!

Posted in tips and advice, unplanned outage | Comments Off