How VTel’s phone/internet outage happened

By Shawn Cunningham
© 2020 Telegraph Publishing LLC

Multiple equipment failures brought to a standstill VTel phone and internet service to Chester, Andover, Grafton and Windham — first on Thursday, then on Friday and into Saturday of last week, prompting questions about access to public safety agencies and how to make that more robust.

VTel engineers and technicians worked from Thursday morning at VTel’s building in Chester to diagnose and fix the outage. Photos by Shawn Cunnningham

According to a detailed timeline of the “network event” provided to The Telegraph by VTel, the issues began on Thursday midday when technicians became aware of a problem communicating with Chester, which at first looked like a cut somewhere in the system’s fiber optic cables. None were found, but it appeared to technicians in Springfield that traffic was not passing through a router in Chester.

Within 15 minutes, according to the timeline, two technicians and two engineers arrived and began diagnosing the cause of the outage. That work was hampered — ironically — by the lack of internet service to verify equipment settings. An hour or so of working on getting the router to respond resulted in finding that the device’s configuration was blank.

Photo by Markus Spiske from Pexels

VTel Chief Technology Officer Arianna Robinson noted that experience has shown that uploading a new configuration is more effective if done in batches, so engineers began the hour-long task of uploading 3,905 lines of code. By late Thursday afternoon, the controller cards in the router rebooted and began to provide service. After taking care of a few other problems, service was back around 5 p.m.

But, early Friday morning the router again failed and, after troubleshooting the problem, the installation of a new router was begun. At the same time, engineers tried to bring the old router back online whenever it crashed to provide some service.

With new hardware installed, “multiple engineers” working onsite and remotely, migrated services to the new router.  This was done slowly as reprogramming was also being done at the same time. According to Robinson, the process took until the early evening when another crash occurred and the process began over again with engineers resolving problems as they went. After the situation seemed to be resolved and had been watched for an hour, the crews headed home around 11 p.m. Friday.

On Saturday engineers did some de-bugging to solve some left-over problems with video and internet that a few customers were having and service was returned to normal.

Phone company addresses public safety concerns

The lack of phone service meant that customers within the affected area were unable to access 911 services, a situation that has been a topic of conversation among many customers who wonder how a repeat of the outage could be avoided.

“This outage was an eye-opening experience, and one that has led to plans to ensure we don’t see a repeat of this widespread occurrence,” said Amber Wilson, VTel’s director of Customer Care, who told The Telegraph that one example of changes under development includes a division of functions within VTel’s network design.

Wilson explained that the company is looking to separate larger processes that occur on its network on a daily basis into more compartmentalized pieces. “That way, any failure or errors on one component will affect fewer customers and lead to quicker restoration.”

“This and other system changes are critical for our network,” said Wilson, who noted that while VTel does not have a reputation for outages, if another unexpected outage were to occur for any reason, the company’s  goal will be to deliver a much quicker, much smoother recovery. “The reliability of our network has always been a priority and will continue to be a priority, during the pandemic and beyond,” she said, adding that that VTel’s commitment to customers “has never been more important as the world’s needs have seemingly changed overnight.”

 

 

 

 

Print Friendly, PDF & Email

Filed Under: AndoverChesterFeaturedGraftonLatest NewsWindham

About the Author:

RSSComments (9)

Leave a Reply | Trackback URL

  1. L Bratton says:

    Copper wire carries enough residual power for home phones to function during a power outage. Fiber optic cable does not have this capability. This needs to be made clear by telephone companies.

  2. Tom Knockenhauer says:

    Everyone’s worrying about a 3 day rebate. How about all the kids going to school using zoom. And not being able to use it. Shame on people’s petty thoughts. Try and remember the children.

  3. Lee Herrington says:

    I agree with Chris’ above comment. We not only lost our internet and Vtel landline but for a while our cell phones also went down as it appears the closest cell tower depends on a Vtel network connection. Vtel needs to do better as a simple equipment chokepoint needs to have redundancies to ensure emergency communication to a region is maintained. What if this happened during a major storm or during an emergency at one of our schools?

  4. Christopher Wallace says:

    I wonder if some rural customers would not be better served by copper or, better yet, copper redundancy for home phone service? In addition to tech crashes, extended power outages can also result in days without phone.

  5. Craig Miller says:

    In addition to the redundancy two other questions come to mind:

    1 – Configurations in high end networking gear do not just “Go Blank” without some sort of human interaction. And access to this layer should be logged by various security monitoring systems.

    2 – Why did they have to restore 3905 lines of codes in batches. Again high end networking gear has the ability to save configurations for easy restore capabilities. Why was there no backup of the Router configuration?

    This is SOP for network jocks and IT folks.

  6. Phil Perlah says:

    Beyond the technical issues, V-Tel was very poor from a customer relations pov. First, there was no communication when the systems were up and running. So, we had to constantly check the phone for a dial tone. Second, they have no plans to voluntarily credit accounts for the 3 days without service.

    Call them and ask for a 3 day credit on all services!!!

  7. J Westclark says:

    This does not satisfactorily answer why there wasn’t redundancy in place to avoid this single point-of-failure that affected a critical service for so many. But it does hint that will be addressed.

  8. Cynthia Prairie says:

    It may have been two separate incidents. But we will ask! Thanks for the comment!

  9. Philip Bravin says:

    what i do not understand was why did it impact ATT service here as well…this really cut off all possible communication…if that depended on that same router, then its bad telecom design..also was there redundancy built in