Massive outage at Rogers impacted Canadian phones, internet, ATMs and debit cards

Canadian telecommunications company Rogers is experiencing a major outage that started early this morning affecting landlines, cellular and internet connections across Canada. Down detector listed thousands of reports of the issues pouring in when people started getting up around 5AM ET and unable to get online.

Rogers first tackled the malfunction in a tweet from his official support account just before 9AM ET and then went silent for a few hours. On Friday afternoon the company tweeted that its technical teams are working to restore services “along with our global technology partners, and make progress.”

Late Friday Night, Rogers CEO & President Tony Staffieri posted a letter to the Canadians on the company website.

Dear Canadians,

We know you count on Rogers to connect you to emergency services, make payments, serve your customers, keep in touch with your work and keep in touch with friends and family. We take that responsibility very seriously and today we let you down. We can and will do better.

As you know, we experienced a network outage on both wireless and landline that started early this morning.

We have made significant progress in bringing our networks back online and many of our wireless customers are starting to see services return. We do not yet have an ETA on when our networks will be fully restored, but we will continue to share information with our customers as we restore full service.

We know that a full day without connectivity has a major impact on our customers and all Canadians. On behalf of all of us here at Rogers, Rogers for Business, Fido, chatr and cityfone, I would like to offer my sincere apologies for this service interruption and the impact it is having on people from coast to coast to coast.

As our teams continue to work to resolve the situation, I want to make two commitments:

First, we are working to fully understand the cause of this outage and will make any necessary changes to ensure we meet and exceed your expectations for our networks in the future.

Second, we will make it right for our valued customers. We will proactively apply credit to all our customers affected by the outage and will share more details shortly.

I take full responsibility for ensuring that we at Rogers regain your complete trust and am once again here to connect you to what matters.

Honestly,

Tony Staffieric

President and CEO

There is still no ETA for full restoration. In another tweetthe company promised it would “proactively credit all customers”, saying it had put “all technical resources and partners” in place to get its network back up to date.

The CBC reported Friday evening that people are starting to see their home internet and wireless services again, while Cloudflare Radar data shows that traffic is starting to flow to and from the Rogers network again, although still not near usual levels.

The data of cloudflare, a content delivery network and DDoS mitigation company that helps many high-traffic sites manage their operations, shows communication between its load-sharing network and Rogers in Canada exhibiting its normal nightly pattern before falling to near zero at about 3–4AM ET falls and has since registered a flat line. At the same time, there was a massive spike in Border Gateway Protocol (BGP) announcements for the network, indicating changes in routing.

In a blog post published on Friday afternoonCloudflare staff speculated that the outage is the result of an internal error at Rogers, as opposed to a cyberattack or some other cause. Citing that pattern of BGP announcements, they noted that with the Rogers network not announcing its presence, the rest of the internet can’t find it. BGP is a fundamental part of the technology that helps move information from one place on the Internet to another, and a problem with BGP routing information caused a massive outage of the Facebook network last fall.

The malfunction has also disconnected Interacthe network Canadian banks use, disabling debit cards, ATMs and the e-transfer services that recently carved over 1 billion transactions in one year.

Passport offices and Canada’s Tax Collection Agency are among the government services unavailable today due to the outage. Both agencies also warned users that the outage will cut off multi-factor authentication codes sent by voice or text message, so people who are logged out may not be able to log in at this time.

Even the Canadian Radio-Television and Telecommunications Commission (CRTC), which regulates broadcasting and telecom, says his phones don’t work due to the malfunction.

Internet network disruption tracker NetBlocks shared real-time data showing that Canada’s national connectivity has fallen to just 75 percent of normal levels.

A report from CBC News noted that Rogers daughter Fido also has issues. The article says issues have impacted card payment processing and ATMs across Canada. A CBC radio station in Kitchener, Ontario is also offline, and some telephone services for Ottawa’s transit agency are disabled. Photos from Toronto shows people crammed into coffee shops or going to the library to access wifi now that their other options are offline.

TekSavvy Customer Service has told customers the outage extends to “Ontario, Quebec and the Eastern Provinces”, with no ETA for resolution, while the CBC reports that Thunder Bay operator Tbaytel says the Rogers outage is nationwide.

Toronto Police Department tweeted early this morning after some people in the city experienced connectivity issues calling 911. “The Rogers Network is experiencing some technical difficulties,” according to the Toronto Police Operations Twitter account. “We are working to resolve these issues.” In a continued tweetPolice confirmed that their 911 center is operational and advised anyone who needs to call to stay on the line if they connect and call again if they don’t.

Update 9:52 AM ET: Added information about the Bank of Montreal

Update 11:20 AM ET: Persistent outage noted, added information from Cloudflare Radar, TekSavvy and Tbaytel

Update 11:33 AM ET: Added Rogers statement

Update 12:40 PM ET: Added last update of NetBlocks

Update 13:31 ET: Added additional information about Interac’s banking network, — thanks, mbarriault.

Update 14:13 ET: Added information about CRTC, passport offices and tax outages.

Update 3:39 PM ET: Added Cloudflare comments about BGP as a possible cause of the outage, as well as Rogers’ statement that it is “making progress” in resolving the issue.

Update 11:47 PM ET: Added statement from Rogers CEO and note that services are slowly coming back online.