Imagine you own a fleet management service that depends on real-time data for minute-by-minute operations. Suddenly, the network goes down and everything stops.
Increasingly, business owners are concerned about carrier outages–and for good reason. Network failure especially impacts industries in which Internet of Things (IoT) devices are integral to operational efficiencies. For these businesses, cellular connectivity is a cornerstone of reliable IoT functionality.
Let’s explore the AT&T network failure, its impact on businesses using connected devices, and how incorporating a dual-SIM cellular router can mitigate risk in the future.
What Caused the AT&T Outage
At 4:00 am EST on Thursday, February 22, 2024, a significant network outage hit the AT&T network, affecting customers across the country. Mobile phones and IoT hardware appeared to be equally impacted.
According to a statement from AT&T, the outage resulted from “incorrect process(es)” during network expansion. Put simply, it appears human error was to blame. While customers can take some comfort from the fact that security breaches were not a factor, the outage still had a paralyzing effect on business operations.
All in all, the network was down for over eight hours (between 4 am and approximately noon EST), underscoring the complexity of resolving network challenges on such a large scale. During that time, business owners and operators scrambled to find solutions.
The Business Impact
In the wake of the AT&T outage, a wide range of smart devices and monitoring equipment went offline.
Digital signs stopped projecting images. Security systems failed to report information or send alerts. Point-of-sale machines, ATMs, and other revenue-critical IoT devices were suddenly useless to owners, operators, and customers, who had to abandon transactions altogether or find offline solutions.
Businesses relying on cellular connectivity typically experienced three main challenges.
Prolonged Disruption
Those relying on connected equipment were unable to monitor or control those devices in real time, which led to further operational delays and financial losses.
Recovery Delays
IoT operators saw further delays as they worked to diagnose connectivity issues, strategize solutions, and implement potential workarounds.
Future Fears
Business stakeholders felt a growing concern about their single-carrier cellular solutions and discussed the need for more reliable connectivity strategies in the future.
Eight Hours Down: Counting the Costs
While it may not be possible to accurately tally all costs related to the AT&T outage, analysts estimate millions in lost revenue during the 8 hours of network downtime.
After the outage, AT&T reached out to “potentially impacted customers” with an offer to credit their accounts with the average cost of a full service day.
To illustrate, suppose every customer was impacted. Then, multiply a five-dollar credit by 75 million, the current total of AT&T customers - resulting in $357 million in potential lost revenue for AT&T alone.
Businesses hit hardest by the outage had their own losses to report. In particular, those most dependent on cellular connectivity (e.g., retail, education, transportation, transportation, emergency services, health care, and a wide array of others) reported millions in lost sales, productivity, and wages.
To further illustrate the potential cost, each year the retail sector calculates an average cost of $10 million from carrier outages, hinting at the depth and breadth of revenue losses across other industries.
The Inside Track
Because our OptConnect team regularly interacts with people across the cellular IoT landscape, we heard some compelling stories about potential losses, including:
- Up to $1,000 in lost revenue generation for POS systems, retail ATMs, Bitcoin ATMs, kiosks, and micro markets during 8 hours of downtime
- Loss in product from POS/credit-card terminal downtime, cash-only transactions, and lost access to nearby ATMs
- $150+ (per location) average truck roll costs to reboot hardware or check connections
- Up to 6 average work hours lost from increased customer support calls
Beyond the obvious costs, businesses wrestled with soft losses, including seeing customers turn to other vendors in the face of a poor experience or lack of a reliable solution.
Business owners also lost data and metrics since they couldn’t stay in touch with their connected equipment during 8+ hours of downtime.
A Proactive Response to the AT&T Outage
At OptConnect, we’re in the business of connecting critical equipment to the Internet. So any network outage, even minor, can impact businesses we serve. That’s why we’re also in the business of making cellular connectivity reliable no matter what.
Here’s how the AT&T outage played out for our team on February 22nd.
At 2:26 a.m. MST, our internal systems alerted us to a potential issue with AT&T. Immediately, our critical response team jumped into action, working with AT&T engineers to analyze alerts, check individual devices, and confirm the origin of the issue.
Because each of our devices automatically checks in with our servers every 30 minutes, we are instantly alerted if OptConnect systems recognize any anomalies in the expected number of devices checking in. At 2:48 am, our team recognized AT&T as the common denominator for missed check-ins and the root cause of the internal alert. At that time a ticket was opened with AT&T and we immediately began working with their engineers.
While our critical response team was taking action to see if starting any manual processes could restore connectivity, all OptConnect devices impacted by the outage autonomously began to attempt a reconnection on their own. OptConnect’s proprietary software on every one of our devices enables this autonomous behavior for self-healing.
When connectivity was not restored through self-healing and an automatic reboot of the cellular signal, our dual-SIM devices (e.g., the neo2) autonomously switched to the second carrier – in this case, Verizon. The result was automatic failover to the Verizon network. This entire process took no longer than two to three minutes and happened in the background with no manual effort involved or required by OptConnect customers.
By 6:40 am, the majority of OptConnect devices using AT&T as their primary carrier were able to switch back successfully.
While the network failure impacted all AT&T customers, our OptConnect dual-SIM devices quickly, safely, and automatically failed over to a secondary network, significantly boosting cost savings and uptime for critical infrastructure across the country.
A Case for Dual-SIM Technology
To an increasing number of connectivity-based businesses, dual-carrier redundancy is not merely a backup; it's an essential component of a resilient IoT ecosystem. Dual-SIM technology from OptConnect
Ensures continuous operation by providing an immediate alternative when one carrier faces an outage.
Enhances coverage, especially in areas where one carrier may experience a weak signal due to environmental or other conditions.
Optimizes costs by providing active dual-carrier SIMs for one low monthly rate. No site visits, physical SIM swaps, or costly manual efforts to restore connectivity.
Making the Switch
For customers using AT&T as their primary carrier, our OptConnect devices equipped with dual-SIM routers automatically switched from the faltering AT&T network to Verizon, ensuring uninterrupted service.
This autonomous switchover capability not only mitigated the immediate impacts of the outage but also proved the value of redundancy in network connectivity.
Dual-carrier failover is built into all of the latest proprietary OptConnect hardware, including our most popular cellular routers, mylo and neo2.
The mylo
OptConnect mylo is a dual-carrier, 4G LTE Category M1 router specifically designed for slower-speed applications such as ATMs, POS, irrigation, wastewater, and sensor applications.
Due to the special class of M1 LTE connectivity in which mylo operates, it works better than other classes inside of buildings and basements. It also works well farther away from cellular towers than other classes of LTE.
While older cellular technologies might be close enough to work with only one cell tower, mylo can see and communicate with several more distant towers, ensuring businesses are less affected by connectivity issues at a single tower.
During peak usage periods at concerts, sporting events, and other large gatherings when network congestion is high, the mylo keeps working when phones, tablets, and other products lose their connection. Cellular carriers automatically give mylo priority over other technologies. This means when everyone is trying to connect to the network and other machines are faltering, mylo keeps IoT equipment going strong.
The neo2
The OptConnect neo2 is a compact, LTE Category 4, dual-carrier cellular router perfectly suited for kiosks, micro markets, digital signs and hundreds of other applications that require high-speed connectivity and the peace of mind of redundant carrier connections.
Today, businesses are connecting more devices that need faster access to the cloud and to data. As connectivity to the cloud becomes more critical, downtime becomes more costly.
The neo2 adds connections, speed, and carrier redundancy to ensure greater uptime. It is a truly compact, full-featured, high-speed cellular router that fits almost anywhere, allowing users to more easily connect their equipment to cloud services.
As with all OptConnect-branded devices, dual SIMs in the neo2 offer peace of mind while fully managed wireless services by OptConnect make for a headache-free experience.
Building a Resilient Future, One Cellular IoT Device at a Time
The AT&T outage is not an isolated incident; it's a cautionary tale, highlighting the need for robustness in our connectivity infrastructure. The adoption of dual-SIM routers offers a path forward, ensuring that IoT operations can withstand the vicissitudes of network reliability.
Speaking about the outage, the CEO of AT&T stated, “This is not our first network outage, and it won’t be our last – unfortunately, it’s the reality of our business.”
Recognizing that network outages are not totally unavoidable, the goal of every solution provider should be to use a solution that is as reliable as possible. That is where the value of dual-SIM devices shines.
As we navigate the aftermath of the AT&T outage, let's reflect on the lessons learned and the pathways opened by technological advancements like dual-SIM routers. In a world where connectivity is akin to oxygen for IoT operations, ensuring a steady supply is not just prudent—it's imperative.
To learn more about upgrading your single-SIM device portfolio to more robust dual-SIM devices, contact an OptConnect expert today.
Want to prevent downtime during outages?
Talk to a solution engineer about dual-SIM connectivity options.