Building a Resilient Network Across Carriers, Datacenters & Cloud
From a Single Internet Connection to BGP, Disaster Recovery and Hybrid Cloud
BGP • Cisco ASR/C8300 • Dual ISP • Palo Alto • IPsec • Public Addressing • Azure • BYOIP • DR • Colocation
When I joined Mobile Heartbeat in 2013, the company was a small startup. Internet connectivity consisted of a single small-business connection. That was appropriate for an early-stage company, but as the business grew — and as infrastructure became increasingly important to customer-facing services — the network had to evolve with it.
Over the following years, I progressively redesigned the environment through several generations of architecture. What began as a single Internet connection eventually grew into a resilient network spanning multiple carriers, disaster-recovery infrastructure, public BGP, private cellular connectivity, company-owned address space, colocation and Microsoft Azure.
The First Steps
The earliest improvements were straightforward. Business-class Internet service provided better support and static public addressing. As the environment grew, however, Internet connectivity became increasingly important to both normal operations and disaster recovery.
The major transition came after the company was acquired in 2016. We now had both greater availability requirements and the resources to build a more resilient network.
Dual Providers
The next architecture introduced two independent Internet providers. One served as the primary production path. The second also provided connectivity associated with the remote disaster-recovery environment. Under normal conditions, most production Internet traffic followed the primary provider, while backup and replication traffic could use the secondary path. But simply having two providers does not automatically create useful redundancy. The important question is what happens when one of them fails. That led to BGP.
Introducing BGP
The environment initially used provider-assigned public address space. Each Internet provider connected through its own Cisco edge router. The routers were interconnected and participated in the same autonomous-system architecture, allowing routing policy to determine which provider was preferred for normal traffic. This gave us substantially more control than traditional static routing or simple firewall failover.
The network could make routing decisions based on reachability and policy rather than depending entirely on a single physical connection. It also established the architectural foundation for later changes in disaster recovery, public addressing and cloud connectivity.
Disaster Recovery
Disaster recovery is only useful if the network architecture can actually reach the recovery environment. The remote DR site therefore became part of the routing design rather than being treated only as a place to store replicated data. Routing configuration was prepared so public services could be redirected during a disaster. Failover remained a controlled operation rather than something we wanted occurring unexpectedly because of a transient network event.
The important part was that the architecture and configuration already existed. We regularly tested the process so recovery was based on a known procedure rather than an emergency experiment.
Carrier and Datacenter Changes
Over the years, offices and infrastructure moved, circuit capacities increased and providers changed. Each transition required more than ordering a new circuit. Carrier networking involves the entire lifecycle:
- requirements
- provider selection
- contracting
- demarcation
- installation
- routing
- firewall integration
- testing
- cutover
- decommissioning the old service
I have managed that process repeatedly across production, office and disaster-recovery environments. The underlying technologies are important, but so is understanding how to move from one architecture to another without unnecessarily disrupting the business.
Modernizing the Edge
As the original Cisco routers aged, I redesigned the Internet edge around newer Cisco Catalyst routers. Each provider terminates on a separate router. The routers are interconnected, participate in the same routing architecture and apply policy to control preferred and alternate paths. The design maintains provider independence at the edge while avoiding unnecessary complexity. The objective isn't simply to own two routers and two circuits. It is to ensure that the loss of an individual carrier, router or path does not automatically isolate the environment.
Private Cellular Connectivity
One of the more unusual networking projects involved private connectivity with a major U.S. cellular carrier. Rather than treating mobile devices as ordinary Internet clients, the solution used specially provisioned mobile connectivity that allowed traffic to enter a private network path. The architecture incorporated:
- private mobile addressing
- IPsec connectivity
- BGP
- Palo Alto firewalls
- private autonomous-system routing
This was a good example of using familiar networking technologies in a less familiar environment. BGP is not only an Internet-edge protocol. The same routing concepts can be applied to private connectivity when multiple networks need to exchange reachability dynamically and predictably.
Owning the Address Space
The next major step was reducing dependence on provider-owned addressing. In 2026, we acquired our own provider-independent public IPv4 address space. Owning the address space gives us substantially greater flexibility in how public services can be hosted, migrated and protected as the environment continues to evolve. One particularly interesting capability is Azure BYOIP.
Rather than treating public addressing as something permanently associated with one physical carrier or datacenter, owned address space can become part of a broader hybrid-cloud and disaster-recovery strategy.
Colocation (Detail TBD)
Another major transition is moving production infrastructure away from an office-based datacenter and into a dedicated colocation facility. That changes the network architecture in several useful ways. More details to come at a later date.
The Hybrid-Cloud Direction
The long-term architecture is no longer centered on one physical datacenter. Microsoft Azure is increasingly part of the design. Company-owned public address space and Azure BYOIP create options that were much harder when public addressing belonged entirely to individual carriers. That allows physical colocation, cloud infrastructure and disaster-recovery environments to be treated as parts of a broader architecture rather than isolated destinations. The exact placement of a workload can change. The network architecture should make those changes easier rather than becoming the reason they are difficult.
An Architecture That Kept Evolving
The network did not jump directly from a small-business connection to hybrid cloud. It evolved as the business and its requirements changed:
- single small-business Internet connection
- business-class connectivity and static addressing
- dual Internet providers
- BGP
- remote disaster-recovery connectivity
- tested routing failover
- Office network extended into hybrid cloud
- private cellular connectivity using IPsec and BGP
- provider-independent public IPv4 space
- colocation and Azure BYOIP
- hybrid-cloud resilience
Each stage solved the problems that existed at that point in time while creating options for what came next. That is how I prefer to design infrastructure, not around the assumption that today's architecture is permanent, but around the expectation that the environment will continue to change.