From Startup Server Closet to Enterprise Infrastructure
Fixing the Architecture Instead of Buying More Servers
VMware • Cisco UCS • EMC • Fibre Channel • iSCSI • Cisco Networking • HA/DR • Datacenter Infrastructure
When I joined Mobile Heartbeat in 2013, it was a small startup with just a handful of employees. The infrastructure reflected that reality. We had just a few physical servers of varying ages installed in a small server closet. Cooling came from a portable air conditioner vented into the ceiling plenum. Using the free edition of VMware as independent hosts, each used local storage and was managed individually. Other systems, including domain controllers, file services and backup, still ran on dedicated physical servers. It wasn't enterprise infrastructure, but for a small startup working with limited resources, it had gotten the company that far. Then the requirements changed.
We Need More Servers
Shortly after I joined, a large enterprise customer wanted to test our application at much greater scale. That required substantially more virtual flexibility. I was given a budget to buy additional servers so we could run more virtual machines. After looking at the environment, I recommended that we not buy more servers. The existing compute wasn't the fundamental problem. We had VMware hosts, but without shared storage and proper VMware licensing they effectively operated as separate islands. Adding another island wouldn't fix that. The better investment was in the architecture around the compute we already owned.
Building the First Shared Platform
We redirected the available budget toward several foundational improvements:
- shared storage
- licensed VMware
- a proper backup platform
- iSCSI networking
- A few more sticks or RAM
This gave us capabilities that simply adding more standalone servers would not have provided. Workloads could be managed as part of a virtualization platform rather than being permanently tied to individual hosts, resources could be used more effectively, virtual machines could be moved between hosts, cloning and templating became reality. Failures became easier to recover from, backup and recovery improved, and we had the beginnings of actual infrastructure redundancy. For a startup environment, the solution didn't need to be elaborate or expensive. It needed to solve the right problem.
Finding the Next Bottleneck
Fixing one architectural constraint often exposes the next one. Once shared storage and VM mobility were in place, the existing network became the limiting factor. The switching that had been adequate for ordinary server traffic did not handle the new workload well, particularly during virtual-machine migrations. Rather than treating that as a virtualization or storage problem, I traced the performance issue back to the network. We replaced the bottleneck with an appropriate Cisco switch stack. The difference was immediate. VM migrations no longer overwhelmed the network, and the switching environment remained stable under workloads it had not previously been designed to handle. This is one of the reasons I prefer looking at infrastructure as a complete system. Compute, storage and networking do not operate independently. Improving one part of the architecture can simply move the constraint somewhere else.
Increasing Compute Density
As older servers reached replacement age, we also changed the way we purchased compute. Instead of replacing inexpensive servers with similar inexpensive servers, we moved toward higher-memory, dual-processor systems designed to support substantially greater virtualization density. That allowed us to increase capacity without continually consuming additional rack space. Over time, the environment became more capable while remaining physically compact. Then the company changed again.
From Startup Budget to Enterprise Requirements
In 2016, the company was acquired by a large corporation. Suddenly the infrastructure requirements — and the resources available to meet them — were very different. When I proposed the next infrastructure investment, the new environment gave us the opportunity to think beyond incremental startup upgrades and design an enterprise-class platform. We could finally build infrastructure comparable to our customer environments.
The next generation was built around:
- Cisco UCS compute
- EMC all-flash storage
- Fibre Channel
- Cisco Core and Fabric
- VMware
- EMC DataDomain
This wasn't simply a hardware refresh. The architecture now emphasized redundancy throughout the platform. Compute, storage, network and supporting infrastructure were designed so that individual component failures would not automatically become service outages.
The Same Footprint — A Very Different Platform
One of my favorite ways of looking at the transformation is physical. The earlier environment represented relatively modest standalone servers, local storage and limited redundancy. Several years later, approximately the same physical footprint contained a highly virtualized enterprise platform with dense compute, all-flash shared storage, Fibre Channel connectivity, high-speed networking and fully redundant architecture. The raw performance difference was substantial. But performance wasn't the most important improvement. The environment was also considerably easier to manage, maintain, recover and operate, including remotely. The real improvement wasn't simply how much computing power fit into the rack. It was what the infrastructure could survive.
Designing for Failure
High availability isn't created by purchasing one very reliable server. It comes from assuming that components will eventually fail and designing the system so that they can. The mature environment incorporated redundancy across the major infrastructure layers. That philosophy extended beyond compute and storage. The original makeshift cooling arrangement was replaced by dual dedicated cooling systems, providing redundancy for the server-room environment itself. Remote-management capabilities were improved as well, reducing the number of failures that required someone to physically enter the server room.
More Capability, Not More Space
Over several years, the environment progressed from standalone servers to proper shared virtualization, dense enterprise compute with: shared all-flash storage and Fibre Channel fabric. Proper 10G networking with full redundancy, modern backup and recovery, remote management and redundant cooling and power, while remaining within roughly the same physical rack footprint. The company gained dramatically more capability without solving every new requirement by adding another rack of equipment. The important result wasn't a percentage. It was a platform with substantially greater capacity, resilience and operational flexibility.
Architecture Should Evolve With the Business
The first shared-storage solution I built for the company was not Cisco UCS and Fibre Channel. It shouldn't have been. At the time, we were a small startup. The right solution was an economical shared-storage and VMware architecture that solved the immediate problem with the resources available. Several years later, the company had different requirements, different risks and a very different budget. Then UCS, Fibre Channel, all-flash storage and extensive redundancy made sense. I don't believe good architecture means always selecting the most sophisticated technology available. It means selecting an architecture appropriate to the business at that point in time — while creating a foundation that can evolve when the requirements change.
That is exactly what happened here.