MHVG

Designing a High-Scale SIP Communications Platform

Linux • Asterisk • SIP • RTP • MySQL • Apache • Java/Tomcat • HA • Enterprise PBX Integration

Mobile communications in a hospital present an unusual voice problem. Thousands of mobile users need basic telephone functionality, but a traditional full-featured PBX brings capabilities, complexity and overhead that may not be necessary for the application. While working at Mobile Heartbeat, we needed something much more focused. We needed a SIP communications appliance capable of supporting thousands of simultaneously registered mobile endpoints while integrating cleanly with the hospital's existing enterprise telephone system. I designed and built the backend architecture that became MHVG.

The Problem

An existing voice platform had already been built around an off-the-shelf open-source product. I initially cleaned up and improved that environment, but its fundamental limitations quickly became apparent. Performance deteriorated badly as configurations grew beyond approximately 2,000 endpoints. The expected scale was considerably larger. Adding hardware around an architecture that wasn't designed for the workload wasn't going to solve the underlying problem. We needed a purpose-built platform.

Keep the Job Small

MHVG did not need to become another enterprise PBX. Hospitals already had those. The appliance needed to provide the calling functionality required by our mobile SIP endpoints and integrate with the customer's existing communications infrastructure. That distinction drove many of the architecture decisions that followed. Simple dialplan, if a user calls another number configured locally, it routes to inside contexts. If the destination does not exist locally, the call is sent on SIP trunk to customer's PBX, loop Prevention terminates trunk incoming that do not exist onboard. The enterprise PBX remains responsible for the customer's broader telephone environment.

Architecture

I built the platform using established open-source components.

• Ubuntu Linux provided the operating environment.
• Asterisk provided SIP registration and call processing.
• MySQL stored users, telephone numbers and configuration data.
• Apache HTTPD participated in the management and API layer. A management interface was developed by an engineering consultant using Java and Tomcat.

Behind that interface, I designed a backend based heavily on Unix principles: small components performing focused operations, with higher-level components orchestrating them. Over time, the system grew to include hundreds of scripts. Management operations were submitted through an API and queue mechanism. Backend processes picked up those operations and performed the necessary work. When configuration changes were committed, the backend creates/updates the appropriate Asterisk configuration and initiates reload. The goal was not architectural complexity. It was predictable behavior, stability and performance.

Designing for Scale

SIP workloads can expose bottlenecks that have little to do with raw CPU capacity. During development and testing, I spent considerable time identifying what limited density and call-processing performance. One significant factor was disk I/O and the interrupts associated with it. That influenced both operating-system tuning and application design. MHVG uses a non-standard Linux kernel, and unnecessary disk activity was minimized wherever practical. Features also had to justify their performance cost. Voicemail, for example, substantially reduces system capacity. Since voicemail wasn't fundamental to the purpose of the appliance, adding traditional PBX features at the expense of scale would have worked against the architecture. The objective wasn't to build the PBX with the longest feature list. It was to build the appliance required for this particular workload.

Scale

The resulting system was designed to handle more than 10,000 configured users per instance, and 4,000 simultaneously registered endpoints per active server. Initial load testing went beyond that. MHVG systems are deployed as high-availability active/passive failover pairs. A single active server is designed to carry the complete endpoint workload. Scaling beyond an individual system is accomplished by deploying additional MHVG instances rather than continually increasing the complexity of one central platform. That architecture is particularly appropriate for customer-premises deployment. A large hospital can operate an MHVG system of its own, while a multi-hospital system can deploy multiple instances placed as appropriate. MHVG entered active customer use in 2015. More than a decade later, the architecture remains deployed across multiple healthcare customers, tens of thousands of users.

Enterprise PBX Integration

MHVG was designed to work with the customer's existing communications infrastructure rather than replace it. The platform integrates with enterprise PBX environments through SIP trunking while supporting the redundancy and distributed architecture commonly found in large healthcare communications systems. I worked extensively with environments based on Cisco Unified Communications Manager and Avaya Aura, including complex multi-node and multi campus deployments. A fundamental design principle was maintaining a clear boundary between MHVG and the customer's enterprise voice environment, allow the customer PBX to do all inter-site dialplan and pstn routing. This allowed MHVG to remain focused on its purpose while taking advantage of the routing, carrier connectivity and enterprise telephony capabilities customers already had.

More Than Basic Dialing

Keeping the core architecture focused did not mean every call was simple.

MHVG developed capabilities including:

• DIDs and additional-user-number handling
• pattern-based routing manipulation
• dynamic call-destination selection (http api calls to application servers)
• configurable SIP signaling (e.g. Emergency call handling)
• enterprise PBX interoperability

One particularly useful capability allowed call-destination behavior to be determined dynamically through an HTTP API request. That connected application logic with real-time communications behavior, allowing applications to direct calls to the correct resources based on criteria like shift and availability.

Cisco and Avaya

MHVG was not the only architecture available for our communications platform. I also worked extensively with deployments in which mobile SIP endpoints registered directly to the customer's enterprise PBX. That included significant work with:

• Cisco Unified Communications Manager
• Avaya Aura

Dynamic calling functionality was integrated using platform-specific technologies including Cisco CURRI and Avaya TSAPI (AES). I was also running formal interoperability and compliance test work with both vendors covering direct endpoint registration and dynamic calling, as well as SIP-trunk interoperability with MHVG. Across the broader direct-registration architecture, these deployments reach 6 digits user counts.

Emergency-Call-Handling Assistance

Mobile SIP endpoints introduce another difficult problem: physical location. A hospital's street address may not be sufficient information during an emergency. Building, floor, wing or room can matter. Our application is not a 911, E911 or NG911 system, and maintaining that distinction is important. Instead, we developed functionality capable of assisting a customer's emergency-call-handling environment. With appropriate permissions, the mobile application can obtain information such as the device's IP address and the BSSID of the wireless access point to which it is connected. Engineering developed support for carrying this information in configurable SIP headers. I designed the corresponding MHVG functionality so selected information could be preserved and forwarded as a call moved toward the customer's communications infrastructure. I also worked on equivalent integration approaches with Cisco Unified Communications Manager, including SIP Normalization Scripts using Lua. The result is not a claim that MHVG determines emergency location. It provides useful information to the customer's communications and emergency-call-handling infrastructure so that systems designed for that responsibility can use it.

What Made the Project Interesting

MHVG is one of the projects that best represents how I like to work. It required much more than knowing Asterisk. The challenge crossed:

• Linux
• Asterisk
• SIP and RTP
• application behavior
• database and configuration management
• APIs
• performance engineering
• enterprise networking
• Extensive Scripting
• Cisco and Avaya communications platforms
• customer infrastructure
• high availability (Load Balancer Use)
• operational support

← Overview · Next: Infrastructure →