How InMotion Hosting Solved MySQL Memory Leaks at Scale with TCMalloc

Database performance sits at the heart of every hosting operation. When MySQL or MariaDB starts consuming excessive memory without releasing it, the consequences ripple through your entire infrastructure: performance degradation, out-of-memory (OOM) crashes, and frustrated customers experiencing downtime.

Our Tier 3 Systems Administration Technical Team Lead, Sean Combs, recently took on the challenge of solving this issue fleet-wide on InMotion Hosting’s servers. The solution transformed how we manage database memory allocation, and the results speak for themselves.

The Memory Management Challenge

MySQL and MariaDB are workhorses of the web hosting industry, powering countless WordPress sites, e-commerce platforms, and applications. But they share a common operational challenge: memory fragmentation and retention that appears as gradual memory growth over time.

This issue manifests in several ways:

• Resident memory usage increases steadily despite stable workloads
• The database process consumes far more RAM than configuration settings suggest
• Restarting the database temporarily resolves the issue, but growth resumes
• Eventually, the operating system’s out-of-memory killer terminates the database process

Many administrators mistake this pattern for a memory leak in MySQL itself. In reality, the culprit often lies one layer deeper: the memory allocation library handling requests between the database and the operating system.

Understanding the Root Cause

By default, most Linux distributions use the GNU C Library’s (glibc) malloc implementation for memory allocation. While glibc malloc works well for general-purpose applications, it has known limitations with database workloads characterized by frequent, varying-size memory allocations and deallocations.

The specific problems include:

Memory Fragmentation: As databases allocate and free memory continuously, the memory space becomes fragmented. Small gaps between allocated blocks prevent efficient memory reuse, forcing the allocator to request more memory from the operating system rather than reusing what’s already available.

Conservative Memory Release: The glibc malloc implementation tends to retain memory attached to the process rather than returning it to the operating system. This optimization reduces expensive system calls, but it creates the appearance of ever-growing memory usage as the process holds onto memory it no longer actively needs.

Thread Contention: In highly parallel database environments with numerous concurrent connections, threads competing for memory allocation access create performance bottlenecks through lock contention.

The TCMalloc Solution

After extensive research and testing, our team implemented TCMalloc (Thread-Caching Malloc), Google’s high-performance memory allocator specifically designed for multi-threaded C/C++ applications. Originally developed for Google’s infrastructure needs, TCMalloc directly addresses the limitations affecting MySQL and MariaDB performance.

How TCMalloc Works

TCMalloc employs a three-tier architecture optimized for parallel execution:

Front-End Caching: Per-CPU or per-thread caches handle most allocations and deallocations without requiring locks. This eliminates the contention problems that plague traditional allocators in multi-threaded environments. Modern TCMalloc defaults to per-CPU mode, where each logical CPU core maintains its own cache for maximum efficiency.

Middle-End Management: The CentralFreeList and TransferCache components manage memory movement between front-end caches and back-end allocation. When a cache needs refilling, these components batch-transfer memory to reduce lock acquisition overhead.

Back-End Allocation: The PageHeap interfaces with the operating system to acquire memory in large chunks (typically 1GB regions). By requesting large contiguous address spaces upfront, TCMalloc minimizes expensive system calls while maintaining fine-grained control over physical memory usage.

Key Performance Advantages

The architectural differences translate into measurable benefits:

Reduced Lock Contention: For small object allocations (under 32KB), TCMalloc achieves 2 to 2.5 million operations per second per CPU with large thread counts, compared to 0.5 to 1 million ops/sec for traditional malloc implementations.

Better Memory Utilization: Size-class allocations minimize wasted space. For example, allocating N 8-byte objects consumes approximately 8N × 1.01 bytes, representing just 1% overhead. Traditional allocators often require 16N bytes for the same workload, doubling the memory footprint.

Reduced Fragmentation: TCMalloc’s approach to memory organization naturally reduces fragmentation. By grouping allocations into size classes and using pages dedicated to specific object sizes, the allocator maintains better memory locality and reusability.

Scalability: Unlike per-thread arena approaches that can lead to memory blowup scenarios, TCMalloc’s per-CPU caching scales efficiently with modern multi-core processors without proportional memory overhead increases.

Implementation Across Our Fleet

Deploying TCMalloc across InMotion’s hosting infrastructure required careful planning and execution. Here’s how we approached the rollout:

Installation and Configuration

The implementation process involves three main steps:

1. Install TCMalloc packages from distribution repositories or compile from source
2. Configure the database to preload TCMalloc via systemd service modifications
3. Restart services to activate the new memory allocator

For MariaDB environments, we modified the systemd service configuration:

[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4"

[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4"

This environment variable instructs the database process to use TCMalloc for all memory allocation operations, overriding the default glibc malloc.

Verification and Monitoring

After deployment, we verified the change using MariaDB’s built-in diagnostic:

SHOW GLOBAL VARIABLES LIKE 'version_malloc_library';

SHOW GLOBAL VARIABLES LIKE 'version_malloc_library';

This query confirms which memory allocation library the database is actively using. The response should show TCMalloc instead of “system,” indicating successful implementation.

We also established ongoing monitoring to track:

• Resident Set Size (RSS) stability over time
• Database performance metrics under varying load conditions
• Memory allocation patterns and fragmentation levels
• System resource utilization across the hosting environment

Results and Impact

The results validated our decision to implement TCMalloc across our infrastructure:

Memory Stability: Database memory usage stabilized at expected levels based on configuration settings, eliminating the gradual growth pattern that previously required periodic restarts.

Performance Consistency: Query response times became more predictable, with reduced variance during high-concurrency periods when thread contention previously created bottlenecks.

Operational Efficiency: The elimination of memory-related database restarts reduced administrative overhead and improved service reliability for our customers.

Scalability Headroom: Better memory management created additional capacity within existing hardware allocations, effectively increasing the density of customer accounts we can reliably serve per server.

Alternative Memory Allocators

While we chose TCMalloc for our implementation, other alternatives exist for addressing MySQL memory management:

Jemalloc: Another high-performance allocator designed by Facebook (now Meta) engineers. Jemalloc excels at reducing memory fragmentation and is widely used with database workloads. Some hosting providers report even better results with Jemalloc than TCMalloc, particularly in certain workload patterns. The choice often depends on specific usage characteristics.

Differences from default malloc: Both TCMalloc and Jemalloc provide more aggressive memory defragmentation, better thread scalability, and more efficient memory reuse compared to glibc’s default implementation. The choice between them often comes down to specific workload characteristics and architectural preferences.

Technical Considerations

When implementing alternative memory allocators, several factors deserve attention:

Virtual Memory (VSS) vs. Resident Memory (RSS): TCMalloc reserves large address space regions (typically 1GB chunks) that appear in Virtual Size metrics but don’t consume physical memory until actually used. This means the VSS of database processes may be substantially larger than RSS. Attempting to limit applications by restricting VSS will fail long before approaching actual physical memory usage.

Compatibility: Most modern Linux distributions and database versions support TCMalloc without modification. However, testing in development or staging environments before production deployment remains essential to verify compatibility with your specific configuration.

Not Loading at Runtime: Never attempt to load alternative memory allocators into running processes (for example, through Java’s JNI). Applications that have allocated objects using the system malloc cannot safely pass them to TCMalloc for deallocation, resulting in segmentation faults or undefined behavior.

Tuning Opportunities: While TCMalloc’s default configuration works well for most database workloads, advanced users can tune parameters like cache sizes, page sizes, and memory release aggressiveness to optimize for specific usage patterns.

When to Consider This Solution

This memory allocator optimization makes sense in several scenarios:

• Growing Memory Usage: Database processes steadily consuming more memory over time despite stable workloads
• Frequent OOM Events: The operating system’s out-of-memory killer regularly terminating database processes
• High Concurrency: Environments with many simultaneous database connections experiencing performance degradation
• Memory-Constrained Servers: Systems where efficient memory utilization directly impacts capacity and cost

Conversely, if your database memory usage remains stable and you’re not experiencing memory-related issues, the default allocator may be sufficient. As with any infrastructure change, implementing solutions to address actual problems rather than theoretical ones ensures the best return on engineering investment.

Conclusion

Memory management optimization represents the kind of infrastructure engineering that happens behind the scenes at InMotion Hosting. By implementing TCMalloc across our shared hosting fleet, we enhanced database reliability, improved performance consistency, and increased operational efficiency without requiring any changes from our customers.

For hosting providers, system administrators, or anyone managing MySQL or MariaDB at scale, alternative memory allocators like TCMalloc offer a proven path to better resource utilization and more predictable database behavior. The implementation is straightforward, the impact is measurable, and the benefits compound over time as workloads grow.

Technical details and implementation documentation are available in Google’s TCMalloc repository and official documentation.

This solution was developed and implemented by Erik Soroka and the InMotion Hosting Systems Team. For questions about infrastructure optimization or hosting with enterprise-grade reliability, contact our solutions team.