Featured Geek

Reliam's most important asset is our talented team of Internet Application Engineers. Every month exceptional individuals from our team are featured. They discuss issues or solutions that are dear to the featured geeks heart at the present moment. These issues just might be dear to your heart too.

Tuning LAMP Stack

By Chris Harshman, Internet Application Engineer

One of the most critical aspects of Linux, Apache, MySQL, PHP[1] application management is tuning the software components that make up this "stack" of software.  Without proper benchmarking and tuning, your application could slow to a crawl under load, being bogged down by excessive swapping or lengthy database queries -- or worse, it could fail completely.  Here, I will briefly survey some of the tools and techniques available to help avoid such situations.

The constraints of any computer application are processor (CPU) capacity, memory (RAM) capacity, and I/O (disk and network input and output).

Measuring Usage
The Linux command 'top', run from a command line, provides a useful overview of system activity, including an approximation of memory usage, CPU usage (we're concerned mostly with the '%us' value, that is, the amount of CPU use by user applications, which includes Apache and MySQL), and the overall system load average.   

Load, shown as an average of samples taken over the last minute, five minutes, and fifteen minutes, measured as the number of processes actively using the processor(s); anything over the number of CPU cores available to the system represents a potentially overloaded system.  That is, on a dual CPU core system, a load average of 3.0 indicates that, on average, three processes were running, or 1.5 per core.  If your application routinely shows a high load average, and tuning the stack doesn't alleviate the high CPU usage, you may wish to explore moving the stack to a larger server -- one of the advantages of, e.g., using virtualized servers is that additional virtual CPUs can be brought online with a simple reboot, rather than a migration to an entirely new physical server.

One nice feature of top is that it can be used to watch a single user's processes.  On most Linux systems, the Apache httpd processes run as user 'apache' (or 'nobody'), and MySQL tends to run as user 'mysql'.  To watch the resource utilization of just those processes, you can use 'top -uapache' or 'top -umysql'.

The only drawback to top is it paints an incomplete picture of actual memory usage, as it does not properly factor in the use of shared memory.  To determine the actual process size of, for instance, each incremental Apache process, I like to summarize the values found in the statm information under the /proc filesystem for the Apache process identifiers (PIDs), e.g., /proc/<PID>/statm.  To find a list of the Apache processes, I use 'pgrep httpd'.  That information can be used with, e.g., the output of 'free', to determine the maximum number of Apache processes a system can run before memory would be exhausted.

Another specialized trick only applicable to PHP[2] is to configure a custom LogFormat directive with an entry for %{mod_php_memory_usage}n to capture the memory usage of each PHP script invocation.  Capture this information during benchmarking to provide further insight into your application's memory requirements, and you'll be able to tune the number of Apache servers in httpd.conf appropriately.

CPU and RAM capacity are, today, likely to be your application's limiting factors, but input and output, i.e. disk usage and network usage, may limit your LAMP stack's capacity as well.  The tool 'vmstat' can provide a single snapshot, or a periodically updating (e.g., 'vmstat 2' will provide new information every 2 seconds) history, of the memory use, CPU use, blocks of disk input/output, and interrupts and context switches of the Linux system, which can be useful to watch during benchmarking to see what specifically is most heavily tasked.  Other tools include 'iostat', 'sar', and 'sysstat' (found in the 'sysstat' package, which is generally not installed by default; on a CentOS Linux box, for instance, you'd install it with 'yum install -y sysstat').  More specialized tools include mytop (to measure MySQL's resource utilization in detail) and ntop to measure network activity.

Bottlenecks
Finding bottlenecks in a LAMP application can be a tedious process, but the performance increases can be substantial.  A full analysis of system profiling is beyond the scope of this article, but there are a few universal resources that should get you started.

PHP
Profiling PHP has been made relatively easy with the creation of the Xdebug package.  This is a PECL module, installed from the command line, that must be activated in php.ini or a separate .ini file in php.d.  Once activated with xdebug.profiler_enable=1, each Apache process will provide pico-second resolution debugging information in cache files on the path specified using xdebug.profiler_output_dir.  These files can be read by a graphical tool, such as KCacheGrind, MacCallGrind, or WinCacheGrind.  Using these tools it becomes easy to see, e.g., that your application spends an inordinate amount of time waiting for a set of rows from a database query, or recursively parsing an array.   

MySQL
MySQL is, generally, an incredibly fast database server.  Most MySQL performance issues are related to unoptimized tables or queries.  An easy way to identify MySQL performance bottlenecks is to turn on slow query logging.  With a log of your slowest queries in hand, it should be relatively straightforward to determine which tables need an index added, etc.

If slow queries aren't the issue, the MySQL daemon itself may need to have its configuration parameters tuned.  Several tools exist to help identify potential problem areas, including mysql-tuning-primer.sh and Kontrollbase.  With a couple of days of normal MySQL usage for these tools to analyze, tuning suggestions for things like the maximum number of connections, the thread cache values, etc., can be optimized.

Benchmarking
I recommend benchmarking early, and benchmarking often.  Before you start tuning, benchmark the system to obtain a baseline performance metric, and as you proceed through different tuning steps, benchmark again to measure the effectiveness of the change.  Since we're tuning the entire LAMP stack, it makes sense to test scripts (PHP, Perl, etc.) that make use of the database, through Apache, hitting the whole stack at once.   

One of the simplest tools is ApacheBench, which should have been packaged with your Apache web server software.  ApacheBench is useful for concurrently hammering a single URL as a stress test, but it's not meant to simulate real-world conditions.  Another tool, siege, can be run in 'internet' mode, taking a list of URLs from a file and firing off concurrent hits against them randomly.  This comes closer to simulating the scenarios your application is likely to see in the real world, and for a rapid change-and-test cycle it may be sufficient.  To come as close as possible to real world use cases, however, the reigning champion is JMeter, which can be configured with an HTTP proxy to capture sample users' traffic patterns through a web application, complete with timers to record the delays between HTTP requests, etc.  There's a substantial learning curve, but for distributed testing and use case simulation, JMeter is one of the best options available.  Finally, if you just want to test MySQL independently of the rest of the LAMP stack, Super Smack may be all you need.

In Conclusion, taking some time to learn the tools and techniques presented here can provide dramatic insight into your LAMP applications, and enable you to get the maximum performance out of the server software, enabling greater capacity and/or lowering resource utilization, enabling you and your application to do more with less.

Notes

1 Although the "P" in LAMP can refer to one of several languages, this article deals with the common use of PHP for web application development.  Similar techniques are available for Perl, Python, etc.

2 Unfortunately, a review of, e.g., Google Code Search (http://www.google.com/codesearch?q=ap_table_set+notes) reveals the Apache daemon's module notes structure to be sparsely utilized