XHProf Archives - developed.be

Our main RSS-feed at DeWereldMorgen.be is the most requested page next to our homepage.

It seems logic, think of how many rss-readers hourly check the feed. And, think of how many cpu and RAM that consumes, certainly with a fat system like Drupal.

An RSS-feed is easy to make in PHP. All you need is one custom query and a decent library like the Universal Feed Generator to generate the XML.

Create stripped version of Drupal setup

I used the minimal code that is needed to work in the Drupal framework. So I made a blank php-file in my www-root with this code:

require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

This code has the security and functions of Drupal, but without the menu’s, theme’s, and a lot of module hooks. You don’t need a menu or a theme to create an RSS-feed, do you?

So I created my RSS-feed with just this in my php-file:

  • the drupal_bootstrap function
  • the Universal Feed Generator library included
  • one query with db_query()

With cache

Let’s see performance wise. This is the report of a feed generated based on a user’s blog and articles. Memcache was not cleared before execution. Stats are generated with XHProf.

Normal setupMy stripped setup
Number of function calls115.20018.400
Consumed RAM126 MB62 MB
Total execution time1.169 MS462 MS
Number of database queries3315
Query execution time137 MS80 MS

Performance wise, my solution is twice as fast.

Without cache

These stats are generated when caches were cleared.

Normal setupMy stripped setup
Number of function calls3,284,34383,330
Consumed RAM172 MB76 MB
Total execution time9,131 MS1,000 MS
Number of database queries926131
Query execution time769 MS183 MS

9x as fast. 7x less queries.

Of course, this is just for the first run, but still.

More soon. Please discuss this idea.

Key manual: http://techportal.ibuildings.com/2009/12/01/profiling-with-xhprof/

Install XHProf

I made sure xhprof-0.9.2 was installed inside the www-directory of the webserver.

wget http://pecl.php.net/get/xhprof-0.9.2.tgz
tar xvf xhprof-0.9.2.tgz
cd ./xhprof-0.9.2/extension/
phpize
./configure --with-php-config=/usr/local/bin/php-config
sudo make
sudo make install
sudo make test

The php-config file on my machine was located in /usr/local/php-config . Php-config is a file that lists the location of your php-installation. Xhprof needs it for its configuration. If you can’t find the file, you can simply execute ./configure without the parameter.

If phpize is not found, you have to install php5-dev

sudo apt-get install php5-dev

The first time, “make install” and the “make test” both failed with me, but xhprof worked nevertheless.

Set up PHP

In php.ini I set up this at the end of the file:

[xhprof]
extension=/path/to/xhprof-0.9.2/extension/modules/xhprof.so
xhprof.output_dir="/var/tmp/xhprof"

Create the directory: /var/tmp/xhprof

xhprof will keep its logfiles in there (files that are needed for the page analysis)

Set up your webserver

Create an entry for /path/to/xhprof-0.9.2/ in your webserver’s configuration (point xhprof.localhost to the /path/to/xhprof-0.9.2/. If local, setup your hostfile correctly (eg: 127.0.0.1  xhprof.localhost)

Restart php and your webserver.

Set up Drupal

When using Drupal, install Devel, and activate the XHProf settings (admin/settings/devel) and point to /path/to/xhprof-0.9.2/ and the path to the website: http://xhprof.localhost/xhprof_html

Devel will create a link at the end of each page that points to XHProf.

Troubleshooting

If you get the error that the function xhprof_enable is unknown, then it’s because XHProf wasn’t properly installed. Try to install it again and make sure you have restarted php.

If the link in Drupal to XHProf doesn’t work, check if the run parameter in the url is supplied. If not, try to install XHProf again and make sure you have restarted php. A typical error for this is: “No XHProf runs specified in the URL.

For errors concerning php-config and phpize, see above.

Let’s face it: Drupal can be a snail. When you attract lots of visitors, or have a lot of content, you performance will go down. To speed up Drupal, you need to install other software on your server that will make it appear like Drupal goes faster (but in fact stays as slow).

You can do any of these or a combination of these:

  1. Use memcache. Memcache replaces the classic cache-database-tables and puts the cache in the RAM (instead of in the database). This is the fastest way of getting your data.
  2. Use nginx instead of Apache. Nginx is a lightweight webserver that can handle more traffic than Apache. While it will not make your site magically faster, it can surely help up.
  3. Code-improvements in Drupal:
    • Disable menu_rebuild every time a view is saved. Run menu_rebuild only when cache clear is explicitly asked. (this is in fact core hacking, which is wildly disapproved, but it clearly helps)
    • Rewrite heavy queries generated by Views. Views don’t make the nicest queries. Certainly complex views can be made faster when you edit the query yourself. You can do this with a module that hooks into the view.
    • Check for node_load() calls everywhere. These functions eat up memory and should be replaced where possible (a custom query could do). You wouldn’t believe what happens when you call node_load().
    • Cron:
      • inspect all the cron hooks in your Drupal installation. Decide if the tasks are really necessary, and/or edit them. You’ll notice that the cron spends most of its time with indexing the search words.
      • Use visitorstats (like Google Analytics) to see when traffic peeks. If your website peeks at noon, edit the crontab and disable cron around that time. At least all cpu will go to your visitors. (make sure there are no real important tasks to be done). Cron should run “just enough”. I set my cron to run every hour from 11PM to 7AM.
      • Drupal calls home once a day to see if there updates for modules (with fread). This action consumes cpu. I think that once a week is more than enough. Even every forthnight. You can always check for new modules manually.
    • * In order to do find bottlenecks:
      • Use the Devel module. It displays all queries that are made to the database.
      • Use XHProf. A free php-profiler developed by Facebook to find slow components. It displays function calls and generates a graph (also install Dot for that). My article on how to install XHProf.
      • If you have the possibility, use New Relic, a tool similar to XHProf, but more advanced.
  4. Boost: Boost is a Drupal module that caches entire  webpages as static html files for anonymous users. You wouldn’t believe what a boost that gives. However: take in consideration that websites with a lot of content changes will need to refresh this cache a lot. It’ll take some time to configure. Still, it doesn’t speed-up the admin-environment in any way. This article explains how to install Boost with Nginx [mysqlperformancetuning.com].
  5. Varnish: A similar, but better, approach to Boost is Varnish. Varnish also caches pages, but puts them in the RAM, while Boost uses the hard discs (and creates amounts of files). A downside of Varnish is that it’s complex to set up (you have to put it in front of your webserver) and is difficult to config. With Varnish Drupal can’t use anonymous cookies, so you have to patch your entire Drupal setup, and use “Pressflow” as Drupal core instead of the regular Drupal releases (this is just for Drupal 6). Any module that uses sessions will have to be patched. Varnish however promises what it delivers and gives a serious boost to your website and is far more advanced than Boost. My article on how to install Varnish.
  6. If you can afford it, use a seperate (database) server. Drupal generates a lot of cpu-pressure. This makes the database on the server getting less cpu. Some queries take x50 times as much time as on a non busy site. By taking the heath of the database, this should give a performance boost.

attrack