FYI: PHP 5.2.4 Sucks.... use PHP 5.2.11 (minimum)

[calguy1000]

Today I will tell you a little story of some server upgrade hell, so that hopefully others don't have the same problem.

We were experiencing problems with the server load on one of our VPS's going EXTREMELY high on a fairly regular basis.  It is running [url=http://"http://ubuntu.org"]Ubuntu[/url] 8.04 LTS (long term support). We thought that this may be related to some kind of security vulnerability of some sort (either in CMSMS or something else).  So it was time to upgrade the software.

I first verified that my backups were current (I love [url=http://"http://rdiff-backup.nongnu.org/"]rdiff-backup[/url] and [url=http://"http://www.rdiffweb.org/"]rdiff-web[/url] btw).

Then I thought I was going to be smart and do the package upgrades on the server, before upgrading [url=http://"http://isp-control.net/"]ispcp[/url] and all the cms made simple sites that are hosted there. I logged in as root and did the obligatory

Code: Select all

apt-get update && apt-get upgrade

The updates were WAY behind on this (virtual) machine, so there were like 50 upgrades to do... including apache2, bind9, sendmail, postix, php, and mysql....   the activity output from apt-get upgrade told me that it was replacing:
php-5.2.4-2ubuntu5.5 with php-5.2.4-2ubuntu5.10  where the 2ubuntu... stuff refers to ubuntu's own packaging versions.   I thought that this was all dandy, cuz 'latest versions of everything is better right?'.

Anyways, immediately after the upgrade, and subsequent restart of services... everything broke. I was getting a '500 - Internal Server Error'  and absolutely none of the sites worked.   A quick search through the logs (once I found where ispcp had put the error logs) told me that [url=http://"http://eaccelerator.net/"]eaccelerator[/url] was incompatible with the new php and needed to be recompiled.... okay this isn't a problem... I downloaded the latest and greatest package, followed the instructions and installed it, then restarted apache once again.

Now we were on to the next problem...  Every page I was going to (except one with a simple phpinfo() call) was giving me an open_basedir restriction error.... that was gross because the site used to work (almost) perfectly.  And to top it all off, because the site had now been down for like 1/2 an hour or so... customers were starting to call... it was getting ugly.

I hunted through all the config files in /etc/apache2, found (what I thought) were all the places where the open_basedir stuff was called and tried changing them and restarting apache.... nothing worked, my changes were having no effect.  Then I did a wider search (see he command below) and found a bunch more files for fastcgi that had open_basedir lines in them.  I changed these files and restartedapache, and there was STILL no effect...I still got the dreaded open_basedir restriction error.

Note: Here's a useful command I have used on many unixes to find files containing certain contents.

Code: Select all

find / -name '*' -exec grep open_basedir {} /dev/null \; 

So now what to do?  I tried a simple 3 line php script to try to isolate the behaviour:

Code: Select all

<?php
echo 'DEBUG: PWD: '.getcwd().'<br/>';
echo 'DEBUG: open_basedir: '.ini_get('open_basedir'); echo '<br/>';
include('/the/absolute/path/to/the/site/htdocs/tmp/cache/phpinfo.php');
?>

This little script output the correct current working directory, the correct open_basedir path, but still gave me that dreaded error.   Now things were getting desperate.  Customers were getting irate, and I had isolated the problem to php itself.   Unfortunately an hour or so of googling hadn't given me any obvious answers. 

I had two choices:
1.  Revert to Backup. 
     - The problem with this, is it would take hours and hours to restore the site from an offline backup.
2.  Go to an even newer version of php
     - This was the only realistic option because I knew with the newer versions of the E-commerce suite and CGExtensions, etc.  PHP 5.2 is required and PHP 5.2.10 or better is preferred as a bunch of bugs wrt OO programming and zend-allocator and stuff have been fixed in the later versions.... so I opted for the second choice.

This time google was helpful, within about 5 minutes I had an answer as to how to upgrade php on ubuntu 8.04 to a newer version, in this case php 5.2.13.   Infact it lead me to a valuable resource:  http://www.dotdeb.org

The answer was to add these lines to the /etc/apt/sources.list file:

Code: Select all

deb http://packages.dotdeb.org stable all
deb-src http://packages.dotdeb.org stable all

Then repeating the same apt-get command as earlier:

Code: Select all

apt-get update && apt-get upgrade

Along with upgrading php to 5.2.13, it upgraded mysql, bind9 and a few other things even further than the ubuntu repositories had.
Another reboot (just to be safe)... and VOILA everything was working again, some rudimentary testing verified it.... whew... just when the customers were reaching their boiling point I got everything working again.

So let this be a lesson:
1) Make sure you have verified backups (cuz not always can you get out of things the way I did today).
2) Don't arbitrarily trust apt-get upgrade (it would have been nice to have a staging server to test with).
3) Earlier versions of php 5.2 are buggy as hell,  try to keep up with the newer versions... at this time it's 5.2.13

[replytomk3]

Whoa, must have been one adrenaline rush!