Took some time, but we finally made it, upgrading not only the Apache server, but also the boxes on which it runs. We’re now using 64-bits boxes, with 8GB RAM and running Apache 2.2, effectively reducing by two the number of boxes we used with 32-bits boxes we used for Apache 1.3, with a better quality of service.
Most of the migration time was taken by evaluating our configuration and checking the differences between Apache 1.3 and Apache 2.2, plus evaluating that everything worked well. We didn’t have any test procedures before to automatize the testing of the server and see that everything works as usual. I started a small piece of code to test the aaa setup (see here below) and now plan to migrate to the Apache perl test framework, which I found very useful. This will minimize the migration time next time this issue arises. Some time was also needed to update different scripts and databases to cope with difference between 32 and 64-bits architectures.
At W3C we use some custom aaa modules (Authentication and Access Control/Authorization in Apache notation) that basically let you do more fine-grained access-control than the one proposed by Apache by default. In a nutshell, they let use your IP@ automatically in guise of a password (a feature dating from the CERN server, which has never been available AFAIK in Apache), and fine-grained access control using ACLs, stating who can access which resource using which HTTP method. Although the code for this modules is available on our public CVS server, there are probably not useful to other users because we haven’t cleaned/published the scripts that let you populate the aaa dbm files. You can use the CVS Web interface to see the changes I had to do to port the modules to from Apache 1.3 to Apache 2.2. (not so many).
The Apache 2.2 architecture is cleaner, more modular, and provides more API hooks than the Apache 1.3. This has minimized the number of places where a patch is needed. The only code patches I needed to apply were to allow passing the IP@ to the access control modules in mod_auth_basic and mod_auth_digest (by default, Apache stops the request processing if the user hasn’t been authenticated). I also needed to extend mod_negotiation so that it does take into account the authorized conn-neg resources, discarding the unauthorized ones (the default behavior is to discard everything if any resource is unauthorized). These were very minor extension patches, consisting of only a couple of lines. The plan is to propose them to the Apache developers. For the moment, we made alternate modules for mod_auth_basic, mod_auth_digest, and mod_negotiation that include these minor patches. We use them instead of the ones that Apache proposes. We didn’t need to patch the core Apache code to add these features, as was the case in Apache 1.3. This is good.
On the other hand, we found a bug in the Apache 2.2 server. Connneg and internal rewrites generate something that Apache calls a sub-request. When handling sub-requests, Apache 2.2 simulates a complete request. However, because of an speed optimization change, Apache doesn’t take into account .htaccces files when processing sub-requests… although it did so before in Apache 1.3 and Apache 2.0 This is clearly a bug. I submitted a bug report, a small code patch, and an extension to the Apache test suite in May (corrected since). The bug was recognized as such by some among the Apache team and I got some initial guidance. However, the patch hasn’t yet been committed. The Apache team seems to be too time-overwhelmed to evaluate my patch. We went ahead and installed Apache2 with this patch and it has been working as expected. Perhaps one day the Apache team will adopt it. Compared to a previous patch I sent them a couple of years ago, the delay between submission and committing has increased from one month to five months and counting.
For the packaging and installation of Apache, we’re now using our own Debian packages, derived in 99.99% from the official Apache 2.2 Debian packages. They are clean, efficient, easy to install. Moreover, when you have patches, you just have to pop them to a directory and the Debian packaging system will take care of patching the code when building a new package. As you may guess, we are generating our own Apache2 Debian packages to be able to apply our patch.
Moreover, this lets us benefit from the Debian Apache 2.2 security patches.
Apache 2.2.6 came out at the same time we did our migration to 2.2.4. The Apache-dev list reveals there are some issues regarding mod_perl, so we’re going to wait a while before migrating, probably to 2.2.7.