PHP 5.0.4 pooched my web server
I run a web server (Apache 2.1.x) at work which is the collaboration system for the developers. It provides a Subversion repository and a WebDAV volume we stick shared files onto, etc.
We figure a Wiki will be a cool addition (and, in fact, it really is) so I install PHP 5.0.4 and dokuwiki. While I'm at it, I upgrade from Apache 2.1.1 to 2.1.4 (upgraded APR also), and Subversion from 1.1.4 to 1.2-rc2.
It's all great, the Wiki is awesome (mad props to dokuwiki, it's simple to use and looks nice right out of the box), people are adding to the wiki, yay.
Sometime later, people start complaining that Subversion commits are failing due to some strange permissions error from the server. I look into it and the transactions directory in the repository DB (I'm using FSFS) has directories in it with mode 666 (-rw-rw-rw-). After creating these directories without the execute bit set, it fails to access the files within them and produces this error.
So now I'm wondering what kind of bug Subversion 1.2-rc2 has that wasn't in 1.1.4. I briefly has 1.2.-rc1 installed and it didn't have this problem, so maybe it's a new bug. The Subversion folks weren't biting, since no other such bugs had been reported. Then backing out to Subversion 1.1.4 still had the problem. I'm befuddled. It seems like maybe a umask problem, but it's intermittent, so it's happening in the httpd process. Argh!
We later discover that this problem isn't limited to Subversion. Users of the WebDAV volume are starting to see this issue with newly created files and directories (both cases mode 666, where the usual modes are 755 and 644). It's really not looking like a Subversion bug.
So now I'm building and installing all manner of versions of APR, httpd, and Subversion, trying to isolate this problem to some version of something, and I'm still stumped. I even went as far as to go back to APR 0.9.5, and httpd 2.0.x, older versions than what I had originally, just to have the canonical supported Subversion setup. The only version that I can get to work is the binaries I had from before (which I had cleverly backed up), but that didn't include PHP, which I needed for the WIki.
And then it dawned on me that the other thing I changed was adding PHP to the mix. So I disabled the PHP module and waited. Sure enough, a day later, there have been no permissions problems reported. (Though plenty of "Hey why's the Wiki busted?")
So it appears that PHP 5.0.4 does some strange thing that alters the file mode of files created by httpd to the number of the beast. I won't read too far into that, but I'm going to try PHP 4.3.11, which should be fine for dokuwiki, and see how that fares.
Moral of the story: don't change everything at once. Yes, I've already learned this moral many times.
Comments
Well, PHP 4.3.11 exhibits the same problem. I guess PHP really doesn't work with Apache 2.0. PHP is weak sauce.
Posted by: Wilfredo Sanchez | April 28, 2005 03:14 PM
I was actually just was trying to setup a new subversion server using the packages on your idisk.
Your current APR packages are 1.x.x while mod_dav_svn.so from your subversion package seems to be linked against APR 0.x.x. I can put the older libs there in order to get Apache to start up but i'm not sure what the interdependencies are.
If you're using the same packages you're posting this might possibly be a cause for error.
Posted by: Kelvin McDowell | April 29, 2005 08:50 PM
Oh, and thanks for providing these. I was using your packages from early march before I decided to upgrade everything since I was wiping my harddrive for Tiger (changing everything at once, yes).
Posted by: Kelvin McDowell | April 29, 2005 08:54 PM
You sure? I've been rebuilding those packages a lot last week as a result, and I built the current set all together from scratch… I see it linked against libapr-1.0:
[/usr/local/apache/modules] wsanchez% otool -L mod_dav_svn.so mod_dav_svn.so: /usr/local/subversion/lib/libsvn_repos-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_fs-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_fs_fs-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_delta-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_subr-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/apache/lib/libaprutil-1.0.dylib (compatibility version 2.0.0, current version 2.1.0) /usr/lib/libiconv.2.dylib (compatibility version 5.0.0, current version 5.0.0) /usr/local/expat/lib/libexpat.0.dylib (compatibility version 6.0.0, current version 6.0.0) /usr/local/apache/lib/libapr-1.0.dylib (compatibility version 2.0.0, current version 2.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 71.1.3)Posted by: Wilfredo Sánchez Vega | May 2, 2005 12:29 AM
This is what i've got. The subversion dmg i'm using is the one from your 10.4 folder, dated 28-apr-05 3:15 PM. I'll delete the current files and try reinstalling the latest subversion from your iDisk.
Mini:/usr/local/apache/modules kelvinm$ otool -L mod_dav_svn.so mod_dav_svn.so: /usr/local/subversion/lib/libsvn_repos-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_fs-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_fs_fs-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_delta-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/subversion/lib/libsvn_subr-1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/local/apache/lib/libaprutil-0.0.dylib (compatibility version 10.0.0, current version 10.6.0) /usr/lib/libiconv.2.dylib (compatibility version 5.0.0, current version 5.0.0) /usr/local/expat/lib/libexpat.0.dylib (compatibility version 6.0.0, current version 6.0.0) /usr/local/apache/lib/libapr-0.0.dylib (compatibility version 10.0.0, current version 10.6.0) /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 365.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)Posted by: Kelvin McDowell | May 2, 2005 01:49 AM
OK. Tried again with the 10.4 version and the same thing. Using the 10.3.9 version gives me what you have above and Apache starts up correctly. w00!
Posted by: Kelvin McDowell | May 2, 2005 02:06 AM
But a checkout using http:// fails although it works fine with file:// taking a look at recent messages on the subversion mailing lists it seems lots of people are having problems under OS X 10.4. I'm just going to give up for now. Thanks for the help though :)
Posted by: Kelvin McDowell | May 2, 2005 06:38 PM
ˇAye caramba! That sucks… Good luck, guys!
Posted by: Travis Cripps | May 2, 2005 09:37 PM
I rebuilt all of the 10.4 packages again from scratch; perhaps I screwed them up, since I've been concentrating on a 10.3-based server. They've also been rebuilt using gcc 4.0, so they will more certainly not run on 10.3. (Well, still might run on 10.3.9.)
Posted by: Wilfredo Sanchez | May 3, 2005 11:42 AM
The link problem is fixed with the rebuilt 10.4 packages but doing a checkout using http:// still aborts because of errors. This also happens with your 10.3.9 packages and the packages (from Mar 3) that I previously was using under Panther. It really does seem to be something specific to the interaction between 10.4 and dav.
Posted by: Kelvin McDowell | May 3, 2005 02:10 PM
Interesting… I'm not having problems doing checkouts from apache.org, nor on our internal server. Not sure what could be going on here. Would be interesting to know if other people's builds have this problem.
Posted by: Wilfredo Sanchez | May 3, 2005 03:12 PM
Er, you may be misunderstanding. I'm trying to run a server on 10.4 over HTTP. I wouldn't expect checking out from apache.org or your internal server (which you mentioned is 10.3 based) to have problems. Or maybe i'm misunderstanding something.
Posted by: Kelvin McDowell | May 3, 2005 04:56 PM
Ah, OK, yes, I thought you were having client issues. I don't have a 10.4 svn server set up yet.
Posted by: Wilfredo Sanchez | May 4, 2005 11:20 AM
Just a "me too" for Kelvin. I'm also having some weird problems checking out from a Subversion server running under 10.4. I'd love-love-love to find a workaround or fix!
Posted by: Brad Oliver | May 4, 2005 05:06 PM
Oh, finally I stumbled across this - I have been wiping my server HD for about the fifth time and trying to build a svn/WebDAV server from sources in 10.4 - I just can't get it to work properly, seems like my files are limited to a maximum of 64k???
At least now I know I'm not the only one - maybe I should just wipe everything again and go back to 10.3 where things worked just fine before?
I guess I'll head over to the svn mailing lists now to see what others are talking about, I haven't actually checked that out yet...
Posted by: Andreas Amann | May 9, 2005 05:44 PM
This is svn mailing list post that you want to look at http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=99926
Posted by: Kelvin McDowell | May 9, 2005 11:34 PM
Thanks Kevin - look like I really should just downgrade my server to 10.3...
What really confuses me though is that some people claim to have it working properly:
http://golem.ph.utexas.edu/~distler/blog/archives/000557.html
Posted by: Andreas Amann | May 10, 2005 12:26 AM
I imagine he hasn't really stress tested it much. I think the reason people using subversion noticed problems before anyone else is because one of the first things you do is svn co often with a fair amount of large binaries. I'd usually get a fair bit into a checkout -- including successfully getting a couple multi-megabyte files -- before hitting errors.
More evidence: apache-users mailing list message
Hopefully someone with access will file a bug with Apple / Apache.
Posted by: Anonymous | May 10, 2005 01:29 AM
Actually, I eventually came around to the conclusion that Tiger + Apache 2.0.x is broken.
I'm also finding another strange issue, in which child processes started with Perl's open2() aren't exiting when they're supposed to. Stuff which worked just fine under 10.3.9 and Perl 5.8.6 ...
Posted by: Jacques Distler | May 19, 2005 10:32 PM
+1 here for the file transfer problem in Tiger. I've been beating my head over it for quite a while. feels good to see I'm not alone.
turn apache's logger to debug and you'll see a pretty obscure error (*writing to network) when trying to download large (>100kb)images or files. I've reinstalled, disabled this and that..no matter what I do, I get the same problem.
Posted by: metto | June 15, 2005 03:53 PM
So is this Apache 2.0x larger-file bug fix in any recent or upcoming releases of OS X Server?
Thanks.
Posted by: Chris Ryland | June 17, 2005 05:03 AM
The problem with Apache 2.x and Tiger involving large files is understood now. I've committed a fix to the Apache APR sources.
Posted by: Wilfredo Sanchez | June 20, 2005 10:09 AM