Category — Braindump

After having to perform a hard reboot on a server, I was quite upset to see no mongrels running. The server had been very stable up this point, and mongrel_cluster was linked into /etc/rc*.d.

Turns out the version of mongrel_cluster was too old to deal with the stale PID files left behind after the hard reset.

sudo gem upgrade mongrel_cluster

Got the new init.d script from /usr/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/resources/mongrel_cluster (to support a global PID_DIR)

Then add the –clean flag to the restart and start commands

  mongrel_cluster_ctl start --clean -c $CONF_DIR
  mongrel_cluster_ctl restart --clean -c $CONF_DIR

Finally, I wanted to check the status.

mongrel_cluster_ctl status

was giving the error “ERROR RUNNING ‘cluster::status’: Plugin /cluster::status does not exist in category /commands”

This happens when you have multiple older versions of mongrel_cluster.

sudo gem uninstall mongrel_cluster

After removing the ancient 0.2.1 it worked, so I just left 1.0.5.

Good help from paul goscicki and the mongrel mailing list.

Had to upgrade MDB2 with pear since the older stable version that worked with a Linux install of MySQL 4 did not work with Windows running MySQL 5. getlastinsertID() was not working, among others.

pear config-set preferred_state stable
pear upgrade MDB2_Driver_mysql

It installed a new MDB2 and some other packages.

The client’s install of a closed source CMS was taking over a minute to render some category archive views. Every other view was okay though, so that ruled out a general database problem.

Without having access to the source it became a lot of guessing.

I enabled MySQL’s slow log, then the general log, but nothing was taking more than 1 second to execute. I verified by running the relevant archive view queries from the log by hand - still no problem.

Next step was to start hacking away at some of the views to make sure nothing was wonky. There wasn’t. As long as it’s main() was running, it would cause the slowdown. Disabling main() loaded the page, just with no content beside the template. Okay.. that’s useless. All other hacks I wrote for this software just regex the hell out of what main() returns - they don’t actually touch it. Without access to what’s going on in main() though, I was still lost.

So after much searching, I found a de-zended version of the code online. Normally that would explain it all, but this code is a mix of HTML, PHP, and SQL, with tragic variable names, no error handling, etc. 3000 lines or so of spaghetti for main() - Fun stuff!

After making a valiant effort to clean up the de-zender’s formatting I ended up with something passable enough to look into. Turns out main() contains 30-40 function declarations inside of it, which then get put into the global namespace when main runs, and can then call the appropriate function based on the view.

I could have used Runkit to override the specific view that gives troubles, but PHP can’t handle overwriting functions. Even if it did, the function was ghetto-namespaced inside another function which would override mine, basically negating it unless I could override and rewrite the whole main() function.

Oh well!

In the end, I dug around the view’s code, and it was doing the standard:
@opendir
readdir
strstr each file name to see if a certain file exists

That works okay up to a point, but there were 11,000 files in the directory it searched, and it ran the loop up to 100 times per archive entry, up to 100 entries per page. So 110,000,000 times per page view, all because the programmers couldn’t be bothered to either regex the directory listing a whole, or do up to 100 file existance tests.

So how did it get solved? I had to remove half the files because they were old and would never get seen in an archive anyway, and blog this so in 2 years when the problem hits again I’ll remember how to solve it

Had a server where IIS was not serving the websites out consistently, or at all. Trying to restart it via the GUI didn’t work, nor did iisreset. iisreset at least gave an error:

“IIS admin service or a service dependent on IIS Admin is not active.”

Trying to start IIS Admin Service via the service manager wouldn’t start either. I ended up killing inetinfo.exe with the task manager, then running iisreset and the admin service. That worked.

http://dev.rubyonrails.org/ticket/10733

Got tired of running all rake tests to try a patch, so I just need to run a few files.

/code/rails/actionpack travis$ /opt/local/bin/ruby -Ilib:test "/opt/local/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb" "test/controller/caching_test.rb" "test/controller/new_render_test.rb" "test/controller/rescue_test.rb"


while the original project to use python appears to be dead, i still want to do more with python.

i prefer the macports install, but apple’s stock config is listed here too.

python stack:

1. python binary/release

os x runs behind. ships with 2.3 for now.

/System/Library/Frameworks/Python.framework/Versions/Current -> 2.3
/System/Library/Frameworks/Python.framework/Versions/2.3
/usr/lib/python2.3 -> /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3
/usr/lib/python2.3/site-packages is where our packages can live.

2. readline is broken in apple’s python

http://www.friday.com/bbum/2006/03/06/python-mac-os-x-and-readline/

install readline 5.1, install python hack, good to go

3. package management - would like to use macports.

macports wants python 2.4.4 - okay. going to ignore apple’s then.

sudo port install py-sqlite py-docutils py-mx
export PYTHONBIN=/opt/local/Library/Frameworks/Python.framework/Versions/Current/bin
export PATH=$PYTHONBIN:$PATH

4. put django into our site library. original project was untested past svn 5024.

ln -s ~/code/django-5024/django /opt/local/lib/python2.4/site-packages


5. django-admin.py needs love so we can run it

ln -s /opt/local/lib/python2.4/site-packages/django/bin/django-admin.py /opt/local/bin/django-admin.py


SERVER:su
su www-data
cd /var/lib/subversion/repository
svnadmin create project.name

LOCAL:cd ~/code
svn co http://svnhost/project.name
cd project.name
cp -r ../empty_svn_initial_layout/* ./
svn add *
svn ci -m 'initial project import'
rm -rf ./project.name
svn co http://svnhost/project.name/trunk ./project.name

drutil info

Interconnect: ATAPI
SupportLevel: Apple Shipping
Cache: 8192k
CD-Write: -R, -RW, BUFE, CDText, Test, IndexPts, ISRC
DVD-Write: -R, -RW, +R, +RW, BUFE, Test
Strategies: CD-TAO, CD-SAO, CD-Raw, DVD-DAO


Aptitude was erroring about a missing key. I had to add the key manually with GPG.

http://bsd.b3ta.org/2007/03/27/apt-get-update-no_pubkey/
http://lists.netisland.net/archives/plug/plug-2007-04/msg00060.html
ex: if key A70DAF536070D3A1
was missing
gpg --keyserver wwwkeys.eu.pgp.net --recv-keys A70DAF536070D3A1
gpg --armor --export A70DAF536070D3A1 | apt-key add -

/etc/logrotate.d - holds all files to be run

group together in one file all rails/mongrel logs, and apache in another so you dont restart apache too much.