software, business, and fun

A client’s hosting company gives 2TB of bandwidth per server, but they got hit with a 1.3TB overage last month. That seems strange seeing how the sites hosted are low traffic sites - definitely not 3.3TB worth of usage.

Even stranger is the host with the overage is an older MySQL/FTP only server, no Apache or anything else. Strangest of all is that he RRDTool bandwidth graph provided by the hosting company shows 3.1TB data in, 200MB out! What?!

After ruling out any exploits or incoming DOS, I installed vnstat, ntop, and darkstat to let them log for a while.

Meanwhile, tcpdump showed a constant stream of MySQL traffic between the database server and the main web server. A quick look at vnstat (since it could provide the quickest answers) showed the exact opposite of the hosting co’s RRD graph - 10 times the data going out, not in - over the next hour.

The next day, all tools reported different results. darkstat showed that incoming MySQL traffic to be around 1/100th of the outgoing traffic. This makes sense since the db server is basically a massive SELECT box. darkstat’s hosts section though reported a near equal amount of traffic in and out between the web and db server.

ntop returned similar results.

This is quite disturbing because after a few days of logging, the bandwidth graphs do not correlate to the usage generated by RRD (which is what they bill by). So, what to do? RRD bandwidth graphs for the web server show a strongly correlation between amount of traffic going out and what the db server says is incoming. Hosting co says they monitor and bill traffic at the switch. I couldn’t get the numbers to jive, but had enough of a clue to make a hypothesis/action plan before they get another overage bill:

1. TCP/IP overhead between the web/db server is causing the most traffic, or at least enough to push it over.
2. The poorly written, closed-source CMS the web server runs is making inefficient use of the database.
3. Due to some proprietary Windows software, the web server must stay Windows, so the database for the CMS needs to be moved to the web server.

(They offer a private network connection between 2 hosts that would normally be the Right Thing, but several factors prevent it from being used for this client. Not an option in this case.)

With a hypothesis in hand and a plan in mind, I set about to do the unbearable - migrate a super-stable Debian based database server to a 4 year old Windows server.

[Part 2 will cover getting MySQL running on the Windows server, dumping the database from the existing db server, and reconfiguring the CMS/web server to use the new local db.]

http://dev.rubyonrails.org/ticket/10733

Cool. It worked. Seeing the failed CruiseControl message for it was driving me crazy.

http://dev.rubyonrails.org/ticket/10733

Got tired of running all rake tests to try a patch, so I just need to run a few files.

/code/rails/actionpack travis$ /opt/local/bin/ruby -Ilib:test "/opt/local/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb" "test/controller/caching_test.rb" "test/controller/new_render_test.rb" "test/controller/rescue_test.rb"


The ssl setup is as expected with Apache, but we had to add

RequestHeader set X_FORWARDED_PROTO 'https'


into the conf so mongrel is aware the connection came in over SLL. Otherwise my SslRequired actions were redirected forever since the request was SSL in apache, but not to mongrel eyes.

http://blog.innerewut.de/2006/06/21/mongrel-and-rails-behind-apache-2-2-and-ssl

A new Debian development server needed trac installed. I would rather use the new trac .11, but etch comes with .10 as a package so we used that.

The install is not that big of a deal. The biggest problem was that I upgraded to python 2.5 instead of stock 2.4 for debian, so clearsilver fails it - it does not know about 2.5 in it’s make files.

Following the advice here:

https://launchpad.net/ubuntu/+source/clearsilver/+bug/86685

I installed the newest clearsilver, and it stopped whinging about neo-cgi. But then it couldnt find the svn bindings because they are in python 2.4 site-packages. That is my one beef with python - I’m sure it’s site-packages is a good idea, but I have had a hard time wrapping my head around them pretty much anytime I have to deal with a Python stack.

Oh well. On this devel box, python 2.4 is fine. I’ll unlink python 2.5 and let it rot. The clearsilver hack was probably not needed anyway, and worrying about this anymore takes away from time actually using Trac.

Project had a model without a timestamp field, but we needed to figure out (roughly) how old each one was. They needed a report of their age to make them pay up - the model was for paid advertisers.

Best fit was to find when their logo was last updated. That day will be at least since they were entered online, and in general pretty close to the original date.

Advertiser.find(:all).each do |a|
if a.logo
puts "#{a.market.name} - #{a.name}:" + File.mtime(a.logo.full_filename).to_s
end
end

while the original project to use python appears to be dead, i still want to do more with python.

i prefer the macports install, but apple’s stock config is listed here too.

python stack:

1. python binary/release

os x runs behind. ships with 2.3 for now.

/System/Library/Frameworks/Python.framework/Versions/Current -> 2.3
/System/Library/Frameworks/Python.framework/Versions/2.3
/usr/lib/python2.3 -> /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3
/usr/lib/python2.3/site-packages is where our packages can live.

2. readline is broken in apple’s python

http://www.friday.com/bbum/2006/03/06/python-mac-os-x-and-readline/

install readline 5.1, install python hack, good to go

3. package management - would like to use macports.

macports wants python 2.4.4 - okay. going to ignore apple’s then.

sudo port install py-sqlite py-docutils py-mx
export PYTHONBIN=/opt/local/Library/Frameworks/Python.framework/Versions/Current/bin
export PATH=$PYTHONBIN:$PATH

4. put django into our site library. original project was untested past svn 5024.

ln -s ~/code/django-5024/django /opt/local/lib/python2.4/site-packages


5. django-admin.py needs love so we can run it

ln -s /opt/local/lib/python2.4/site-packages/django/bin/django-admin.py /opt/local/bin/django-admin.py


Capistrano 2.1 did not agree with me. I am a huge Capistrano fan, but the newest version just doesn’t do it. My install of version 1 isn’t broken, I’m not going to fix it.

Version 2.1 has 37 signals only code in it and doesnt work with mongrel_cluster 1.0.3. palmtree 0.0.6 will give it a new mongrel cluster recipe, but why do you have to use that too? Not really interested in installing another gem to get Capistrano working like it did before.

Project need a screenscraper to pull out the google maps portion of a page and embed on our own.

Problem is Radiant cms likes to take over all your urls, so how to make a custom ruby page that lives within a Radiant powered site?

Radiant by default is a systemwide gem that would prefer you didnt do things like this, and going to the trouble of adding a custom plugin to manage one page was overkill. I don’t remember if there was a static content plugin - I think I found one, but it wasn’t able to run custom Ruby code.

What we needed was really really simple, no edibility or site layout on the page so here’s what I made.

Here we go:

* start by adding the app/controllers and app/views folders.
* add in the controller you want with these specifics:

skip_before_filter :authenticate

up_path = File.join(File.dirname(__FILE__), '..')
render_file up_path + '/views/location/index.rhtml' and return true


The skip filter removes Radiant’s checking for admin, and the render_file explicitly loads the correct template. Radiant is using the system gem path for views, so it’s necessary to run relative to it’s install.

* routes.rb hacking. at the bottom:
builder = ActionController::Routing::RouteBuilder.new
route = builder.build('YOURURL', {:controller => 'NEWCONTROLLER'})
custom_routes = Array.new
custom_routes << route

# put it in first place
ActionController::Routing::Routes.routes = custom_routes + ActionController::Routing::Routes.routes

Pretty cool! The new url as just a controller will exist on the Radiant powered domain, but not barf over Radiant. I tried this with 0.6.4. YMMV

The actual screenscraping was done hpricot/open-uri, and was pretty straightforward.

http://www.igvita.com/blog/2007/02/04/ruby-screen-scraper-in-60-seconds/
http://weblog.jamisbuck.org/2006/10/2/under-the-hood-rails-routing-dsl

The end result was a big hack, but I got in running in about an hour.

SERVER:su
su www-data
cd /var/lib/subversion/repository
svnadmin create project.name

LOCAL:cd ~/code
svn co http://svnhost/project.name
cd project.name
cp -r ../empty_svn_initial_layout/* ./
svn add *
svn ci -m 'initial project import'
rm -rf ./project.name
svn co http://svnhost/project.name/trunk ./project.name