So, I'm looking to save off a data structure, much in the same fashion as Storable in Perl. I found Shelve but wasn't sure if anyone else had any suggestions. The data is small and will be not be read from or written to often.

If I don't hear anything, I'll just roll with shelve.

Thanks!

So I'm writing you from python land today. I'm kinda a newbie to python, but I ran across something fun that I thought I should point out..

So I was messing with urllib2 and I discovered that there is no way to set the default timeout. That's not cool! To circumvent possible timeouts with the http requests I'm making (bunches of them) I found myself doing two things.

The first thing I did was pack all the urls I want to hit in a dict of a list. A key, and a list of urls as the values. So for each key in the dict I spawn a thread and make my requests for each item in the list.
Check it out:

class LastThread(threading.Thread):

    def __init__(self,site,boxen):
        self.site  = site
        self.boxen = boxen
        threading.Thread.__init__(self)

    def run(self):
        goodboxen = []
        for url in self.boxen:
            try:
                checkres = urllib2.urlopen(url)
            except urllib2.HTTPError,e:
                print "Error: %s:%s" %(url,e)
                continue
            except urllib2.URLError,e:
                print "Error: %s:%s" %(url,e)
                continue
            goodboxen.append(url)
        print "%s good hits out of %s urls" %(self.site,len(goodboxen),len(self.boxen))

def doStuff():
    # ... some code
    checkboxen = {}
    # ... fill the dict..
    print "Ok, got my inventory... Let's do some work.."
    for k,v in checkboxen.items():
        LastThread(k,v).start()

That helps a bit, but was still having issues with some of the urls timing out. So, looking at the innards of urllib2 I discovered it's using the builtin socket module for it's socket operations. That lead me to discover that I could set my default timeout value.

So now I import socket and socket.setdefaulttimeout(whatever) then run my code again.

But now I'm getting a different exception! timeout: timed out. So now, rather than just capture the urllib2 exceptions, I have to also capture socket exceptions.

Just thought that might be useful information for anyone else looking to set the default timeout when using urllib2 or httplib.

In mid February we'll be moving to a new house! It's much larger than out current shanty.

Three full bedrooms so the kids will have their own. There's also a finished attic with a new bathroom. The main bathroom has been remodeled, and the master bedroom is probably twice the size of our current bedroom. BIG kitchen, with a breakfast nook. TONS of closet and storage space. Nice big yard(for Lakewood) and a Garage.

We're super excited and can't wait to get moved in. The only thing we're not looking forward to is packing..of course.

Oh, and it's on Mars...in Lakewood check out the Google Map!

Rounnnd one....DING!

In preparation for the holiday from hell we're doing some performance tuning of our servers.

So, running valgrind on httpd with our current config yielded:

==7410== More than 100000 total errors detected.  I'm not reporting any more.
==7410== Final error counts will be inaccurate.  Go fix your program!
==7410== Rerun with --error-limit=no to disable this cutoff.  Note
==7410== that errors may occur in your program without prior warning from
==7410== Valgrind, because errors are no longer being displayed.

So I ran it with --error-limit=no and still got the same result. Anyone have any ideas on this? Google wasn't terribly friendly on this subject.

Bullocks

So now I have trimmed up the apache config, and disabled some mod_python debugging. The results aren't pretty, but hey, I got results!

I'd love it if any readers or passer-by's could point me in the right direction on the error above.

So, I've noticed lately that one of my more popular blog entries has taken on some spam. I'll work on cleaning that up and implementing countermeasures.

In the mean time... I'm alive! Been working my tail off in preparation for Valentines day at work. It's the busiest time of the year for us. Just got back from VA where another sysadmin and I installed 50 new servers. It went well! Wrote some python to convert my inventory list from csv to our flat file format.

I've also been engineering a new syslogging system for our shop. It's a combination of a shell script that does archiving and checksum accounting, Splunk, and syslog-ng. Good times.

I've also had the (dis)pleasure of building RPM's lately....Not my favorite past-time. I really _really_ wish I could convince our team that the massive undertaking involved in converting from CentOS to debian would be worth it :). Not that I don't like CentOS, but I do love me some Debian.

That's all for now!