Crocodile hunters versus zoo keepers: Aspiring data scientists should speak python

We're hiring data scientists at Jawbone, which means I've been spending a lot of time reviewing resumes and interviewing candidates.  Making the "bring him/her in" or "pass" decision dozens of times an hour has been a great opportunity to focus on the features that make great data science candidates stand out from the crowd.

                       

What I've learned: Python is a great differentiator for job-hunting data scientists.  In the mix of resumes I review, it's the single biggest thing I look for.  Putting python high on your resume says, "Not only do I grok statistics/machine learning/R/SQL/Matlab/Octave/d3/etc, I can build the pipes and plumbing that make data systems work."

When I see resumes with strong background in practical programming, I think "Crocodile Hunter."  When those things are missing, I think "zoo keeper."  Zoo keepers aren't bad, but I find they need hand-holding every time you go into the jungle to wrangle wild data.

For whatever reason*, I find python sends a stronger signal of these skills than Java, C++, or any other mainstream programming language.  Not having python on your resume is a big handicap in the review process -- not an absolute deal-breaker, but in practice we hardly ever end up interviewing candidates without at least basic python skills.

With that said, "python" is an awfully big field to survey.  What python libraries are the most useful, really?  I've been asked this question several times recently.  Here's what I said in a recent email:

For python, I'd highly recommend installing ipython*, and getting your feet wet with the pandas library.  The matplotlib, json, and requests libraries are also good places to know your way around.  numpy and scipy have good stuff in them, but they're so huge that it's hard to really "know" them.  boto is also worth knowing, but it's only useful if you have a subscription to Amazon Web Services (AWS). 

For hadoop, I'd look specifically at mrJob.  It's a python-based wrapper that makes hadoop easier to use.  mrJob is too new to be covered in books yet, but the online documentation is pretty good.  You can use it in test mode even without installing hadoop.

Disclaimer: I know know that talking up one language over another is one of the cardinal sins of code. I don't want to start a religious war here -- I'm not saying that python is better than C++, or that pandas is the only way to manipulate data in python (although I believe Wes McKinney deserves a platinum-plated "better than sliced bread" award) -- I'm just saying that in the rough-and-tumble of the hiring process, data scientists with real python skills have a big advantage. 

Aspiring data scientists, you have been warned.

Practicing data scientists, what would you add to this list?  I've plugged in my favorite modules, but I'm sure there are others also worth a mention.

*I freely admit that the reason may just be in my head.  However, this preference for python also seems to be in a lot of other people's heads as well.  Whether there's a real reason for it or it's just an cognitive bias, it's working as a gating factor for a lot of resumes.

Sadly, I'm not developing tengolo. Would you like to run with it?

A little over a year ago, I announced that I was working on tengolo, a python alternative to netlogo.  I'm not actively developing it -- didn't get much farther than a rough proof-of-concept, really -- but I still get questions about the package.

As a result, I find myself writing some variation on the email below at least a couple times a month:

Dear python/ABM enthusiast - 

Glad you're interested in python and ABMs.  I started on tengolo after a thorough search turned up no good ABM frameworks in python.  I worked on it for a short while, then moved on when my dissertation committee told me to focus on stuff that would actually help me graduate. :) 

I got far enough in to be confident that a python-based ABM framework like tengolo could work. All the code is in the github repository, and every month I get questions from people asking if it's being actively developed. There's clearly demand for the project, but I don't have time to support it at this point.  I'd love to see someone take this ball and run with it. 

Best,
Abe

Would you like to run with this project?  If you're good with python and want to run a potentially popular academic open-source project, tengolo would be a great fit.  Please get in touch, and I will happily direct potential users and collaborators in your direction.

Python multiprocessing: 8x speed in 8 lines of code

At work last week I demo'ed some python parallel processing tricks.  Nothing fancy -- just standard usage of the multiprocessing library -- but these things can be a revelation if you haven't seen them before.

Like many things, python makes basic multiprocessing very easy: ~8 more lines of code can let you use all 8 cores of your laptop.  In practical terms, it's lovely to improve your workflow from "run algorithm preprocessing overnight" to "run algorithm preprocessing during lunch."

Here's how it's done.

from multiprocessing import Pool

def my_function( a_single_argument ):

    # e.g. "accepts a filename and returns results in a tuple"

    ...

my_data_list = [...]

my_pool = Pool()

my_results = my_pool.map( my_func, my_list )

That's all it takes.  A few tips:

First, the mapped function can only accept a single argument.  This is usually pretty easy to solve, by wrapping the arguments as a tuple:

def my_function( threeple_arg ):

    arg_1, arg_2, arg_3 = threeple_arg

    ...

Second, debugging in multiprocessing is a pain.  I often invoke the function this way first for debugging, then switch to multiprocessing once I know everything works:

#my_results = [my_func(item) for item in my_data_list]

It's a little hacky, but I'll often leave the line as a comment throughout development, switching between serial and parallel processing as need demands.

Last, a hint on Pool: you can pass an integer to the initialization routine to tell it how many subprocesses to use:

my_pool = Pool(5)

If you omit the argument, python assumes you want to run as many subprocesses (not threads!) as you have cores (e.g. 8 on a MacBooks pro).  If you want to save some processing power for other tasks, you might want to specify something lower, like 6.

If you're running tasks with high latency (e.g. web spidering, or lots of disk read/writes across a network) it sometimes makes sense to use more subprocesses than you have cores.  For example, I'll often throw 40 pool workers at a quick web-scraping script, just to speed things up.  However, if performance really matters, Pools with latency are very hard to tune and scale.  For anything more than a one-off data grab, you'll be better off with a queue-based tool, like scrapy, Amazon SQS, or celery.

HTH

Python mapreduce on EC2

Last week, I wrote about getting AWS public datasets onto an EC2 cluster, and then into HDFS for MapReduce.  Now let's get to hello world (or rather, countWords) with python scripts.

#!/usr/bin/env python
# mapper2.py

import sys, re
 
for line in sys.stdin:
    line = line.lower()
    words = line.split()
 
    #--- output tuples [word, 1] in tab-delimited format---
    for word in words: 
        print '%s\t%s' % (word, "1")

Here's the reducer script....

#!/usr/bin/env python
# reducer.py
 
import sys
 
# maps words to their counts
word2count = {}
 
# input comes from STDIN
for line in sys.stdin:
    # remove leading and trailing whitespace
    line = line.strip()
 
    # parse the input we got from mapper.py
    word, count = line.split('\t', 1)
    # convert count (currently a string) to int
    try:
        count = int(count)
    except ValueError:
        continue
 
    try:
        word2count[word] = word2count[word]+count
    except:
        word2count[word] = count
 
# write the tuples to stdout
# Note: they are unsorted
for word in word2count.keys():
    print '%s\t%s'% ( word, word2count[word] )

The command to execute all this in hadoop is a bit of a monster, mainly because of all the filepaths.  Note the usage of the -file parameter, which tells hadoop to load files for use in the -mapper and -reducer arguments. Also, I set -jobconf compression to false, because I didn't have a handy LZO decompresser installed.

bin/hadoop jar contrib/streaming/hadoop-0.19.0-streaming.jar -input wex-data -output output/run9 -file /usr/local/hadoop-0.19.0/my_scripts/mapper2.py -file /usr/local/hadoop-0.19.0/my_scripts/reducer.py -mapper mapper2.py -reducer reducer.py -jobconf mapred.output.compress=false

NB: As I dug into this task, I discovered several pretty good python/hadoop-streaming tutorials online.  The scripts here were modified from:  http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_3.2_--_Using_Your_Own_Streaming_WordCount_program


Other sources:

    http://www.protocolostomy.com/2008/03/20/hadoop-ec2-s3-and-me/
    http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
    http://www.larsgeorge.com/2010/10/hadoop-on-ec2-primer.html
    http://www.princesspolymath.com/princess_polymath/?p=137
    http://arunxjacob.blogspot.com/2009/04/configuring-hadoop-cluster-on-ec2.html

    http://wiki.apache.org/hadoop/AmazonS3

    http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html

    http://www.cloudera.com/blog/2009/07/advice-on-qa-testing-your-mapreduce-jobs/
    http://hadoop.apache.org/common/docs/r0.20.2/streaming.html

Running mapreduce on Amazon's publicly available datasets with python

on Monday, I had a preliminary interview at a really interesting tech startup. In the course of the conversation, the interviewer mentioned that he'd used some of the technical notes from compSocSci in his own work.  And I thought nobody was reading!

Anyway, I've been sitting on some old EC2/hadoop/python notes for a while.  The talk gave me the motivation to clean up and post them, just in case they can help somebody else.  The goal here is threefold:

1. Fire up a hadoop cluster on EC2
2. Import data from an EBS volume with one of AWS' public data sets
3. Use hadoop streaming and python for quick scripting

In other words, we want to set up a tidy, scalable data pipeline as fast as possible.  My target project is to do word counts on wikipedia pages -- the classic "hello world" of mapReduce.  This isn't super-hard, but I haven't seen a good soup-to-nuts guide that brings all of these things together.

Phase 1:
Follow the notes below to get to the digits-of-pi test.  Except for a little trouble with AWS keys, this all went swimmingly, so I see no need to duplicate.  If you run into trouble with this part, we can troubleshoot in the comments.

    http://wiki.apache.org/hadoop/AmazonEC2#Running_a_job_on_a_cluster
    http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

Phase 2:
Now let's attach an external dataset.  Here's the dataset we'll use:  Wikipedia Extraction (WEX).  It's a processed dump of the English language Wikipedia, hosted publicly on Amazon Web Services under snapshot ID snap-­1781757e.

This dataset contains a dump of 1,000 popular English wikipedia articles.  It's about 70GB.  At Amazon's $.12/GB rate, maintaining this volume costs about $8 for a whole month -- cheap!  If you want to scale up to full-size wikipedia (~500GB), you can do that too.  After all, we're in big data land.

Here's the command sequence to create an EBS volume for this snapshot and attach it to an instance. You can look up the ids using ec2-describe-volumes and ec2-describe-instances, or get them from the AWS console at https://console.aws.amazon.com.  (Hint: they're not vol-aaaaaaaa and i-bbbbbbbbb.)

    ec2-create-volume -snapshot snap-1781757e -z us-east-1a
    ec2-attach-volume vol-aaaaaaaa -i i-bbbbbbbb -d /dev/sdf

It took a while for these commands to execute.  Attaching the volume got stuck in "attaching" status for several minutes.  I finally got tired of waiting and mounted the volume, and then the status switched right away.  Can't say whether that was cause-and-effect or coincidence, but it worked.

Once you've attached the EBS volume, login to the instance (instructions here) and mount the volume as follows. This should be pretty much instantaneous.


    mkdir /mnt/wex_data
    mount /dev/sdf /mnt/wex_data

Now import the data into the Hadoop file system:

    cd /usr/local/hadoop/
    hadoop fs -copyFromLocal /mnt/wex_data/rawd/freebase-wex-2009-01-12-articles.tsv wex-data

If you want, you can now remove and delete the EBS volume.  The articles file is stored in the distributed filesystem across  your EC2 instances in  you hadoop cluster.  The nice thing is that you can get to this point within less than an hour, meaning that you only have to pay a tiny fraction of the monthly storage cost.


    ec2-detach-volume vol-aaaaaaaa -i i-bbbbbbbbb -d /dev/sdf
    ec2-delete-volume vol-aaaaaaaa

I had some trouble detaching volumes until I used the force flag: -f.  Maybe I was just being impatient again.

That's enough for the moment.  I'll tackle python in my next post.

Getting started with django, heroku, and mongolab

Long post here -- I just spent a couple hours going from zero to mongo with django and heroku.  Here are my working notes.

For this test, I started from a pure vanilla django project:

django-admin startproject testproj

I made a single edit to settings.py, adding "gunicorn" to the list of INSTALLED_APPS.

For setting up django on heroku, I basically followed the description given in these very good notes by Ken Cochrane.  I've done this before and there were no surprises, so I'll skip the stuff about setting up heroku and git.  I did nothing special except install gunicorn.

Here's my Procfile:

web: python testproj/manage.py collectstatic --noinput; python testproj/manage.py run_gunicorn -b "0.0.0.0:$PORT"

This is overkill at the outset--we don't have any static files, so collecting them doesn't do anything.

And my requirements.txt:

Django==1.3.1
gunicorn==0.13.4
psycopg2==2.4.4
simplejson==2.3.2

That got me to the famous django opening screen, live on heroku:

It worked!
Congratulations on your first Django-powered page.

Just to emphasize that I've done nothing special so far, here's the tree for the git repo I'm using.  (I've suppressed .pyc and *~ files.)

    .
    ├── Procfile
    ├── requirements.txt
    └── testproj
        ├── __init__.py
        ├── manage.py
        ├── settings.py
        └── urls.py

(Come to think of it, there should probably be a .gitignore file in there as well.)

Let's pick up from the point.

I added the mongolab starter tier (free up to 240 MB) to the heroku app.  I did this from the heroku console, because I wanted to see what options were there.  (Oddly, none of the dedicated tiers show up there.)  In the future, I'll probably just use the command line:

heroku addons:add mongolab:starter

Next, I followed the instructions on heroku's documentation, and grabbed the URI:

heroku config | grep MONGOLAB_URI

The MONGOLAB_URI is in the following format:

mongodb://username:password@host:port/database

At this point, heroku's documents stopped being much help, because they don't cover python.  So I switched over here instead.  I wanted to understand all the steps, so I refrained from copy-pasting.

I installed a few supporting libraries

pip install pymongo django-mongodb-engine djangotoolbox

And added the appropriate lines to requirements.txt

pymongo==2.1.1
django-mongodb-engine==0.4.0
djangotoolbox==0.9.2

I then configured the database in settings.py.

DATABASES = {
    'default': {
        'ENGINE': 'django_mongodb_engine',
        'NAME': 'heroku_app1234567',
        'USER': 'heroku_app1234567',
        'PASSWORD': 'abcdefghijklmnopqrstuvwxyz',
        'HOST': 'ds031117.mongolab.com',
        'PORT': '31117',
    }
}

Note: With versions of django-mongodb-engine over 0.2, ENGINE should be 'django_mongodb_engine', not 'django_mongodb_engine'.  Get this wrong and you'll see something like:

django.core.exceptions.ImproperlyConfigured:
'django_mongodb_engine.mongodb' isn't an available database backend.
Try using django.db.backends.XXX, where XXX is one of:
    'dummy', 'mysql', 'oracle', 'postgresql', 'postgresql_psycopg2', 'sqlite3'
Error was: No module named mongodb.base

I never did anything about the "current bug" that Dennis mentions.  Apparently it's been patched, whatever it was.

Quick check: so far, running locally (./manage.py runsersver) and pushing to heroku both work.  The next step is to add a model or two.  Time to fix that:

django-admin startapp onebutton

Here's models.py:

from django.db import models

class ButtonClick( models.Model ):
    click_time = models.DateTimeField( auto_now=True )
    animal = models.CharField( max_length=200 )

I was going to build a stupid-simple app with a single button to "catch" animals from a random list and store them in the DB, but it's getting late, so let's just jump to the proof of concept.

Going straight to the django shell on my computer...

$python manage.py shell
>>> from testproj.onebutton.models import ButtonClick as BC
>>> c1 = BC( animal="Leopard" )
>>> c1.save()
>>>

It works!  Open up the mongolab console (through the add-ons tab in heroku) and the database shows a single record in onebutton_buttonclick collection.

We still haven't written views and templates, and (more importantly) validated that the heroku app can talk to the DB over at mongolab, but I'm going to call this good enough for now.  Mission accomplished.

Q&A about web scraping with python

Here are highlights from a recent email exchange with a tech-minded friend in the polisci department. His questions were similar to others I've been fielding recently, and very to-the-point, so I thought the conversation might be useful for others looking at getting into web scraping for research (and fun and profit).

> ... I have an unrelated question: What python libraries do you use to scrape websites?

urllib2, lxml, and re do almost everything I need. Sometimes I use wget to mirror a site, then use glob and lxml to pull out the salient pieces. For the special case of directed crawls, I built snowcrawl.


> I need to click on some buttons, follow some links, parse (sometimes ugly) html, and convert html tables to csv files.

Ah. That's harder. I've done less of this, but I'm told mechanize is good. I don't know of a good table-to-csv converter, but that's definitely a pain point in some crawls -- if you find anything good, I'd love to hear about it!

It strikes me that you could do some nice table-scraping with cleverly deployed xpath to pull out rows and columns -- the design would look a little like functional programming, although you'd still have to do use loops. Python is good for that, though.


> Is mechanize the way to go for browsing operations?

From hearsay, yes.

> What's your take on BeautifulSoup vs. lxml vs. html5lib for parsing?

I got into lxml early and it works wonderfully. The only pain point is (sometimes) installation, but I'm sure you can handle that. My impression is that lxml does everything BeautifulSoup does, faster and with slightly cleaner syntax, but that it's not so much better that everyone has switched. I don't know much about html5lib.

> Should I definitely learn xpath?

Definitely. The syntax is quite easy, very similar to jquery/css selectors. It also makes for faster development: get a browser plugin for xpath and you can test your searches directly on the pages you want to crawl. This will speed up your inner loop for development time tremendously -- much better than editing and re-running scripts, or running tests from a console.

HTH.  Cheers!

Don't use netlogo! (4)

Quick search on simplyhired:

Jobs with python in the description: 26,549
Jobs with netlogo in the description: 1

Yes, I'm being snarky about netlogo.  But if you're a person with any talent or ambition, why waste it on a "skill" that has no practical value?

Announcing Tengolo, a python alternative to Netlogo!

Yesterday I wrote about discussion/fallout from my argument against using NetLogo for agent-based modeling.  Today, contra the spirit of Mark Twain's "everyone complains about the weather, but nobody does anything about it,"  I want to give would-be modelers a constructive alternative to NetLogo.

Announcing Tengolo, an open-source library for agent-based modeling in python and matplotlib!  Tengolo is open source, and currently hosted at github.  Preliminary documentation is here.

Tengolo is designed to allow users to

1. Quickly express their ideas in code,
2. Get immediate feedback from the python shell, matplotlib GUI, and logs, AND
3. Scale up the scope of their experiments with batches for repeated trials, parameter sweeps, etc.

Tengolo is designed to do scratch the same itch as netlogo, without making users debase themselves with the ridiculously backwards Logo programming language.  Instead, they can use python's clean syntax and enormous codebase to develop their models.

For the most part, the advantages of Tengolo are the advantages of python and matplotlib:

• Clean, object-oriented code for
• Quick learning
• Rapid prototyping
• Easy debugging
• Great maintainability
• An enormous codebase of snippets and outside libraries. 
• An active and supportive user community
• Powerful, professional graphs and plots with a minimum of hassle
• An intuitive GUI that lets you interact with your model in real time

Development so far
I've just begun development -- about 8 hours of work -- but the advantages are already starting to show.  As a proof of concept, here's a screen shot and the script for a model I'm building in Tengolo.  All told, the script is only 80 lines long, and does not contain a single turtle.

#!/usr/bin/python
"""
P-A model simulator for mixed motives paper
    Abe Gong - Feb 2102
"""

import numpy
import scipy.optimize

from tengolo.core import TengoloModel, TengoloView
from tengolo.widgets import contour, slider

class M4Model(TengoloModel):
    def __init__(self):
        self.beta  = .5
        self.x_bar = 2
        self.a     = 0
        self.b     = 1
        self.alpha = .5
        self.c     = .1

        delta = 0.025
        self.v = numpy.arange(0.025, 10, delta)
        self.w = numpy.arange(0.025, 10, delta)
        self.V, self.W = numpy.meshgrid(self.v,self.w)

        self.update()

    def update(self):
        self.U = self.calc_utility( self.V, self.W, self.linear_rate )
        (self.v_star, self.w_star) = self.calc_optimal_workload( self.linear_rate )
        (self.v_bar, self.w_bar) = self.calc_optimal_workload( self.flat_rate )
        print (self.v_star, self.w_star)

    def calc_utility(self, v, w, x_func):
        z = (v**(self.beta))*(w**(1-self.beta))
        x = x_func(z)

        u = self.alpha*numpy.log(v) + (1-self.alpha)*numpy.log(x) - self.c*(w+v)
        return u

    def calc_optimal_workload(self, x_func):
#        return scipy.optimize.fmin( lambda args : -1*self.calc_utility( args[0], args[1], x_func ), [2,2], disp=False )
        result = scipy.optimize.fmin_tnc( lambda args : -1*self.calc_utility( args[0], args[1], x_func ),
                [2,2],
                bounds = [(0,None),(0,None)],
                approx_grad=True,
                disp=False,
            )
        return result[0]

    def flat_rate(self, z):
        return self.x_bar

    def linear_rate(self, z):
        return self.x_bar + self.a + self.b*z



#Initialize model
my_model = M4Model()

#Initialize viewer
my_view = TengoloView(my_model)

#Attach controls and observers
my_view.add_observer( contour, [0.15, 0.30, 0.70, 0.60], args={
    "title":"Utility isoquants",
    "xlabel":"v (hrs)",
    "ylabel":"w (hrs)",
    "x":"V",
    "y":"W",
    "z":"U",
})
my_view.add_control( slider, "alpha", [0.15, 0.15, 0.70, 0.03], args={"range_min":0, "range_max":1} )
my_view.add_control( slider, "beta", [0.15, 0.10, 0.70, 0.03], args={"range_min":0, "range_max":1} )

#Render the view
my_view.render()

This particular model is game-theoretic, not agent-based, but the process of model design is essentially the same.  I want to be able to build and edit the model, and get results as quickly as possible. The process should be creative, not bogged down with debugging. Along the way, I need to experiment with model parameters, and quickly see their impact on the behavior of the system as a whole.

As I said earlier, I've only just started down this road, but there's no turning back.  If you have a model you'd like to port to Tenlogo, let me know.  Cheers!

A presentation on practical approaches to picking programming packages

Last week, I helped kickstart the semester for the Complex Systems grad student group with a presentation on "principles and practices for picking powerful programming platforms, packages, and plugins."

It was a fun talk to give, because 1) it's something I've done many times, 2) it's something that a lot of the other students (and faculty) are affected by, and 3) most people haven't thought about it in great detail. Low-hanging fruit.

I probably came across as a bit of a pythonista. This was partly deliberate: most of the students in Complex Systems learn Java, because that's what's taught in the intro classes, and never get much exposure to other options. I wanted people to know they don't have to spend the rest of their lives typing extraneous semicolons. Or worse -- developing in turtles.

Plus, I am actually, in reality, a bit of a pythonista, so it came naturally.

Here are the slides. They sparked a good discussion. What would you add?

Picking programming packages

View more presentations from Abe Gong.

How to download every episode of This American Life

I recently discovered this lovely script by one Sean Furukawa.  (I don't know him.)  The script downloads every episode of This American Life in mp3 format.  This American Life is far and away my favorite radio show -- Ira Glass is consistently the best storyteller on air.

Anyway, I'm intimidated by perl (too hard too read, too many nonalphnumeric characters) so I rewrote the script in python. The first run will take a long-ish time, since it's downloading all 450+ existing episodes of the program. Subsequent executing of the script will be faster, since it only has to download new episodes. Enjoy!

By the way, AFAIK, this type of webcrawling is completely legal.  The content is already streamable from the TAL website; you're just downloading it er, a little faster than usual.

That said, if you use this script, I'd recommend making a tax-deductible contribution to This American Life -- it's a great program, worthy of support.  The "donate" button is in the upper-right corner of the This American Life webpage.

#!/usr/bin/python 

# Adapted from: http://www.seanfurukawa.com/?p=246
# Translated from perl to python by Abe Gong
# Dec. 2011

import urllib, glob, datetime

def now():
 """Get the current date and time as a string."""
 return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

def log( S ):
  """Write a line to the log file, and print it for good measure."""
  logfile.write(S + '\n')
  print S

#Start up a log file
logfile = file( 'tal_log.txt', 'a' )

#Load all the episodes that have already been downloaded; keep the filenames in a list
episodes = [ f.split('/')[-1] for f in glob.glob('episodes/*.mp3') ]
#print episodes

#As of today (12/11/2011) there are 452 episodes, so a count up to 500 should last a long while.
for i in range(1,500):

 #Choose the appropriate filename
  filename = str(i)+'.mp3'
  #Add the URL prefix
  url = 'http://audio.thisamericanlife.org/jomamashouse/ismymamashouse/'+filename
  
  #Check to see is the file has already been downloaded
  if not filename in episodes:
    #Log the attempt
    log( now() + '\ttrying\t' + url )
    
    #Try to download it
    code = urllib.urlopen( url ).getcode()
    if code == 200:
      urllib.urlretrieve( url, filename='episodes/'+filename )
    
      #Log the result -- success!
      log( now() + '\tsaved\t' + filename )
    else:
      log( now() + '\tfile not found' )

snowCrawl bug fixes and easier downloading

I've made a few small fixes to the snowCrawl python library. (See the updates page for details.)

The most important change is that I've created zipped and tar-gzipped versions of the repository for download. Now you can get snowcrawl even if you don't speak subversion. See the downloads page for one-click downloads.

Python html cleaner

A friend asked for help cleaning html files today --- getting rid of scripts, styles, tags, and whatnot.

Here's a quick 14-line python script to do the job. It takes all the html files in the ./docs directory and writes them out as clean text to the ./output directory.

Note the lxml dependency!

from lxml.html import clean
import glob, re

cleaner = clean.Cleaner( style=True, scripts=True, comments=True, safe_attrs_only=True )

filenames = glob.glob('docs/*')
for f in filenames:
    print '='*80
    text = file(f,'r').read()
    text = cleaner.clean_html( text )    #Remove scripts, styles, etc.
    text = re.sub('<.*?>', '', text )    #Remove html tags
    text = re.sub('\s+', ' ', text )    #Remove whitespace
    print text
    file( 'output/'+f.split('/')[-1].split('.')[0]+'.txt', 'w').write( text )