Python multiprocessing: 8x speed in 8 lines of code

At work last week I demo'ed some python parallel processing tricks.  Nothing fancy -- just standard usage of the multiprocessing library -- but these things can be a revelation if you haven't seen them before.

Like many things, python makes basic multiprocessing very easy: ~8 more lines of code can let you use all 8 cores of your laptop.  In practical terms, it's lovely to improve your workflow from "run algorithm preprocessing overnight" to "run algorithm preprocessing during lunch."

Here's how it's done.

from multiprocessing import Pool

def my_function( a_single_argument ):

    # e.g. "accepts a filename and returns results in a tuple"

    ...

my_data_list = [...]

my_pool = Pool()

my_results = my_pool.map( my_func, my_list )

That's all it takes.  A few tips:

First, the mapped function can only accept a single argument.  This is usually pretty easy to solve, by wrapping the arguments as a tuple:

def my_function( threeple_arg ):

    arg_1, arg_2, arg_3 = threeple_arg

    ...

Second, debugging in multiprocessing is a pain.  I often invoke the function this way first for debugging, then switch to multiprocessing once I know everything works:

#my_results = [my_func(item) for item in my_data_list]

It's a little hacky, but I'll often leave the line as a comment throughout development, switching between serial and parallel processing as need demands.

Last, a hint on Pool: you can pass an integer to the initialization routine to tell it how many subprocesses to use:

my_pool = Pool(5)

If you omit the argument, python assumes you want to run as many subprocesses (not threads!) as you have cores (e.g. 8 on a MacBooks pro).  If you want to save some processing power for other tasks, you might want to specify something lower, like 6.

If you're running tasks with high latency (e.g. web spidering, or lots of disk read/writes across a network) it sometimes makes sense to use more subprocesses than you have cores.  For example, I'll often throw 40 pool workers at a quick web-scraping script, just to speed things up.  However, if performance really matters, Pools with latency are very hard to tune and scale.  For anything more than a one-off data grab, you'll be better off with a queue-based tool, like scrapy, Amazon SQS, or celery.

HTH

Market failure in mechanical turk: Skimming and cherry-picking

This is the first in a series of three posts -- a trilogy! -- about pain points on Amazon's mechanical turk, from a requester's perspective.

I'm a frequent user of mturk.  I like the service, and spend a large fraction of my research budget there.  That means I also feel its limitations pretty acutely. Today I want to write about a problem that I've noticed on mturk: skimming and cherry picking.  (A few weeks ago, I complained about ubuntu.  Why is it that we only hurt the computing systems we love?)

Here's the problem: even within a batch, not all HITs are equally difficult. I've discovered that some workers (smart ones) will skim quickly through a batch and cherrypick the easy HITs. For instance, given a list of blog posts to read and evaluate, some turkers will skip the long ones and only code the short ones.

Individually, skimming makes perfect sense.  If you do, you can certainly make more dollars per hour.  As a bonus, you might even get a higher acceptance rate on your HITs, because short HITs lend themselves to unambiguous evaluation.  The system rewards strategic skimming.

But from a social perspective, skimming is counterproductive.  It wastes time overall, because time spent skimming is time not spent completing tasks*. It's not really fair to other workers. It wreaks havoc on many approaches for determining accuracy. (As a requester, I've experienced this personally.) From a scientific standpoint, it can also ruin the validity of some kinds of data collection.

I first ran into clear evidence of skimming over a year ago.  At first, I didn't want to say anything about it, because I didn't want to give anyone ideas.  At this point, I see it all the time.  One easy-to-observe bit of evidence: the hourly rate on most HITs will start high, and fall over time**.  This is because skimmers grab the quick, easy tasks first, leaving slower tasks for later workers.

I can't really blame turkers for approaching their work in a clever way.  Instead, I lay the blame on Amazon, for making counterproductive behavior so easy.

It's especially galling because it would be very easy to fix the problem. On the HIT design page, they should add a "Turkers can only accept HITs in the order they're presented" flag to each batch.  For tasks with this flag checked, turkers would be be shown one HIT at a time.  They'd be unable to view or accept others in the batch until they'd completed the HIT in front of them***. This would effectively deny turkers control over which HITs they choose to do within a batch****.  It would end the party for skimmers, but make the market more efficient overall.  A simple tweak to the market -- problem solved.

How about it Amazon?


* You can think about the social deadweight loss from skimming like this:
Let T be the total time all workers spend completing HITs.  Skimming doesn't change T -- the total amount of task work is constant. But skimming itself is time consuming. Let S be the deadweight loss due to skimming on a given batch.  Like T, the total wage for a given batch is also constant.  Call it W.

In aggregate, the effective hourly wage for the whole batch without skimming is W/T.  With any amount of skimming it is always less:  W/(T+S). So although skimming may improve the hourly wage of the most aggressive cherry pickers, on the whole it always hurts the hourly wage of the mturk market as a whole.

** Yes, yes -- I know that this is not an acid test: there are other explanations for hourly rates that decline over the life of a task.  Still, it's good corroborating evidence for an explanation that makes a lot of sense to begin with.

*** Only viewing one HIT at a time might make it harder for turkers to get a sense of what a given batch is like. There's a simple fix for this as well: allow turkers to see the next k tasks, where k is a small number chosen by the requester. This might make it harder to build a RESTful interface for turkers, though. I haven't thought it through in detail.

**** It's possible that requesters would abuse this power by doing a bait-and-switch: showing easy HITs first and then making them more difficult once workers have invested in learning the task. This seems like a minor concern---if the tasks get tough or boring, turkers can always vote with their feet. But if we're worried about it, there's an easy fix here as well: take control of the HIT sequence away from requesters, just like we took it away from workers. It would be very easy to randomize the order of tasks when the "no skimming" box is checked.  Or allow requesters to click a separate "randomize tasks" box, with Amazon acting as credible intermediary for the transaction.

Follow up on personal elephants

Three thoughts from the talk about elephants and motivation I posted yesterday...

First, I found a website that talks about Buddhist symbolism, including elephants.  The metaphor is perfect:

At the beginning of one's practice the uncontrolled mind is symbolised by a gray elephant who can run wild any moment and destroy everything on his way. After practising dharma and taming one's mind, the mind which is now brought under control is symbolised by a white elephant strong and powerful, who can be directed wherever one wishes and destroy all the obstacles on his way."

Second, the closing thought in the talk is from Kara S, about getting to know your personal elephant. Her comment reminded me of a conversation from Paulo Coelho's wonderful little book, The Alchemist:

    "My [elephant] is a traitor," the boy said to the alchemist, when they had paused to rest the horses. "It doesn't want me to go on."

    "That makes sense," the alchemist answered. "Naturally it's afraid that, in pursuing your dream, you might lose everything you've won."

    "Because you will never again be able to keep it quiet. Even if you pretend not to have heard what it tells you, it will always be there inside you, repeating to you what you're thinking about life and about the world."

    "You mean I should listen, even if it's treasonous?"

    "Treason is a blow that comes unexpectedly. If you know your [elephant] well, it will never be able to do that to you. Because you'll know its dreams and wishes, and will know how to deal with them."

    "You will never be able to escape from your [elephant]. So it's better to listen to what it has to say. That way, you'll never have to fear an unanticipated blow."

Coelho uses the word "heart," instead of "elephant," but I'm sure he won't mind the substitution.

Finally, I referenced a bunch of studies (in passing) in the talk, but didn't include any citations, because I ran out of time.  If you happen to have links/refs to articles, books, videos, etc. in this area, can you paste them into the comments?

If you haven't seen the original slides yet, I'll leave these as teasers.

How to ride, eat, tame, etc. your personal elephant

This is a talk I gave at the annual "Hill Street TED" activity that my local congregation puts together.  These are short talks in TED format, put together by members of the congregation to share aspects of their life and work that don't get talked about much at church.

Here, I've taken my slides from the talk, added a script, and revised the format to better fit the web.  After going back and forth, I left in the Mormon references, even though I know some of them will be lost in translation. I also added a few slides based on comments and feedback from people at the talk. This let me put in more details and one-off ideas that just didn't fit into 10 minutes.  Enjoy!

EDIT: Slideshare is giving me grief, so here are pdf versions of the talk with speaking notes, and without. 

View more documents from Abe Gong.