A map of me: Life-tracking with funf

I've been running funf ever since I arrived in California -- almost six months now.  If you haven't seen it before, funf is a great little android app for passive life logging.  It can track anything your phone can track: GPS, accelerometer, battery, text messages, etc.

The downside is that it's buggy.  I ended up having to use a very hacky workaround solution to get my data off: exporting data via email, downloading the zips, then running the funf_analyze_mac script to parse them all to sql. It's a clunky pipeline, but works for the moment.

Here's the first payoff: a map of everywhere I went between November and February. It's pretty neat to be able to see the neighborhood where I work in San Francisco, the train line along the edge of the bay, the two cities where I've lived since moving here, and a few trips south and west to Los Gatos and San Jose.

I haven't done much work to make this beautiful, but I find it very engaging anyway.  (It's my data, after all.)

My next plan is to write a script to isolate places where I go often or spend a lot of time, and then mash those locations up with data from other sources based on timestamps.

Design patterns for data-centric software

I wrote a few days ago about software design patterns, including the thought that we're going to discover new patterns for data-centric software.  Let me unpack that concept.

First, by data-centric software, I don't mean software intended for data analysis (e.g. R, excel, or google charts).  I mean any software that collects and/or responds to data in the course of doing whatever else it does.

Web analytics are a great example of this.  The primary purpose of a web page is to serve content.  But at the same time, it's easy to track pageviews and traffic.  Compared to an untracked web site, a site instrumented with google analytics is more data-centric, because it's generating data in the background.

As I read it, the original design patterns are intended mainly to minimize long-term development costs.  The key question is "How should code be structured to make it easy to read, debug, maintain, extend, etc?"  It's all about saving developers' time in the long run.*

Five years after the original set of design patterns was popularized, another book was published, focusing on design patterns for distributed software.  This time, the key questions expanded to include bandwidth and concurrency: "How should we structure code to make the best use out of distributed computing resources?"

I think we're due for another expansion, because data-centric code introduces another optimization target: useful information.**  Just as the list of patterns expanded to deal with networking and multiprocessing, it will expand again as data processing and analytics become integral to software design.***

Off the top of my head, here's a quick list of data-centric patterns.

• A/B testing
• Funnel analysis
• Recommender systems (very broad category!)
• Top hits (most visited, emailed, etc.)
• Automatic bug reports
• Likes, +1s, Retweets

This list isn't complete, and it's clear that best practice is still evolving.  For example, A/B testing has been industry-standard for a long time, but I recently read a good argument that a multi-armed bandit algorithm is better than A/B testing, because it gathers all the same information, plus integrating that feedback directly into the site design.  It's a very natural extension and improvement over an older data-centric design pattern.  I'm sure that many other such improvements are possible.

Anyway, I think it's still too early to try to write a comprehensive list.  But I'd still like to expand this list to cover as many cases as possible.  What else belongs here?

*A few of the patterns address things like limited memory and processing power, but they're the exceptions.

** Defining useful opens up a whole new can of worms, which I won't get into here.

***This relates back to the concept that I've written about before: software design for analytics.

Bay Area data science people, events

A quick favor: I'm headed out to Palo Alto for a family event in a couple weeks.  While I'm there, I'd love to meet people and find out more about the Bay Area data science scene.

Where should I go?  Who should I meet?

I'm free mainly on Monday the 11th through Wednesday the 13th, with some time on Tuesday evening here.

This picture is the first result for gImages: "going to the big city."  I like it.

Summer Course in Computational Models of Political and Social Events

From the Polmeth listserv:

 

Summer Course in Computational Models of Political and Social Events

Claudio Cioffi-Revilla <ccioffi@GMU.EDU>
April 03, 2009 @ 02:24:20

Message

Kindly circulate among colleagues and students interested in computational
social science and/or social simulation:

http://lipari.cs.unict.it/LipariSchool/SocialScience/

The Lipari International School on Computational Social Sciences will take
place this summer, July 18 - 25, 2009, on the Mediterranean island of Lipari
just north of the coast of Sicily, Italy. Researchers interested in the
emerging field of computational social science, especially but not
exclusively computational political science theory and simulation models,
are encouraged to consider this as an opportunity to learn more and become
acquainted with a variety of research frontiers.


--
Claudio Cioffi-Revilla, Ph.D.

Professor of Computational Social Science
NAS Jefferson Science Fellow
Director, Center for Social Complexity
Krasnow Institute for Advanced Study, George Mason University
Research-1 Bldg MS 6B2, 4400 University Drive, Fairfax, VA 22030 U.S.A.
tel (703) 993-1402, fax (703) 993-1399, ccioffi@gmu.edu
Research & Teaching http://socialcomplexity.gmu.edu
http://gazette.gmu.edu/articles/11458/
 MASON Project http://cs.gmu.edu/~eclab/projects/mason/

"All truths are easy to understand once they are discovered; the difficulty
is to discover them."--Galileo Galilei

**********************************************************
             Political Methodology E-Mail List
   Editors: Melanie Goodrich, <melaniegoodrich@nyu.edu>
                    Xun Pang, <xpang@wustl.edu>
**********************************************************
        Send messages to polmeth@artsci.wustl.edu
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************

Key skills for job-hunting data scientists

There's a lot of buzz around data science, but (as I've posted about previously) the term is still murky.

One way to get a look at the emerging definition of data science is to search for "data science" jobs, and see what they have in common: What are the key skills for data scientists?

This wordle sounds like Romney: "jobs jobs jobs..."

I took a few minutes today to run that search -- automated, of course.  Nothing rigorous or scientific, but the results are still plenty interesting.

Methods:  I searched "data scientist" on simplyhired.com, then scraped the ~250 resulting links.  All of the non-dead links (there weren't many dead ones) returned a job posting, usually on a company page, occasionally on another aggregator.  I grabbed the html of each of these pages, and cleaned the html to get rid of scripts, styles, etc.  I didn't do any fancy chrome or ad scraping, so take the results with a grain of salt.

First, I generated the obligatory word cloud.  Thank you, wordle.


Then I skimmed a dozen of the pages, looking for keywords that seemed to pop up a lot: java, hadoop, python.  For the most part, I focused on specific skills that companies are explicitly hiring for.  I also tossed in a few other terms, just to see what would happen.

Here are counts of jobs mentioning keywords (Not keyword counts -- the number of separate job postings that include at least one reference to a given keyword.):

198	data
131	statist
130	java
107	hadoop
85	mining
68	python
49	visuali
39	cloud
37	mapreduce
35	c\+\+
24	amazon
22	ruby
18	bayes
15	ec2
13	jquery
13	fun
3	estimat

Evidently, data science postings put equal value on "fun" and "jquery."

Also, at a glance, Java beats python, beats C++ in terms of employability.  It kind of makes me wish I'd been nicer to Java all these years.

One clear finding is that hadoop and MapReduce skills are in high demand.  That's not news to anyone working in this area, but I was surprised at just how many jobs were looking for these skills.  Almost half (107 of 238, 45%) of total job postings explicitly mention hadoop.

That percentage seems slightly out of whack to me, because there are plenty of valuable ways to mine data without using a MapReduce algorithm.  Maybe Hadoop is a pinch point in the job market because there just aren't enough MapReduce-literate data miners out there?  If that's the case, I would expect demand to come down (relative to supply) in the not-too-distant future -- MapReduce isn't that hard to learn.

Alternatively, there could be some bandwaggoning going on:

"Data is the Next Big Thing.  We need to hire a data person."

"What exactly is a data person?"

"I don't know, but I hear they know how to program in 'ha-doop,' so put that in the posting."

As a third explanation, it may be that the meaning of "data science" is narrowing.  Instead of encompassing all the things that can be done with data, perhaps it's coming to mean "mapreduce."  If that's the case, then "data science" jobs would naturally include hadoop/mapreduce skills.  IMO, that would be sad, because it would be an opportunity missed to change the way data flows into decisions in a more systemic way.

I'd be interested in hearing other explanations for the dominance of hadoop.  Also, if you have other queries to run against this data set, I'm happy to try them out.  What I've put up so far is just back-of-envelope coffee-break stuff.

More about political microtargeting as data science

Two interesting articles about how campaiging is being influenced by data science.

From Slate: Anatomy of a narrow victory: Romney's Iowa win took a lot more than money

And from BusinessWeek (HT Dad): OBAMA Campaign’s Secret Weapon: Geeks.

(Photo from BW.  Love the github ref.)

A few observations on data versus money, data-driven decision making, and whether we really want more data in politics...

Synonyms for "computational social science"

The other day, a friend asked me how to find other "computational social scientists."  Good question.  I wish I had a good answer -- I might be less bored at conferences.

The big data movement is pulling people together from many different backgrounds, and they're still working out a common language to explain what they do.  If I had to guess, either "data science" or "big data" is going to be the label that sticks, but I'm sure there will be plenty of offshoots, like "computational social science."


To get some new ideas, I took a few of my favorite near-synonyms and ran them through google's keyword tool for adwords.  Then I branched out and searched a little by hand.  Low-tech, but it works.  Here are some of the terms that popped to the top.

• computational social science
• data scientist
• analytics
• data mining
• business intelligence
• visualization
• big data
• knowledge discovery
• text mining
• network analysis
• social media analytics
• social network mining
• predictive analytics
• social analytics
• business analytics
• ecommerce analytics
• machine learning
• statistical programming
• predictive modeling

I also found a bunch of articles hyping the emergence of a new data-oriented field.  This stuff makes me feel good about job prospects, so I'll pass a couple along here:

What is Data Science?
Jobs For Data Scientists Explode across the Market