Showing posts with label django. Show all posts
Showing posts with label django. Show all posts

Wednesday, February 22, 2012

Getting started with django, heroku, and mongolab

Long post here -- I just spent a couple hours going from zero to mongo with django and heroku.  Here are my working notes.

For this test, I started from a pure vanilla django project:
django-admin startproject testproj

I made a single edit to settings.py, adding "gunicorn" to the list of INSTALLED_APPS.

For setting up django on heroku, I basically followed the description given in these very good notes by Ken Cochrane.  I've done this before and there were no surprises, so I'll skip the stuff about setting up heroku and git.  I did nothing special except install gunicorn.

Here's my Procfile:
web: python testproj/manage.py collectstatic --noinput; python testproj/manage.py run_gunicorn -b "0.0.0.0:$PORT"

This is overkill at the outset--we don't have any static files, so collecting them doesn't do anything.

And my requirements.txt:
Django==1.3.1
gunicorn==0.13.4
psycopg2==2.4.4
simplejson==2.3.2

That got me to the famous django opening screen, live on heroku:
It worked!
Congratulations on your first Django-powered page.

Just to emphasize that I've done nothing special so far, here's the tree for the git repo I'm using.  (I've suppressed .pyc and *~ files.)

    .
    ├── Procfile
    ├── requirements.txt
    └── testproj
        ├── __init__.py
        ├── manage.py
        ├── settings.py
        └── urls.py

(Come to think of it, there should probably be a .gitignore file in there as well.)

Let's pick up from the point.

I added the mongolab starter tier (free up to 240 MB) to the heroku app.  I did this from the heroku console, because I wanted to see what options were there.  (Oddly, none of the dedicated tiers show up there.)  In the future, I'll probably just use the command line:
heroku addons:add mongolab:starter

Next, I followed the instructions on heroku's documentation, and grabbed the URI:
heroku config | grep MONGOLAB_URI

The MONGOLAB_URI is in the following format:
mongodb://username:password@host:port/database

At this point, heroku's documents stopped being much help, because they don't cover python.  So I switched over here instead.  I wanted to understand all the steps, so I refrained from copy-pasting.

I installed a few supporting libraries
pip install pymongo django-mongodb-engine djangotoolbox

And added the appropriate lines to requirements.txt
pymongo==2.1.1
django-mongodb-engine==0.4.0
djangotoolbox==0.9.2

I then configured the database in settings.py.
DATABASES = {
    'default': {
        'ENGINE': 'django_mongodb_engine',
        'NAME': 'heroku_app1234567',
        'USER': 'heroku_app1234567',
        'PASSWORD': 'abcdefghijklmnopqrstuvwxyz',
        'HOST': 'ds031117.mongolab.com',
        'PORT': '31117',
    }
}

Note: With versions of django-mongodb-engine over 0.2, ENGINE should be 'django_mongodb_engine', not 'django_mongodb_engine'.  Get this wrong and you'll see something like:
django.core.exceptions.ImproperlyConfigured:
'django_mongodb_engine.mongodb' isn't an available database backend.
Try using django.db.backends.XXX, where XXX is one of:
    'dummy', 'mysql', 'oracle', 'postgresql', 'postgresql_psycopg2', 'sqlite3'
Error was: No module named mongodb.base

I never did anything about the "current bug" that Dennis mentions.  Apparently it's been patched, whatever it was.

Quick check: so far, running locally (./manage.py runsersver) and pushing to heroku both work.  The next step is to add a model or two.  Time to fix that:
django-admin startapp onebutton
Here's models.py:
from django.db import models

class ButtonClick( models.Model ):
    click_time = models.DateTimeField( auto_now=True )
    animal = models.CharField( max_length=200 )
I was going to build a stupid-simple app with a single button to "catch" animals from a random list and store them in the DB, but it's getting late, so let's just jump to the proof of concept.

Going straight to the django shell on my computer...
$python manage.py shell
>>> from testproj.onebutton.models import ButtonClick as BC
>>> c1 = BC( animal="Leopard" )
>>> c1.save()
>>>
It works!  Open up the mongolab console (through the add-ons tab in heroku) and the database shows a single record in onebutton_buttonclick collection.

We still haven't written views and templates, and (more importantly) validated that the heroku app can talk to the DB over at mongolab, but I'm going to call this good enough for now.  Mission accomplished.

Thursday, February 16, 2012

Working notes on cloud-based MongoDB with python

I've been thinking about getting into mongoDB for a good while.  I'm looking for a platform that works, scales, and integrates with python with a minimum of hassle.  Cheap would be nice too.

Tonight, Google and I sat down to do some nuts and bold research.  Here are my notes.  Have anything to add?


PS - Based on what I found, I'm thinking Heroku + MongoLab + PyMongo + Django is probably the best way to get my feet wet, since I'm already comfortable with django and heroku. 

I'll be trying this in the near future -- will let you know how it goes.



Cloud hosts for mongoDB:
  •     MongoLab
  •     MongoHQ
  •     MongoMachine -- bought by MongoHQ

Reviews here say MongoLab > MongoHQ w.r.t customer service
    http://www.quora.com/Heroku/How-would-I-use-the-mongolab-add-on-with-python

python ORMs for mongo
  •     mongonaut
  •     mongoengine
  •     mongokit/django-mongokit
  •     pymongo (simple wrapper, no ORM)
  •     ming
  •     django-mongodb
  •     django-nonrel

Strong recc for mongoengine > mongokit, esp for django developers.
    http://www.quora.com/MongoDB/Whats-the-best-MongoDB-ORM-for-Python

Says mongoEngine is faster than mongoKit
    http://www.peterbe.com/plog/mongoengine-vs.-django-mongokit

Slides also argue for mongoEngine
    http://www.peterbe.com/plog/using-mongodb-in-your-django-app/django-mongodb-html5-slides/html5.html

Says that pyMongo > mongoEngine
    http://stackoverflow.com/questions/2740837/which-python-api-should-be-used-with-mongo-db-and-django

mongoNaut is clearly not mature -- off the island!
    http://readthedocs.org/docs/django-mongonaut/en/latest/index.html

Instructions for setting up django and mongo, if that's your thing
    http://dennisgurnick.com/2010/07/06/bootstrapping-a-django-with-mongo-project/

PyMongo documentation
    http://api.mongodb.org/python/1.7/faq.html

MongoLab's example of integration, plus a small amount of stackoverflow chatter about it.
    https://github.com/mongolab/mongodb-driver-examples/blob/master/python/pymongo_simple_example.py
    http://stackoverflow.com/questions/8859532/how-can-i-use-the-mongolab-add-on-to-heroku-from-python


Decision reached!
    Heroku + MongoLab + PyMongo

Tuesday, December 13, 2011

A worldwide scavenger hunt in 90 minutes!

I helped run a really fun, informal social experiment today. When students in Scott Page's undergraduate "intro to complexity" course showed up to class, they were given a challenge: find one person from each of 60 cities around the world, and take a picture of them before the end of class.  Six teams of 15 students had 90 minutes -- and no prior warning -- to compete.

Me, proving that I live in Ann Arbor for Team Bass.
(All the teams were named after mathematicians).

We've called it Stanley Darpa, in homage to Stanley Milgrom's small world experiment, and the more recent DARPA red balloon challenge (more here). The winning teams used a combination of social networking ("My sister just moved there!"), crowdsourcing ("I'll ask all my facebook friends if they know anybody in Shanghai"), and desperate dashing around campus ("IS ANYBODY IN THIS CAFETERIA FROM FORT WAYNE?") to get as many pictures as possible.

I helped come up with the concept, but my main role was to develop the software. I built a simple django site where students could quickly upload pictures as they found them. To ramp up the excitement, I also built tools to let teams track scores and compare finds in real time.  These added a strategic dimension to the game: e.g. "Team Markov is leading -- we need to find people from their cities so they won't get so many bonus points!"

The Stanley-Darpa countdown page

In true Mission Control style, we put everything up on a projector screen and watched results roll in over the course of the class.  It was slow at first, but as teams split up and started canvassing, the competition really heated up. In the end, the six teams found people from 47 cities.  The winning team alone found 30. Surprisingly, many of the hardest cities to find were nearby: Youngstown, OH; Gull Lake, MI; Cadillac; MI.

Tomorrow, I'll try to post a link to the actual site, but first I need to scrub the pictures, to make sure that nobody gets their ID stolen.

This was really fun -- I'd love to do it  again sometime.

Tuesday, December 6, 2011

Fixing rabbitmq after switching servers

I ran into some backend headaches with django, celery, and rabbitmq over the weekend. The EC2 instance I was using crashed, so I had to switch servers. When I did, rabbitmq-server, which the broker behind celery, which is the task queue behind django -- broke.  The broker broke.  And considering that there are still parts of django I'm wrapping my head around, chasing down a system error three levels deep was quiet the pain.

Happily, I got some very good help from the rabbitmq-discuss list:

The database RabbitMQ uses is bound to the machine's hostname, so if you copied the database dir to another machine, it won't work.  If this is the case, you have to set up a machine with the same hostname as before and transfer any outstanding messages to the new machine.  If there's nothing important in rabbit, you could just clear everything by removing the RabbitMQ files in /var/lib/rabbitmq.

I deleted everything in /var/lib/rabbitmq/mnesia/rabbit/ and it started up without trouble.  Hooray!

I also got a warning about using outdated software:

1.7.2 is a *very* old version of the broker.  You should really upgrade, as a lot of new features and a lot of bugfixes have gone in since then.

Now, as best I can recall, I originally installed rabbitmq with apt-get, running in Ubuntu 10.04. To fix this, I got the newest .deb from the rabbitMQ site and installed it.  Here are the steps:

wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.7.0/rabbitmq-server_2.7.0-1_all.deb
sudo apt-get install erlang-nox
sudo apt-get -f install
sudo apt-get install erlang-nox
sudo dpkg -i rabbitmq-server_2.7.0-1_all.deb

Very happy to be out of the woods on this one. I'm not a sysadmin and I don't aspire to be.

Thursday, December 1, 2011

Production mode in django: lxml, celery, the works.

Firing up production mode in django.  These refs were helpful.

LXML
http://answers.bitnami.org/questions/1765/use-lxml-with-djangostack

Celery
http://django-celery.readthedocs.org/en/latest/getting-started/first-steps-with-django.html

Tuesday, November 29, 2011

Configuring django with wsgi

In the last couple weeks, I've fallen in love with django. Here are some notes (just for reference) on getting django to talk with R, numpy, and WSGI


#Getting the latest version of R on an older version of ubuntu:
http://cran.r-project.org/bin/linux/ubuntu/README

#Using PIP and VirtualEnv with django
http://www.saltycrane.com/blog/2009/05/notes-using-pip-and-virtualenv-django/

sudo apt-get install python-setuptools python-dev build-essential
pip install numpy
sudo pip install numpy
sudo pip install scipy
sudo pip install rpy2

I never quite cracked this one. It seems that by default, pip doesn't work with the bitnami djangostack. I just used easy_install, with good results.

#Configuring WSGI
http://bitnami.org/forums/forums/djangostack/topics/getting-started
http://serverfault.com/questions/327380/beginner-installing-first-app-on-ec2-bitnami-stack

Neither of these worked perfectly for me, but they got me to the point where I could figure it out.


#Serving static files with WSGI
https://docs.djangoproject.com/en/dev/howto/static-files/
https://docs.djangoproject.com/en/dev/howto/deployment/modwsgi/
https://docs.djangoproject.com/en/dev/howto/deployment/modwsg/#serving-files

Trust the django docs to have a good built-in explanation.