Installed

sudo apt-get install rabbitmq-server
pip install -E . carrot

Links

• http://github.com/ask/carrot
• http://www.rabbitmq.com/

Testing

# open in two terms to test running
workit hotalert && cd data/generated_files && watch ls -al

# generate 10 files, clear the files, and then watch the system in top
ab -n 10 http://127.0.0.1:5006/hotalert/jobs/generatefile && curl "http://127.0.0.1:5006/hotalert/jobs/clearfiles" && htop

# worker scripts, can run multiple times to get multiple workers
workit hotalert && cd qsat_hotalert_app/scripts && python queue_genfile.py

Server Setup

sudo rabbitmqctl
sudo rabbitmqctl add_user hotalert hotalert
sudo rabbitmqctl add_vhost /hotalert
sudo rabbitmqctl set_permissions -p /hotalert hotalert ".*" ".*" ".*"

Visualization

User makes a controller request that then uses carrot to talk to the rabbitmq server and create a job to perform.

One of the waiting workers from the scripts directory picks up the jobs and completes it.

+----------------------+          +----------------------+            +-------------------------------+
|   Pylons app         |--------->|                      |----------->|  RabbitMq Server              |
|     /controller      |          |                      |            |                               |
+----------------------+          |     Carrot Library   |            | +---------------------------+ |
                                  |                      |            | |  vhost = /                | |
                                  |                      |            | |                           | |
                                  |                      |            | |                           | |
+----------------------+          |                      |            | |                           | |
|  Pylons app          |<---------|                      |<-----------| |                           | |
|    /scripts          |          |                      |            | |                           | |
+----------------------+          +----------------------+            | |                           | |
                                                                      | |                           | |
                                                                      | +---------------------------+ |
                                                                      |                               |
                                                                      | +---------------------------+ |
                                                                      | |  vhost = /hotalert        | |
                                                                      | |                           | |
                                                                      | |    +----------------+     | |
                                                                      | |    | queue 'files'  |     | |
                                                                      | |    +----------------+     | |
                                                                      | |    +----------------+     | |
                                                                      | |    | queue 'email'  |     | |
                                                                      | |    +----------------+     | |
                                                                      | +---------------------------+ |
                                                                      |                               |
                                                                      +-------------------------------+

Publisher Setup

This is actually the pylons app sending jobs over to the message queue

Comsumer Setup

A long running script that's firing off the required jobs.

RabbitMQ

Carrot

Python library that makes talking to RabbitMQ much easier. Used by both the producer/consumer to push/run jobs from the rabbitmq server.

Celery

(not in current demo)