Job/Task queue quick tutorial using retask

---

Posted: 2012/12/12 15:44:15 | Source | More posts about FOSS Python

---

Here is one quick tutorial of Job/Task queue using retask

You need to first install retask & redis-py from pypi.

$ pip install retask redis

Then start the redis server (if it is not already running, in my case I installed it from Fedora package repository).

service redis start

In out producer code we will create tasks to get current stock values of few companies. The workers will get the company symbols, then they will fetch the value and pass to the producer.

from retask.task import Task from retask.queue import Queue

import time
queue = Queue(‘sharevalue’)
queue.connect()
symbols = [‘AAPL’, ‘ORCL’, ‘MSFT’]
jobs = []
for sym in symbols:
    info = {‘name’: sym}
    task = Task(info)
    jobs.append(queue.enqueue(task))


print [job.result for job in jobs if job.wait()]

Here we have a list of symbols and creating tasks based on them, as they are getting enqueued, a Job object is returned for each one of them.

In the workers we will use requests, so let us install it first.

$ pip install requests

then the actual worker code

import requests
from retask.queue import Queue


queue = Queue(‘sharevalue’)
queue.connect()
while True:
    task = queue.wait()
    print “Received %s” % task.data[‘name’]
    url = ‘http://download.finance.yahoo.com/d/quotes.csv?s=%s&f=l1' % task.data[‘name’]
    r = requests.get(url)
    queue.send(task, r.content.strip())

Here we wait for new tasks in the queue. After getting a new task, we find the latest share value and then return it to the producer. Please go through the documentation for more examples.