Using twisted’s inlineCallbacks in 5 steps

Explaining how to use twisted’s inlineCallbacks
rarely goes smoothly. It usually ends in Infinite Frustration®.
The learning curve is just too steep for a junior developer.
My usual attempts to break it down involved describing

  • decorators
  • generators/coroutines
  • Deferred Objects
  • how inlineCallbacks works

This bit from
Guido’s talk about the new asyncio module, made me revise my approach. It is possible to use inlineCallbacks correctly 88.8% of the time by following 5 simple steps:

  1. Make sure you decorate the function with twisted.internet.defer.inlineCallbacks
  2. When calling any method/function that is documented to return a Deferred, use yield and pretend it is a sequential “blocking” call.
  3. Structure the function’s control flow and exception handling as you normally would
  4. In case you want to gather results from several Deferred Objects, add them to a list and yield a DeferredList with that list.
  5. return a value with returnValue (it should not be a Deferred though)
from twisted.internet.defer import inlineCallbacks, returnValue, DeferredList

# you_would_write_this
def get_google():
    body = yield getPage("")

# you_would_pretend_its_this
def get_google():
    body = getPage("")
    return body

# you_would_handle_multiple_calls_like_this
def get_google_10_times():
    url = ""
    deferrreds = list(getPage(url) for _ in xrange(10))
    results = yield DeferredList(deferreds, consumeErrors=True)
# full_example
from __future__ import print_function
from twisted.internet import defer
from twisted.web.client import getPage

def get_multiple(pages):
    results = yield defer.DeferredList([getPage(url) for url in pages])

if __name__ == "__main__":
    from twisted.internet import reactor, task

    t = task.deferLater(reactor, 0, get_multiple, [""] * 2)
    t.addBoth(lambda _: reactor.stop())

Published by pgk


%d bloggers like this: