Using Libcloud in multi-threaded and async environments

    Most of the time you want to perform more operations in parallel or just want your code to finish faster (for example starting a lot of servers or periodically polling for node status).

    Problems like this are usually solved using threads or async libraries such as Twisted, Tornado or gevent.

    This page contains some information and tips about how to use Libcloud in such environments.

    gevent has an ability to monkey patch and replace functions in the Python , urllib2, httplib and time module with its own functions which don’t block.

    You need to do two things when you want to use Libcloud with gevent:

    • Enable monkey patching
    • Create a separate driver instance for each Greenlet. This is necessary because a driver instance reuses the same Connection class.

    For an example see Efficiently download multiple files using gevent.

    If you don’t use it properly it can block the whole reactor (similar as any other blocking library or a long CPU-intensive task) which means the execution of other pending tasks in the event queue will be blocked.

    A simple solution to prevent blocking the reactor is to run Libcloud calls inside a thread. In Twisted this can be achieved using threads.deferToThread which runs a provided method inside the Twisted thread pool.

    The example below demonstrates how to create a new node inside a thread without blocking the whole reactor.

    1. from __future__ import absolute_import
    3. from pprint import pprint
    5. # pylint: disable=import-error
    6. from twisted.internet import defer, threads, reactor
    7. from libcloud.compute.types import Provider
    8. from libcloud.compute.providers import get_driver
    10. @defer.inlineCallbacks
    11. def create_node(name):
    12.     node = yield threads.deferToThread(_thread_create_node,
    13.                                        name=name)
    14.     pprint(node)
    15.     reactor.stop()
    18. def _thread_create_node(name):
    19.     Driver = get_driver(Provider.RACKSPACE)
    20.     conn = Driver('username', 'api key')
    22.     node = conn.create_node(name=name, image=image, size=size)
    23.     return node
    26. def stop(*args, **kwargs):
    27.     reactor.stop()
    30. d = create_node(name='my-lc-node')
    31. d.addCallback(stop)  # pylint: disable=no-member
    32. d.addErrback(stop)  # pylint: disable=no-member
    34. reactor.run()