Using Libcloud in multi-threaded and async environments

    Most of the time you want to perform more operations in parallel or justwant your code to finish faster (for example starting a lot of servers orperiodically polling for node status).

    Problems like this are usually solved using threads or async libraries suchas Twisted, Tornado or gevent.

    This page contains some information and tips about how to use Libcloud insuch environments.

    gevent has an ability to monkey patch and replace functions in the Python, , and module with its ownfunctions which don’t block.

    You need to do two things when you want to use Libcloud with gevent:

    • Enable monkey patching
    • Create a separate driver instance for each Greenlet. This is necessarybecause a driver instance reuses the same Connection class.

    For an example see .

    If you don’t use it properly it can block the whole reactor (similar asany other blocking library or a long CPU-intensive task) which means theexecution of other pending tasks in the event queue will be blocked.

    A simple solution to prevent blocking the reactor is to run Libcloudcalls inside a thread. In Twisted this can be achieved using which runs a provided method inside the Twistedthread pool.

    The example below demonstrates how to create a new node inside a threadwithout blocking the whole reactor.

    1. from __future__ import absolute_import
    2.  
    3. from pprint import pprint
    4.  
    5. # pylint: disable=import-error
    6. from twisted.internet import defer, threads, reactor
    7. from libcloud.compute.types import Provider
    8. from libcloud.compute.providers import get_driver
    9.  
    10. @defer.inlineCallbacks
    11. def create_node(name):
    12.     node = yield threads.deferToThread(_thread_create_node,
    13.                                        name=name)
    14.     pprint(node)
    15.     reactor.stop()
    16.  
    17.  
    18. def _thread_create_node(name):
    19.     Driver = get_driver(Provider.RACKSPACE)
    20.     conn = Driver('username', 'api key')
    21.     image = conn.list_images()[0]
    22.     node = conn.create_node(name=name, image=image, size=size)
    23.     return node
    24.  
    25.  
    26. def stop(*args, **kwargs):
    27.     reactor.stop()
    28.  
    29.  
    30. d = create_node(name='my-lc-node')
    31. d.addCallback(stop)  # pylint: disable=no-member
    32. d.addErrback(stop)  # pylint: disable=no-member
    33.  
    34. reactor.run()