Dopamine





    Our design principles are:

    • Easy experimentation: Make it easy for new users to run benchmark
    • Flexible development: Make it easy for new users to try out research ideas.
    • Compact and reliable: Provide implementations for a few, battle-tested
    • Reproducible: Facilitate reproducibility in results. In particular, our
      1.             setup follows the recommendations given by

    In the spirit of these principles, this first version focuses on supporting the
    state-of-the-art, single-GPU Rainbow agent ()
    applied to Atari 2600 game-playing (Bellemare et al., 2013).
    Specifically, our Rainbow agent implements the three components identified as
    most important by :

    For completeness, we also provide an implementation of DQN
    ().
    For additional details, please see our
    documentation.

    This is not an official Google product.

    • 01/11/2018: Download links for each individual checkpoint, to avoid
      having to download all of the checkpoints.
    • 29/10/2018: Graph definitions now show up in Tensorboard.
    • 16/10/2018: Fixed a subtle bug in the IQN implementation and upated
      the colab tools, the JSON files, and all the downloadable data.
    • 18/09/2018: Added support for double-DQN style updates for the
      ImplicitQuantileAgent.
      • Can be enabled via the double_dqn constructor parameter.
    • 18/09/2018: Added support for reporting in-iteration losses directly from
      the agent to Tensorboard.
      • Include the flag --debug_mode in your command line to enable it.
      • Control frequency of writes with the summary_writing_frequency
        agent constructor parameter (defaults to 500).
    • 27/08/2018: Dopamine launched!

    Instructions

    Installing from source allows you to modify the agents and experiments as
    you please, and is likely to be the pathway of choice for long-term use.
    These instructions assume that you’ve already set up your favourite package
    manager (e.g. apt on Ubuntu, homebrew on Mac OS X), and that a C++ compiler
    is available from the command-line (almost certainly the case if your favourite
    package manager works).

    The instructions below assume that you will be running Dopamine in a virtual
    environment    . A virtual environment lets you control which dependencies are
    installed for which program; however, this step is optional and you may choose
    to ignore it.

    Dopamine is a Tensorflow-based framework, and we recommend you also consult
    the
    for additional details.

    Finally, these instructions are for Python 2.7. While Dopamine is Python 3
    compatible, there may be some additional steps needed during installation.

    Ubuntu

    First set up the virtual environment:

    1. sudo apt-get update && sudo apt-get install virtualenv
    2. virtualenv --python=python2.7 dopamine-env
    3. source dopamine-env/bin/activate

    This will create a directory called in which your virtual
    environment lives. The last command activates the environment.

    During installation, you may safely ignore the following error message:
    tensorflow 1.10.1 has requirement numpy<=1.14.5,>=1.13.3, but you’ll have
    numpy 1.15.1 which is incompatible    .

    Finally, download the Dopamine source, e.g.

    1. git clone https://github.com/google/dopamine.git

    Mac OS X

    First set up the virtual environment:

    1. virtualenv --python=python2.7 dopamine-env
    2. source dopamine-env/bin/activate

    This will create a directory called dopamine-env in which your virtual
    environment lives. The last command activates the environment.

    Then, install the dependencies to Dopamine:

    1. brew install cmake zlib
    2. pip install absl-py atari-py gin-config gym opencv-python tensorflow

    During installation, you may safely ignore the following error message:
    tensorflow 1.10.1 has requirement numpy<=1.14.5,>=1.13.3, but you’ll have
    numpy 1.15.1 which is incompatible    .

    Finally, download the Dopamine source, e.g.

    Running tests

    You can test whether the installation was successful by running the following:

    1. cd dopamine
    2. export PYTHONPATH=${PYTHONPATH}:.
    3. python tests/atari_init_test.py

    The entry point to the standard Atari 2600 experiment is
    .
    To run the basic DQN agent,

    1. python -um dopamine.atari.train \
    2.   --agent_name=dqn \
    3.   --base_dir=/tmp/dopamine \
    4.   --gin_files='dopamine/agents/dqn/configs/dqn.gin'

    By default, this will kick off an experiment lasting 200 million frames.
    The command-line interface will output statistics about the latest training
    episode:

    1. [...]
    2. I0824 17:13:33.078342 140196395337472 tf_logging.py:115] gamma: 0.990000
    3. I0824 17:13:33.795608 140196395337472 tf_logging.py:115] Beginning training...
    4. Steps executed: 5903 Episode length: 1203 Return: -19.

    More generally, the whole of Dopamine is easily configured using the
    .

    An easy, alternative way to install Dopamine is as a Python library:

    Depending on your particular system configuration, you may also need to install
    zlib (see “Install via source” above).

    Running tests

    From the root directory, tests can be run with a command such as:

    1. python -um tests.agents.rainbow.rainbow_agent_test

    Bellemare et al., The Arcade Learning Environment: An evaluation platform for
    general agents    . Journal of Artificial Intelligence Research, 2013.

    Hessel et al., Rainbow: Combining Improvements in Deep Reinforcement Learning.
    Proceedings of the AAAI Conference on Artificial Intelligence, 2018.

    Mnih et al., Asynchronous Methods for Deep Reinforcement Learning. Proceedings
    of the International Conference on Machine Learning, 2016.

    Marc G. Bellemare, Pablo Samuel Castro, Carles Gelada, Saurabh Kumar, Subhodeep Moitra.
    Dopamine, https://github.com/google/dopamine, 2018.