Building Flink from Source

    In order to build Flink you need the source code. Either download the source of a release or .

    In addition you need Maven 3 and a JDK (Java Development Kit). Flink requires at least Java 8 to build.

    NOTE: Maven 3.3.x can build Flink, but will not properly shade away certain dependencies. Maven 3.2.5 creates the libraries properly.To build unit tests use Java 8u51 or above to prevent failures in unit tests that use the PowerMock runner.

    To clone from git, enter:

    The simplest way of building Flink is by running:

    This instructs Maven (mvn) to first remove all existing builds (clean) and then create a new Flink binary (install).

    To speed up the build you can skip tests, QA plugins, and JavaDocs:

    1. mvn clean install -DskipTests -Dfast

    If you want to build a PyFlink package that can be used for pip installation, you need to build Flink jars first, as described in Build Flink.Then go to the root directory of flink source code and run this command to build the sdist package and wheel package:

    1. cd flink-python; python3 setup.py sdist bdist_wheel

    The sdist and wheel package will be found under ./flink-python/dist/. Either of them could be used for pip installation, such as:

    Flink some of the libraries it uses, in order to avoid version clashes with user programs that use different versions of these libraries. Among the shaded libraries are Google Guava, Asm, Apache Curator, Apache HTTP Components, Netty, and others.

    The dependency shading mechanism was recently changed in Maven and requires users to build Flink slightly differently, depending on their Maven version:

    Maven 3.3.xThe build has to be done in two steps: First in the base directory, then in the distribution project:

    1. mvn clean install -DskipTests
    2. cd flink-dist

    Note: To check your Maven version, run mvn —version.

    Hadoop Versions

    Flink has optional dependencies to HDFS and YARN which are both dependencies from Apache Hadoop. There exist many different versions of Hadoop (from both the upstream project and the different Hadoop distributions). If you are using an incompatible combination of versions, exceptions may occur.

    Flink can be built against any Hadoop version >= 2.4.0, but depending on the version it may be a 1 or 2 step process.

    To build against Hadoop 2.4.1, 2.6.5, 2.7.5 or 2.8.3, it is sufficient to run (e.g., for version 2.6.5):

    1. mvn clean install -DskipTests -Dhadoop.version=2.6.5

    To package a shaded pre-packaged Hadoop jar into the distributions /lib directory, activate the include-hadoop profile`:

    1. mvn clean install -DskipTests -Pinclude-hadoop

    Custom / Vendor-specific versions

    If you want to build against Hadoop version that is NOT 2.4.1, 2.6.5, 2.7.5 or 2.8.3,then it is first necessary to build flink-shaded against this version.You can find the source for this project in the section of the download page.

    Run the following command to build and install flink-shaded against your desired Hadoop version (e.g., for version 2.6.5-custom):

    After this step is complete, follow the steps for Pre-bundled versions.

    To build Flink against a vendor specific Hadoop version, additionally activate -Pvendor-repos profile when buildingflink-shaded`.

    The list of supported vendor versions can be checked .

    Info Users that purely use the Java APIs and libraries can ignore this section.

    Flink has APIs, libraries, and runtime modules written in . Users of the Scala API and libraries may have to match the Scala version of Flink with the Scala version of their projects (because Scala is not strictly backwards compatible).

    Since version 1.7 Flink builds with Scala version 2.11 (default) and 2.12.

    To build FLink against Scala 2.12, issue the following command:

    1. mvn clean install -DskipTests -Dscala-2.12

    Encrypted File Systems

    If your home directory is encrypted you might encounter a java.io.IOException: File name too long exception. Some encrypted file systems, like encfs used by Ubuntu, do not allow long filenames, which is the cause of this error.

    The workaround is to add:

    1. <args>
    2.     <arg>-Xmax-classfile-name</arg>
    3. </args>

    in the compiler configuration of the pom.xml file of the module causing the error. For example, if the error appears in the flink-yarn module, the above code should be added under the tag of scala-maven-plugin. See this issue for more information.