Testing

    • Makes you more confident that your software is correct.
    • Makes it easier to make changes to the code, the tests will catch changes to way the software functions.
    • Tests communicate how software is intended to function.

    These points do not apply to prototype / throwaway phase.

    To run testing, perform code quality analysis and build artifacts a Continuous Integration server can be used. The build will be performed every git push and pull request. Using a CI server will help with problems.
    The Netherlands eScience Center uses continuous integration services as much as possible when creating code.

    continuous integration (CI), public on

    CI meaning: compile, unit test, integration test, quality analysis etc.
    Once there is some build process established and tests set up, CI should be configured too.
    It will save you a lot of time on debugging and allow for much quicker problem diagnosis.

    The Netherlands eScience Center public repositories should be built with Travis-CI.
    Travis-CI is free for Open Source projects.
    A Github repository can be added to Travis-CI by a Github user with admin right on the repository.
    At the moment Travis-CI performs builds in Ubuntu and OS X operating systems.

    To build repositories inside the Microsoft Windows operation system use AppVeyor.
    AppVeyor is free for Open Source projects.

    Most CI builds are triggered by a git push, but sometimes the repository must be build every night.
    Possible reasons for nightly builds:

    • Make sure the repository stays working even if there are no changes pushed to the repository, but it’s dependencies are changing possibly breaking the code in the repository.
    • The build performs an action that needs to be performed daily like updating a cache.

    For triggering nightly builds in Travis-CI can be used.

    All major CI services support some form of cctray.xml feed. This feed can be read by polling tools to automatically keep an eye on your project builds. For instance, BuildNotify, and CCTray give you a tray icon that turns red when a build fails.

    Code coverage is a measure which describes how much of the source code is exercised by the test suite.
    At the Netherlands eScience Center we require minimum of 70% coverage.

    Setting up code coverage for a repository depends on the programming language, see the for setup instructions.

    The publishing of the code coverage can be performed during a Continuous Integration build job.
    The code coverage service offers a visualization of the coverage and a metric which can be displayed as a badge/shield icon on the repository website.
    See the language specific guides which code coverage services are available and preferred for that language.

    Code coverage services support many languages and a usually free for Open Source projects.
    Below is a short list of services and their strengths.

    Codecov

    Shows unified coverage and separate coverage for e.g. different Python versions.
    For example project see https://codecov.io/gh/NLeSC/Xenon, with a Java 7/8 and Linux/Windows/OSX OS build matrix.

    Coveralls

    More popular then Codecov.
    For example project see

    For (web) user interfaces. example with protractor and angular

    Once the web page has any interface, e2e tests should be implemented.

    Checking for dependency updates should be done regularly. It can save a lot of time,
    avoiding code dependent on deprecated functionality.