Why we started running tests on pull requests at Hipo

Continuous Integration and Continuous Deployment (CI/CD) are two crucial topics when it comes to agile development. There is a breakdown here if you want to dig deeper into those.

When it comes to continuous integration, automated tests are a key point that you want to integrate to your project and system. There are great tools such as Travis and CircleCI, and most of the cloud providers have out-of-the-box solutions (like Google Cloud Build and AWS CodePipeline) which might also worth a shot. In short, you don’t need to reinvent the wheel for automated testing, which is quite nice, but there are some best practices that you might want to follow to get the most out of these tools.

Generally, continuous integration suggests that “test breaking changes” should not be deployed to remote servers. In order to achieve that, we can run tests before each deployment, and terminate the deployment if any of the tests fail. This is a good starting point, but we can be more defensive, and run our tests for each commit if that commit is associated with a pull request. You can set this up in various ways for each CI/CD tool, and CircleCI even has a setting (under Advanced Settings) like the following to make things easier:

This is especially useful when it comes to detecting some breaking changes and take action before it is too late, i.e. break staging/production. So we think it is a must to run tests for each PR commit.

As you could probably guessed, we use CircleCI at Hipo, and we think it is great if you don’t want to spend lots of time trying to set things up for your pipeline to work as expected. You only need to have a single config.yml and have jobs defined as regular terminal commands, which is quite convenient. You can also have workflows in the same file if you want to pipeline things, which you generally need. An example workflow we use is like the following:

workflows:
  version: 2
  deployment:
    jobs:
      - linter
      - tests:
          requires:
            - linter
      - deploy_staging:
          requires:
            - tests
          filters:
            branches:
              only: staging

Each job in the workflows section is executed from top to bottom, and we have built our pipeline so that each step requires the previous step to be executed successfully. This way, we can detect any failure as soon as possible.

Another approach that we follow when testing our code is to test each PR commit against the prospective merge of that PR’s branch. For example, if you have a feature_a branch which you have a PR to master , we test the merged version of feature_a instead of running tests on feature_a . This can further prevent some bugs when your base branch is updated frequently and you don’t want to merge the base branch to your feature branch every time base branch is updated. This feature is native in some tools like Travis, but as of the time of writing this article, CircleCI does not provide it out-of-the-box. What we use is an initial step¹ in our test jobs:

- run:
    name: Checkout merge commit
      command: |
          set -ex
          if [[ -n "${CIRCLE_PR_NUMBER}" ]]
          then
              FETCH_REFS="${FETCH_REFS}
              +refs/pull/${CIRCLE_PR_NUMBER}/merge:pr/${CIRCLE_PR_NUMBER}/merge"
              git fetch -u origin ${FETCH_REFS}
              git checkout "pr/${CIRCLE_PR_NUMBER}/merge"
          fi

"pr/${CIRCLE_PR_NUMBER}/merge" part represents the merged version of our feature branch. This way, we can first check out to the “prospective merge branch” and then run our test commands.

An important thing to note here is that if you are using GitLab, you need to update your reference as such:

+refs/merge-requests/${CIRCLE_PR_NUMBER}/merge:merge-requests/${CIRCLE_PR_NUMBER}/merge

but the idea is the same.

This was a quick breakdown of some of our preferred methods of automated testing. Let us know about your approaches and best practices.

Thanks for reading!

[1] https://ideas.circleci.com/ideas/CCI-I-926