Cloud Foundry Buildpacks Concourse Deployment

James Wen

Completed work

DevOps Engineer

Docker

GitHub

Computer Software

Introduction

This contains the configuration for the Cloud Foundry Buildpacks team Concourse deployment.

Pipelines

dependency-builds: build binaries for Cloud Foundry buildpacks

buildpacks: test and release all of the buildpacks

brats: run BRATS against the master branch of buildpacks

buildpack-verification: generate static site for buildpack verification

buildpacks-ci: testing tasks for correct usage, rebuild CI docker images

cf-release: deployment of latest buildpacks to cf-release develop

Concourse State

Jobs and tasks in the buildpacks-ci repository store state in public-buildpacks-ci-robots. See repository README for details.

Commands and recipes

Updating all the Pipelines

./bin/update-pipelines

Debugging the build

fly intercept -j $JOB_NAME -t task -n $TASK_NAME

Clearing the git resources

fly intercept -c $RESOURCE_NAME rm -rf /tmp/git-resource-repo-cache

To build a new version of a binary

Check out the binary-builds branch

Edit the YAML file appropriate for the build (e.g. ruby-builds.yml)

Find the version number and package SHA256 of the new binary. For many binaries, the project website provides the SHA256 along with the release (for example, jruby.org/download provides the SHA256 along with each JRuby release). For others (such as Godep), you download the .tar.gz file and run shasum -a 256 <tar_file> to obtain the SHA256.

Add any number of versions and their checksums to the array, e.g.

git commit -am 'Build ruby 2.2.2' && git push

Build should automatically kick off at https://buildpacks.ci.cf-app.com/pipelines/binary-builder and silently upload a binary to the buildpacks.cloudfoundry.org bucket under dependencies/, e.g. https://buildpacks.cloudfoundry.org.s3.amazonaws.com/dependencies/ruby/ruby-2.2.2-linux-x64.tgz

Note that the array is a stack, which will be emptied as the build succeeds in packaging successive versions.

Making Changes to Build Scripts

When you want to change how a binary gets built, there are two places you may need to make changes. All binaries are built by the dependency-builds pipeline, and you may need to change the task that builds them. For many binaries, the dependency-builds pipeline runs recipes from the binary-builder repo; for those binaries, you will usually need to change the recipe rather than the concourse task.

For the list of currently supported binaries, check out our dependency-builds pipeline.

The concourse task that orchestrates the building is buildpacks-ci/tasks/build-binary-new/builder.rb; many of the recipes are in binary-builder.

To test these changes locally, you can execute the concourse task for it, but point to local changes. For instance:

$ cd buildpacks-ci
$ STACK=cflinuxfs2 fly -t buildpacks e -c tasks/build-binary-new/build.yml -j dependency-builds/build-r-3.4.X -i buildpacks-ci=.

For binaries that use recipes in binary-builder, you can also test in Docker. For instance:

$ docker run -w /binary-builder -v `pwd`:/binary-builder -it cloudfoundry/cflinuxfs2:ruby-2.2.4 ./bin/binary-builder --name=ruby --version=2.2.3 --md5=150a5efc5f5d8a8011f30aa2594a7654
$ ls
ruby-2.2.3-linux-x64.tgz

Buildpack Repositories Guide

buildpacks-ci pipelines and tasks refer to many other repositories. These repos are where the buildpack team and others develop buildpacks and related artifacts.