Cloud Foundry Buildpacks Concourse Deployment by James WenCloud Foundry Buildpacks Concourse Deployment by James Wen

Cloud Foundry Buildpacks Concourse Deployment

James Wen

Completed work

DevOps Engineer

Docker

GitHub

Computer Software

Introduction

This contains the configuration for the Cloud Foundry Buildpacks team Concourse deployment.

Pipelines

dependency-builds: build binaries for Cloud Foundry buildpacks

buildpacks: test and release all of the buildpacks

brats: run BRATS against the master branch of buildpacks

buildpack-verification: generate static site for buildpack verification

buildpacks-ci: testing tasks for correct usage, rebuild CI docker images

cf-release: deployment of latest buildpacks to cf-release develop

Concourse State

Jobs and tasks in the buildpacks-ci repository store state in public-buildpacks-ci-robots. See repository README for details.

Commands and recipes

Updating all the Pipelines

./bin/update-pipelines

Debugging the build

fly intercept -j $JOB_NAME -t task -n $TASK_NAME

Clearing the git resources

fly intercept -c $RESOURCE_NAME rm -rf /tmp/git-resource-repo-cache

To build a new version of a binary

Check out the binary-builds branch

Edit the YAML file appropriate for the build (e.g. ruby-builds.yml)

Find the version number and package SHA256 of the new binary. For many binaries, the project website provides the SHA256 along with the release (for example, jruby.org/download provides the SHA256 along with each JRuby release). For others (such as Godep), you download the .tar.gz file and run shasum -a 256 <tar_file> to obtain the SHA256.

Add any number of versions and their checksums to the array, e.g.

git commit -am 'Build ruby 2.2.2' && git push

Build should automatically kick off at https://buildpacks.ci.cf-app.com/pipelines/binary-builder and silently upload a binary to the buildpacks.cloudfoundry.org bucket under dependencies/, e.g. https://buildpacks.cloudfoundry.org.s3.amazonaws.com/dependencies/ruby/ruby-2.2.2-linux-x64.tgz

Note that the array is a stack, which will be emptied as the build succeeds in packaging successive versions.

Making Changes to Build Scripts

When you want to change how a binary gets built, there are two places you may need to make changes. All binaries are built by the dependency-builds pipeline, and you may need to change the task that builds them. For many binaries, the dependency-builds pipeline runs recipes from the binary-builder repo; for those binaries, you will usually need to change the recipe rather than the concourse task.

For the list of currently supported binaries, check out our dependency-builds pipeline.

The concourse task that orchestrates the building is buildpacks-ci/tasks/build-binary-new/builder.rb; many of the recipes are in binary-builder.

To test these changes locally, you can execute the concourse task for it, but point to local changes. For instance:

$ cd buildpacks-ci
$ STACK=cflinuxfs2 fly -t buildpacks e -c tasks/build-binary-new/build.yml -j dependency-builds/build-r-3.4.X -i buildpacks-ci=.

For binaries that use recipes in binary-builder, you can also test in Docker. For instance:

$ docker run -w /binary-builder -v `pwd`:/binary-builder -it cloudfoundry/cflinuxfs2:ruby-2.2.4 ./bin/binary-builder --name=ruby --version=2.2.3 --md5=150a5efc5f5d8a8011f30aa2594a7654
$ ls
ruby-2.2.3-linux-x64.tgz

Buildpack Repositories Guide

buildpacks-ci pipelines and tasks refer to many other repositories. These repos are where the buildpack team and others develop buildpacks and related artifacts.