For teams that run Buildkite Pipelines and build Xcode based software projects for macOS, iOS, iPadOS, tvOS, and watchOS apps, you can now run your Buildkite Builds on AWS EC2 Mac instances using a CloudFormation template. 🎉

This experimental template creates an Auto Scaling group, launch template, and host resource group to launch a pool of EC2 Mac instances that run the Buildkite Agent.

📣 Big shoutout to Buildkite customer Oliver Koo for his early input into this feature 🙏

📚 For details on how to prepare and deploy this template to your AWS Account, checkout the Auto Scaling EC2 Mac documentation, or jump straight into Elastic CI Stack for EC2 Mac.

Identify, track, and fix problematic tests with ✨Test Analytics✨, now in private beta.

🔍 Identify flaky tests: See which tests are most disruptive with automatic flaky test identification and analysis over time.

👀 Monitor speed & reliability: Get alerts when slow tests are introduced.

📈 Deeply analyze performance: First-class framework integrations provide in-depth performance analysis, helping you find external dependency bottlenecks and more.

🏎 View real-time results: Watch your test suite run in real-time and get immediate performance insights.

Learn more and join our waitlist at https://buildkite.com/test-analytics.

In November 2018, we posted a changelog deprecating BUILDBOX_* environment variables from generating for new jobs.

From today, we no longer send BUILDBOX_* environment variables.

You can see our environment variable documentation for a complete list of current job environment variables.

The 3.33.3 version of the buildkite-agent and the 5.7.0 version of the AWS elastic stack are now available. 🚀

The 3.33.3 Agent release has added:

• Support for unset environment variables in Job Lifecycle Hooks

The 5.7.0 Elastic Stack release has added:

• Support for storing builds, git-mirrors, and Docker data on NVMe Instance Storage
• Retried login for ECR and generic Docker registries
• Experimental CloudFormation service role, listing the IAM Actions required to create, update, and delete the template
• A README feature matrix for Linux and Windows
• qemu and binfmt hooks for cross-architecture Docker image builds
• Support for AWS SSM sessions

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

For teams running a lot of jobs who want more control over what order their jobs run in, we've introduced the Job Priority attribute to specify its priority within its queue. 🙋🏻‍♀️

It's a new attribute defined on a step in the pipeline configuration that changes how the dispatcher assigns work to agents. The higher the value you set for priority, the sooner a job will be prioritised. 🔝

Our recently released Eager Concurrency feature can also be used in conjunction with Job Priority.

Enterprise teams using Okta or Azure AD for SSO with their Buildkite organization can now optionally enable SCIM to automatically deprovision users ⚡️

Okta and Azure AD SSO with SAML is still available to all users.

A step-by-step guide to enabling SCIM support is available in our Single Sign-On with Okta and Azure AD documentation 📚

Traditionally, concurrency groups enforce strict ordering of the jobs within them. Concurrent jobs will be limited to the number set for the group, and will only start executing in the order they were created. 📝

However, sometimes you only need the limit, and it's fine for eligible jobs to run in whatever order their dependencies allow. 🏃🏻‍♀️

Which is why we've added a concurrency_method step attribute you can set to 'eager'. This will allow any job, up to the concurrency limit, to start as soon as it's eligible, regardless of creation order. 🔀

As an example:

1
2
3
4
5
steps:
  - command: echo "Using a limited resource, only 10 at a time, but we don't care about order"
    concurrency_group: saucelabs
    concurrency: 10
    concurrency_method: eager

For more information, check out our guide to Controlling Concurrency.

The 3.32.3 version of the buildkite-agent and the 5.6.0 version of the AWS elastic stack are now available.

The 3.32.3 Agent release has added:

• Support for cross-region artifact buckets
• Improved error logging around AWS Credentials
• Logging to the artifact upload command to say where artifacts are being sent

The 5.6.0 Elastic Stack release has added:

• Cross-region secrets bucket support to git-credentials-s3-secrets
• AssumeRole support in the ECR Login plug-in

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

From 1 October 2021, build artifacts hosted by Buildkite will be retained for six months from time of upload, after which they will be deleted. Artifacts uploaded before 1 April 2021 will also be deleted at this time.

Previously, build artifacts were retained indefinitely, which means we're currently storing over 7PB of data 🤯📈

Custom-hosted build artifacts are not affected by this change, and remain available to any customer who wants more control over their retention.

As always, you can reach out to us with any questions about this change.

We've released v5.5.0 of the Elastic Stack CI for AWS ☁️

Included in this release:

• Template validation rules for the Buildkite Agent token
• Secret redaction in build logs
• Support for the pre-bootstrap Buildkite Agent lifecycle hook

You can read the full release notes on the v5.5.0 release on GitHub.

We've released v3.32.0 of the Buildkite Agent 🦾🤖

The release adds a new pre-bootstrap lifecycle hook which can accept or reject jobs before environment variables are loaded, providing an additional layer of security and control over your Buildkite agents. See the documentation on lifecycle hooks for details on how to use it.

You can read the full release notes on the v3.32.0 release on GitHub. To upgrade, follow the instructions in the Agent docs.

For teams running their own compute scheduler or those needing single-use agents, we've added the --acquire-job flag to the Buildkite Agent 🧰

With this flag, agents are mapped 1:1 with jobs. This makes them perfect for folks using a system like Kubernetes, Nomad, or ECS, or anyone who wants a clean-room environment.

To get started, pass the --acquire-job flag with the job ID to the buildkite agent start command. When you start an agent with this flag it will run the job then exit rather than polling for more work.

You can use acquire-job with agents v3.17.0 and above, anywhere that the agent can run 🏃🏻✨

For details on how to use the flag, see the agent cli documentation on Running a single job.

For teams wanting to add an extra layer of security to their webhooks, we have just added the option to include HMAC signatures with your Webhook Notification services 🔐

You'll find the HMAC signature option under the Token section when editing or creating Webhook Notifications.

For more information on verifying webhook signatures and defending against replay attacks, see our Webhook API documentation.

We've released v1.2.0 of the Buildkite CLI 📎💪🏼🪟

Included in this release:

• The bk build create command now has a --meta-data argument, for setting Build Meta-Data when creating a build
• Running local builds (via bk local run) now works on Windows
• An improved GitHub authentication flow

You can read full release notes on the v1.2.0 release on GitHub

To upgrade, if you're using Homebrew on a Mac run brew upgrade bk, otherwise download the latest release for your platform from the GitHub release page.

For complete control over when to trigger builds, use conditional expressions to filter incoming webhooks 📥

Build filtering with conditionals is available with any repository provider. You can add a conditional on your Pipeline Settings page or via the REST API 🔀

For more information on supported conditionals, check out our guide to Using Conditionals.

The latest release of the Buildkite Terraform provider adds buildkite_meta: a data source providing the IP addresses Buildkite uses for webhooks ✨

The new data source can be used in your terraform project to dynamically set firewall and ingress rules, allowing in traffic from Buildkite 🚦

You can find a code sample and attribute reference in the Buildkite provider docs on the Terraform Registry. For more detailed information about the addition, check out the v0.4.0 release on GitHub.

Send Change Events to your PagerDuty services whenever a build completes 🚨

Filter Change Events notifications with conditionals to make sure you're sending only the information your team needs

The Buildkite integration can be installed from the PagerDuty Service Directory 📚

Find the step-by-step guide to sending PagerDuty change events in our PagerDuty integration guide.

To simplify querying jobs with GraphQL we're shipping a fix to make sure broken jobs return a "BROKEN" state 🚦

Starting on Tuesday, 1st June 2021 at 00:00 UTC broken jobs fetched via the GraphQL API will return BROKEN instead of SKIPPED.

If you have scripts or other clients of the GraphQL API that rely on jobs returning a SKIPPED state then you will need to update these to accept the BROKEN state as well.

When fetching jobs from our REST API, broken jobs already return a state of BROKEN and will continue to do so.

If you need this switch made for your organization prior to 1st June 2021, please reach out to support@buildkite.com 👋🏻

We've had a lot to talk about at Buildkite lately! We relaunched our blog, we've been updating and revamping our documentation, and have a ton of upcoming projects, customer stories, and tutorials we can't wait to share.

If that sounds like a lot to keep track of, we've got you covered! We're launching a 📰 newsletter 📰 to provide curated highlights, so you can stay in the know whatever your schedule ✨

Sign up here and you'll get a handy digest of our favorite features, updates, and blog posts delivered directly to your inbox 🗞📬🚀

For the latest news as it happens, this changelog will remain your one-stop source for Buildkite updates. You can also check out the blog and our docs directly to stay up to date on all things Buildkite 💚

Nobody likes accidentally leaking secrets in a build log. The Buildkite agent can now redact secrets present in your environment variables before they are uploaded to your logs 👷🏻

Turn it on for your agents, and add your custom variable patterns.

Thanks to Buildkite customer David Baggerman for doing the initial work on this feature 💚