TeamCity Continuous Integration
We use JetBrains TeamCity for various continuous integration and testing tasks. Our instance is available at https://teamcity.cockroachdb.com - you may sign in with your GitHub OAuth credentials.
TeamCity is triggered for each new or updated pull request to this repository. It runs a set of tests triggered by the GitHub CI job and reports the results back to the pull request as a GitHub status check. This job succeeds when all of its dependencies succeed. The tests are as follows:
Each of these is run by a separate build agent in parallel.
TeamCity represents GitHub pull requests by branches named by the number of the pull request:
Some tests, such as the SQL Logic Tests, produce artifacts like extra logs that can be viewed from TeamCity after a test. When trying to find the artifacts for a particular job, say testrace, be sure to navigate to the actual testrace job and not the top-level GitHub CI job that triggered it.
For example, when the nightly stress job or master branch test failure detector posts a comment to GitHub, it will link to the top-level GitHub CI job. Let's look at #14632, for example. After clicking the link to the failing TeamCity build, click (or hover) the Artifacts link next to the failing build, not the one on the top right:
From there, you should be able to see all of the TestLogic artifacts.
If tests fail on a pull request due to a flaky test or some other kind of transient error, you can retrigger the tests from TeamCity by running the GitHub CI job against the pull request branch you're trying to rebuild. Make sure to rebuild the GitHub CI job itself and not any of the individual test jobs - those sub-jobs that failed will be automatically triggered when you request a GitHub CI build.
Here's a mini screencast of how to view a test failure and retrigger TeamCity from a failed status check on GitHub:
In some rare circumstances, TeamCity will fail to notice a new pull request on GitHub. As a workaround, you can manually trigger the GitHub CI job which will report status to GitHub.
To do this, navigate to the GitHub CI page. Click the ... button next to the Run button to enter the Run Custom Build dialog. Select the branch number of your pull request under the Changes tab and click run - TeamCity will then run your tests and report status to GitHub.
When code is merged to master, TeamCity's Release Build job gets triggered for that branch. This job triggers all of the tests run by GitHub CI and posts any failures as GitHub issues. If there are no failures and the updated branch is master, it will kick off a set of binary builds and push them to Docker Hub and our binary release storage area in AWS S3.
There are an additional set of tests that get run each night against master. They live under the Nightlies superproject and get triggered by the All Nightly Tests job.