USPS Team:

The Bamboo build for the USPSObject's SOAP test running against SOAP-bridge now reports as 'successful'. Success means that the number of failing tests is less than the threshold set in Sonar. If the number of failing tests exceeds the threshold, the build will be marked failed.

Therefore, failures of this plan should now be taken seriously. I've re-enabled notifications on this build, so the team will be notified as with other plans.

The project is now processed by Sonar, so you can see the project as "usps.objects.test.uspsr" to review the failing tests and trends.

Matt : The threshold is set in Sonar under Configuration->Quality Profiles. There is a special profile called "USPS-Objects-USPSR". Under the "Alerts" you'll see the limits for the "test errors" and "test failures". You are now a Sonar administrator, so you can change the values as needed.

The only puzzle I couldn't solve was detecting "new failing" tests. Normally, Bamboo would track this. But Bamboo doesn't allow you to treat a build with failed tests as "successful". So I had to disable Bamboo's processing of test results (so the build would pass), but this also disables it's tracking of test history. So Bamboo will not help us determine which specific tests have started failing.

To compensate, I saved the test results as an Artifact in the Bamboo build. So when a build fails, we can download the test results from two builds and diff them to find the failing test. The zip file contains the full test result XML files as well as a "summary.txt" which is diff-able.