infra - Use pipeline for daily builds

Using a pipeline is not required. jenkins considers jobs linked (upstream/downstream) if they share the same fingerprinted object(s). Since we are now doing that with the build-project-* jobs, we should be able to get aggregated results by adding a post-build option to aggregate downstream test results. In jenkins-job-builder, that is a publisher called: aggregate-tests

This technique does not work. Jenkins can correlate jobs based on artifacts, but it does not consider them "downstream" for purposes of aggregation. So for aggregation to work, the jobs need to be actually triggered from the parent job.

Unfortunately, triggering jobs downstream in this way defeats Zuul (gearman) management of the jobs. For instance, when the jobs should be cancelled, the downstream jobs continue to run. So, it is not clear if we can aggregate test results through the normal mechanisms.

To make this feature useful we'd also have to use junit reporting for all our tests, such as sql dev regressions.

Another idea is to use jenkins API to write an external tool to query and correlate jobs and aggregate data. This also seems problematic given the that there seems to be no api for the fingerprint info, at least in our current version of jenkins. Parsing the job descriptions to find all the test jobs might be possible, since zuul already gives the list there. That might miss some use cases like same builds being used in multiple queues such as check and check full. Would probably work for daily jobs.

Given the complexity with current tool set, I'm going to lower the priority of this project to shelve it for now.

Work items:
Experiment with option to aggregate downstream tests: DONE