Ubuntu Automation Test Harness

RENAME IT!!!!
Proposed names:
UTAH (Ubuntu Test Automation Harness)
Enterprise
forecast
[nuclearbob] change name
[gema] change name
* Provisioner
- Physical machine provisioning
    MaaS
- Cloud
    OpenStack (package testing)
- Complex testing environments
    juju deployment

- ARM (3 options)
    1. MAAS (rbasak) for the future, but currently severe shortage of available systems
    2. panda testing, provisioner available today (nuclearbob to integrate from rbasak's, also carlos)
    3. ARM server/netboot testing via QEMU-libvirt on x86 (needs investigation)
- Local machine
    LXC, KVM to match the current iso testing framework (installer testing)
- Custom machine provisioning (ISO/Kernel/InitRD/Preseed)
     * Software virtual machine support
- Config file
max
- Support for users other than utah
Low priority (max)
- Possible rename of catalog to manager/provisioner/provisioning manager/etc.
rename to inventory (max)
- Different provisioning managers: (sqlite does table locking, so multi-user will just be single threaded)
   - Smarter SQLite
   - Postgres
   - Libvirt (might require an external datastore as well)
Low priority, we can do it quickly when it becomes a problem (max)
- Percentage output/other output changes to make things jenkins-friendlier
(discussion to be had about how to do this)
- check integrity of the image before proceeding (psivaa)
- Remote KVM (nuclearbob)
- Scheduling SUTs (mmrazik)
  - how to achieve my tests don't collide with other tests
  - if my test "breaks" the machine (e.g. panic) I'd like to keep the machine locked so I can examine what is left there. I.e. a manual release of SUT back to testing pool
* Test runner
- Create argument parser that supports versions of python older than 2.7 (need a decission)
- Create a result object capable of serializing its own data (joe)
- Pass master.run overrides into the test suite (joe)
- Use --master-runlist flag in self tests once it's been added (joe)
- make sure a new Result object is created for each suite. (joe)
- add support for 'run_as'. (joe)
- add proper support for testcase types as we currently only support userland. (joe)
     - This is currently done by excluding 'command' in the tc_control file for build-only tests.
- ts_control timeout is the default for all test cases in the suite. [DONE]
- 0 timeout means no timeout.
- testsuite layout (joe)
- # proposed layout of UTAH directory
       /var/utah/ <-- should be /var/lib/utah or something for FHS compliance (joe to verify it's done) [DONE]
            master.run
            suite_one/ # the Runner will create this and run the fetch_cmd in here
            suite_two/ # the Runner will create this and run the fetch_cmd in here
                suite_two/ # fetched
                    test_one/
                        test_one.py
                    test_two/
                        build.sh
                        test_two.py
                        cleanup.py
                    tslist.run
                    ts_control

[gema] technical list for utah
[rbasak, gema, nuclearbob, joe, ?jamespage] Think about how to do multimachine testing

## Documentation
- man/info pages (nuclearbob)
- Developer-oriented documentation (i.e., how to use this from python instead of one of our scripts), improve and keep up to date
- "How to write a runlist" [DONE]

Feature Request:
- expose virtual watchdog via libvirt
[nuclearbob] libvirt ARM support

Work items:
[rbasak] Provide MAAS scripts to integrate provisioning method: POSTPONED
[nuclearbob] Work with rbasak to integrate MAAS provisioning: POSTPONED
[nuclearbob] Integrate rbasak's panda scripts into UTAH: POSTPONED
[rbasak] Provisioning: Investigate ARM server/netboot testing via QEMU-libvirt on x86: POSTPONED
[mrazik] Scheduling SUTs: DONE
[joetalbott] Clean up client tests: DONE
[javier.collado] Complete support for 'run_as': DONE
[nuclearbob] expose virtual watchdog via libvirt: POSTPONED
[nuclearbob] libvirt ARM support: POSTPONED
[joetalbott] Support any parameter from the tslist.run should be inherited from ts_control and only have to define in tslist.run if it needs to overwrite the value in the ts_control file: DONE
[joetalbott] Add insource documentation to the client: DONE
[javier.collado] Figure out how to regenerate automatically the documentation on the wiki when a new commit is submitted: DONE
[nuclearbob] Do utah testing: DONE
[javier.collado] Do utah testing (include and exclude tests): DONE
[joetalbott] timeouts changes (see design document regarding adding overrides to master.run and allow master.run to override testsuite metadata and testsuite metadata to override testcase metadata (was "Pass master.run overrides into the test suite"): DONE
[javier.collado] Write new test cases to figure out the best way to organize them (do useful test cases that we can keep for smoke testing): POSTPONED
[joetalbott] Make posible nested runlists: DONE
[joetalbott] Document how to do reboot testing: DONE
[joetalbott] Enable the client to run the same test suite more than once: POSTPONED
[joetalbott] Add support for versioned runlists and control flow (while/for and if), i.e. version, timeout, master runlist level: POSTPONED
[javier.collado] Add support for XUnit output: POSTPONED
[joetalbott] Change name to UTAH: DONE
[hggdh2] Create a mailing list to keep everyone up to date (https://lists.ubuntu.com/mailman/listinfo/ubuntu-utah-devel, <email address hidden>): DONE
[joetalbott] Create a base job for utah tests to be run daily and automatically on dev and trunk: DONE
[nuclearbob] Replicate joe's test job for all the provisioning methods that are available and should be working: DONE
[rbasak] Provide nuclearbob with script to provision panda: DONE
[hggdh2] Create jenkins jobs to test panda with UTAH: POSTPONED
[nuclearbob] Provisioning: LXC: POSTPONED
[nuclearbob] Provisioning: Custom machine provisioning (ISO/Kernel/InitRD/Preseed): DONE
[joetalbott] Support for users other than utah: POSTPONED
[nuclearbob] Provisioning: add config file: DONE
[nuclearbob] Provisioning, low priority: rename catalog to inventory: DONE
[nuclearbob] Provisioning: Different provisioning managers (table locking, see whiteboard): POSTPONED
[psivaa] Integrate static analysis of images into UTAH: DONE
[nuclearbob] Remote KVM (?): DONE
[joetalbott] Create a result object capable of serializing its own data: DONE
[joetalbott] Make sure a new Result object is created for each suite: DONE
[joetalbott] Add proper support for testcase types as we currently only support userland: POSTPONED
[joetalbott] ts_control timeout is the default for all test cases in the suite: DONE
[joetalbott] testsuite layout: DONE
[joetalbott] Proposed layout of UTAH directory /var/utah/ <-- should be /var/lib/utah or something for FHS compliance (joe to verify it's done): DONE
[joetalbott] How to write a runlist: DONE
[javier.collado] Provide feedback to user in case of failure (file not found, ssh connection problem, etc.): DONE
[javier.collado] Enable utah to download from lp a runlist and preseeds without needing to provide an http location of the file (it removes the complexity of having to keep the link up to date pointing at the latest version): DONE
[javier.collado] Wiki documentation (or other stuff for new users of utah): DONE
[nuclearbob] Developer-oriented documentation (i.e., how to use this from python instead of one of our scripts), improve and keep up to date: DONE
[nuclearbob] Add insource documentation to the provisioner: DONE
[nuclearbob] Work with pitti on packaging UTAH in a better way: DONE
[nuclearbob] send a question to delete the obsolete ppa to launchpad itself https://answers.launchpad.net/launchpad/+addquestion: DONE