QA Regression Testing codebase improvement and future development

Possible things to add:
Standardized Multiverse requirement
  Submitted a patch for this. Discussion indicated multiverse is enabled by default on standard installs, so maintaining this looks to be unnecessary
Standard method for specifying a clean system OR anti-dependencies (i.e., apache2-mpm-worker anti depends on apache2-mpm-event)
Service for automated remote testing
Standardized source repository requirement
THIS SESSION IS ABOUT QRT test suite
[COMMENTS]
- Please see, in case you want to have some unit tests added to the QA lab: https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions
- There is a web that can install dependancies and run them automatically (QRT test suite, security test suite)
- Specifications and test case requirements per release are not clear and are giving problems in terms of running the test cases reliably. It needs to be done in a way that you don't need to execute the script to know the dependancies
- Some tests have conflicting dependancies.
- Currently there's one big package with all the tests on it, but maybe would be useful to split it in different packages , maybe per target subsystem (one package for kernel, one for ... etc)
- The data is also included in the package, and it is around 200 MB
- Most tests have set up and clean up functionality
- The tests are done in a way they can be run as soak tests but it is not guaranteed that they'll work in this way
- Metadata should be added so that we know that the test case is not going to destruct the system it runs on, also architecture
- We need to make sure we run the tests on bare metal
- Some of the kernel tests fail because the test is written for AMD64, this is a bug on the test code
- Maybe VM or LVM snapshotting could be used for running these tests
- Are there plans to start using autotest? there are plans to begin using that on select packages as disucssed in previous session, there aren't currently plans for using autotest for the qa-regression-testing code
- If not autotest, what will be the framework for running QRT? There are a couple of bash scripts that consume that metadata, and there are other scripts that consume it
- Could we link the creation of the VM to the tests themselves?
- wiki.ubuntu.com/SecurityTeam/TestingEnvironment has information on how to run the tests
- Some tests do depend on an initialization of one particular test
- VirtualEnv and/or Buildout can solve the problems discussed in different ways
[ACTIONS]
- A tag to be added to the test cases when they are potentially harmful to the set up they are going to be running on
- Target architecture tag should be added to the metadata (should be up to test to detect arch, not the framework?)
- Add an isolation tag for tests that are incompatible with some packages or need to run on an isolated environment (a boolean)
- Look into moving to autotest instead of writing new scripts for that
 - autotest wrapper needs to learn how to install packages, drop prives, and run a script (for most qrt scripts)
  The wrapper is written. Autotest isn't packaged. I'll do that.
- Look into using VirtualEnv and/or buildout

Work items for precise-alpha-1:
[nuclearbob] Add multiverse tag: DONE
[nuclearbob] Investigate moving to autotest: DONE

Work items for precise-alpha-2:
[nuclearbob] Add source repository tag: DONE
[nuclearbob] Package autotest: DONE

Work items for ubuntu-12.04-beta-1:
[nuclearbob] Setup autotest (talk to Robbie Williamson, Manoj, Andy): DONE

Work items for ubuntu-12.04-beta-2:
[nuclearbob] Add tag for potentially destructive tests: POSTPONED
[nuclearbob] Get compilation into buildd/a predictable toolchain: POSTPONED
[nuclearbob] Add architecture tag: POSTPONED
[nuclearbob] Add isolation tag: POSTPONED