QA tasks for 2013

Registered by Henrik Ingo on 2013-02-24

A long list of automated QA that we should build during first half of 2013 / before 3.0 GA. If you want to use launchpad blueprints to write more detailed plans, please create new blueprints and make them a dependency / sub-task to this one.

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Henrik Ingo
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
milestone icon 3.0-beta

Related branches

Sprints

Whiteboard

(?)

Work Items

Work items:
*********Automated testing***************************: TODO
Setup Jenkins in AWS: TODO
Jenkins slaves should be run on-demand to save money / enable more heterogenous slave pool: TODO
Jenkins is GUI based, but can we store the configuration in svn/bzr: TODO
*********Automated packaging***************************: TODO
Automated rpms, debs, tars from Jenkins: TODO
*********Platform support matrix***************************: TODO
Make sure sufficient tests are run on all supported operating systems: TODO
Actually testing all the different installation packages (tar, rpm, deb...) and not just "make test" from source tree: TODO
Test things that users actually do, such as "service mysql restart", cluster bootstrap with various methods: TODO
Test upgrade path for each of these, rpm->rpm, tar->tar, etc: TODO
Rolling upgrade can be postponed until we are better at automating clustered tests?: POSTPONED
*********MySQL Test Run for single node******************************: TODO
Disable all tests that don't work, and start routinely run what is left as regression test suite: INPROGRESS
Make all MTR tests work and enable them back, when wsrep_on=off: TODO
Make all MTR tests work for single node with wsrep_on=on: TODO
*********MySQL Test Run for multiple nodes*************************: TODO
To be decided how we run tests for clusters, candidates below: TODO
MTR can run multiple mysqld on localhost, could use same approach for wsrep: TODO
kewpie: TODO
ANSTE cluster testing framework, see http://en.oreilly.com/mysql2009/public/schedule/detail/6757 : TODO
something else: TODO
first run same MTR tests against single master, other nodes slaves: TODO
also find out a way to make queries with multiple nodes: TODO
*********Test coverage************************************: TODO
Go through all old bugs and make sure there is a regression test: TODO
Create test plan (ie a written document) that lists all major use cases that we claim to support (mysql replication compatibility, 5.6 gtid, foreign keys, etc...): TODO
Implement functional tests for those use cases: TODO
*********More tools*********************************************: TODO
Run tests (mtr) under valgrind: TODO
Run tests with lcov: TODO
Check with MariaDB and Percona if there's something they use that is not yet listed here: TODO
*********Cluster robustness tests***************************: TODO
Using same framework as picked above? Test killing a node, network disconnects, split brain, etc: TODO
*********Random Query Generator***************************: TODO
*********Application integration tests***************************: TODO
For example, setup a Drupal stack against Galera, in multi-master configuration, and run the Drupal test suite: TODO
Find other open source apps that have good test suites and do the same, for example e-commerce: TODO
Maybe even some closed source apps could give us a license for the same: TODO
*********Benchmarks as regression tests***************************: TODO
sysbench: TODO
DBT2 or percona-tpcc: TODO
Could we have something nastier?: TODO
Drizzle test suite includes a tool that will compare a sysbench and DBT2 run against previous results and return failure if results are more than 5% lower than previously: TODO

This blueprint contains Public information 
Everyone can see this information.