Stable test suite

Registered by Daniel Nichter on 2011-08-05

The current test suite, which was inherited from Maatkit, needs to be overhauled because over the years it has become unstable under its own size and weight, so to speak. Whereas we tested Maatkit and my vboxes or Baron's, we can no longer do that. We must use http://jenkins.percona.com/ for true continuous integration. Furthermore, we must test on multiple platforms and three versions of MySQL (5.0, 5.1, and 5.5).

At the moment, all the tests work, more or less. prove -r lib/ does not work (unless you're really lucky), but if you run the sub-test suites individually, eventually they will all work. It is precisely this instability we must fix. prove -rs lib/ should work 100% of the time. That way, when a test breaks, we know it's because the code we just merged is causing a regression.

The root of the instabilities is usually related to time. My laptop is fast (dual core, SSD HDD, etc.), but the Jenkins boxes can be *very* slow. Many tests load something and either don't wait or wait a second and then presume the test is ready to proceed. This causes non-determistic failures which are a true pita to track down.

Other causes of instability are being revealed by Jenkins and all its platforms. For example, I used to think 32-bit vs. 64-bit didn't matter because we're using Perl, but now it does matter because we're also using sh and system commands. On 32-bit platforms, for example, awk uses 32-bit integers which is problematic.

Another serious problem that cripples the test suite is that some tests break replication or leave the sandboxes servers not in their original state. MySQL 5.5 seems to cause this pretty easily.

Another problem is PERCONA_TOOLKIT_BRANCH (formerly MAATKIT_WORKING_COPY). This makes it possible to run a test from any directory, but this is no longer manageable because, unlike svn branches which barely work, bzr branches work well so we'll probably end up having many of them.

Yet another problem: this isn't serious but it's highly desirable because it's highly indicative: some tests may leave temp/junk files or directories after running. Or, the tests aren't idempotent and they should be. Every single .t file should be very well-bahaved, tidy, etc.

Finally, the sandbox/ scripts aren't ideal. I wrote them when my Bash wasn't very strong. They all need to be updated and made smarter, more reliable, easier to use, etc.

We need to make a very large investment to make the test suite truly stable. Percona Toolkit seems have to have gain a much wider user base compared to Maatkit, no doubt due to the Percona name brand. So here's what we need to do: (TBD in Las Vegas)

###
### 1.

Blueprint information

Status:
Complete
Approver:
Percona Toolkit developers
Priority:
Essential
Drafter:
Percona Toolkit developers
Direction:
Approved
Assignee:
Daniel Nichter
Definition:
Approved
Series goal:
None
Implementation:
Implemented
Milestone target:
milestone icon 2.1.2
Started by
Daniel Nichter
Completed by
Daniel Nichter

Sprints

Whiteboard

Changes made in lp:~percona-toolkit-dev/percona-toolkit/stabilize-test-suite:

* Daemon.pm now always closes STDIN (used to check -t STDIN but -t is not reliable) -- bug 944420
* Remove Daemon.t test "Does not reopen STDIN to /dev/null when piped"
* Added PTDEVDEBUG
* Made default wait and retry times in PerconaTest shorter and fatser
* Use Perl instead of awk in safeguards.sh to calculate values to avoid 32-bit awk init limit -- bug 940503
* Added sandbox/jenkins-test
* Use PerconaTest::wait_for_table() in many places
* Shortened output of MySQLProtocolParser.t
* Conditionalize IO::Uncompress::Inflate tests in MySQLProtocolParser.t
* Don't require Digest::Crc32 to test our crc32()
* Remove t/lib/bash.t
* Remove t/lib/samples/bash/dummy.sh
* Symlink t/lib/bash/<bash lib>.t to util/test-bash-function
* Bash tests must now call "plan N" instead of setting TESTS=N
* Make util/test-bash-functions accept single arg: symlinked test file name
* Add "IF EXISTS" to DROP TABLE in t/pt-online-schema-change/basics.t
* Fix t/pt-query-digest/daemon.t greps
* Make util/check-dev-env check/print all modules instead of dying on the first not-installed module

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.