Introduce test definitions in separate files

Registered by John H. Embretsen on 2010-02-11

Currently, a Random Query Generator test is not well-defined. However, when testing a product using resources from randgen, it is convenient (for repeatability, reporting, etc.) to be able to define something we call a 'test'. The behavior of a test is determined by the grammar used, data generator settings, various RQG options (threads, Validators, Reporters, ...) etc. In this blueprint, we propose a way to combine all such settings into a human (and programmatically) readable file.

Blueprint information

Status:
Not started
Approver:
None
Priority:
Medium
Drafter:
John H. Embretsen
Direction:
Needs approval
Assignee:
John H. Embretsen
Definition:
Drafting
Series goal:
Accepted for 2.0
Implementation:
Deferred
Milestone target:
None

Related branches

Sprints

Whiteboard

2010-05-06: Deferred due to shifting priorities.

Added 2010-02-18:

Philip Stoev provided feedback on the following:
   - It should be possible to make a test file self-contained. This means that it should be possible to embed grammars etc. in the test definition file itself. This may e.g. reduce the number of configuration files present in the repository. Note that it should still be possible to refer to independent grammar files, in case several tests use the same grammar. This feature would also make it easier to report bugs and reproduce test failures, as only one file reference would be needed.
   - The RQG framework should be able to generate such test files based on its current set of options, grammar etc. This is especially useful in relation to RQG's simplification mechanisms, which may give the user a grammar and/or a set of options that is not pre-packaged in any file.

Current proposal:
----------------------------

Utilize the RQG's Properties.pm module. WIth this, it is easy to read options from both the command-line and a separate file, and to merge these automatically.

Examples which use this facility already are utilities such as util/simplify-mysqltest.pl, util/bughunt.pl, gentest.pl.

The file containing the test configuration defines an anonymous Perl hash, with possible nested hashes for special groups of options.

Example format:
---------------------------

example_test.cfg (extension either ".cfg" or ".test"):

-----------------------------
# Comments here, for describing the test, why settings are as they are, etc.
#
# We may introduce options for providing test descriptions and such as well,
# so that it may be passed along to reporting systems.
#
# Some variables may have default values set by the RQG framework or wrapper scripts.
# Full set and explanation of options to be provided later.

{
    name => 'rqg_example_test',
    category => 'runtime',
    candidate => 'true',
    grammar => 'conf/example.yy',
    threads => 10,
    mysqld => {
        'log-output' => 'file'
    }
}

-----------------------------

Here, options for the wrapper script (interpreter) or the RQG framework itself are defined on the top level.
Options for external products such as the database server under test are defined as a nested structure (hash). In this example one option is set for mysqld, the MySQL Server.

-----------------------------

We will introduce a script (perhaps later even a module) that is able to kick-start a test based on options taken from the configuration file and/or command-line options.

Command-line options will be able to override options from file.

Other considered formats:
 - XML. Universal and pretty standard, well-supported format. However, parsing requires non-core perl modules with possibly known issues on some platforms.
 - JSON (JavaScript Object Notation). Although standard and with relatively good Perl support, there are no apparent advantages compared to the other suggested format. Somewhat increased complexity that is not needed at this time. Requires non-core Perl modules.

Since we, for now, assume that we will stick to perl within the RQG, and that we may relatively easily create tools for converting tests from perl data structures to other formats if necessary, perl data structures seem like the best choice.

---------------------------------------------------------------------------

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.