Better reporting of errors from databases
Today, too many non-fatal errors are reported when RQG is running. This is annoying, and may mask other problems that might need attention. By default, non-fatal errors should not be reported, as they are expected from most grammars (due to the randomness of the queries). However, it should be possible, to turn a certain level of reporting on for grammars where non-fatal errors are unexpected, and for grammar programming/
We should also have a mechanism to regulate the granularity of this reporting.
Also: All possible MySQL errors (and for other databases, possibly) should be classified for this purpose (today only a subset is classified). One possibility is to use the SQLState class (if available, and correctly implemented by the database) as defined in the standard for automatic classification. This has been attempted in the JavaDB and Postgres executors.
Blueprint information
- Status:
- Not started
- Approver:
- None
- Priority:
- High
- Drafter:
- Bernt M Johnsen
- Direction:
- Needs approval
- Assignee:
- Bernt M Johnsen
- Definition:
- Discussion
- Series goal:
- Accepted for 2.0
- Implementation:
-
Not started
- Milestone target:
- None
- Started by
- Completed by
Related branches
Related bugs
Sprints
Whiteboard
Bernt: I think this should be set by a separate option, not --debug.
John: If we need to adjust granularity of different levels of fatality among errors, we seem to need other options anyway, so in that case using --debug for this would not be the best solution. In an E-mail Bernt suggested using Perl code "plugins" to specify which errors to show or ignore etc., and I think this sounds like a good idea. In that case I guess we need an option to specify the plugin only...
Bernt said:
"What about a plug-in structure? A set of plugins on the general cases (ignore all non-fatal errors, print nont-fatal errors first time they appear and print all non-fatal errors). Then plugins could be written for more specific cases e.g. with pattern matches, db specific errror codes etc."
Bernt: Philip has also suggested that the filtering mechanism which allows errors to be reported only once and then surpressed should be enhanced. Now it's per thread, so if you have 100 threads you will get the same error reported 100 times. SHould most likely be done as a part of this blueprint.
dbennett455 said on 2014-09-11: roel11 and I had some discussion on this blueprint yesterday and came up with some ideas. That being said, let's start out with some basic requirements. I'll start with some I have been thinking about today along with Bernt's existing suggestions.
Requirements:
-------
1. Existing application funtionality should be uneffected.
This goes without saying. All existing grammer logic, data generation, options and parameters should function as they currently do.
2. Existing output format should remain intact.
There are a number of existing QA scripts and build systems that may depend on the existing output of randgen for proper operation. This existing output format intact and preferably be the default output format for the initial version of the reporting changes.
3. Reporting should be easier to read and more concise.
Duplicate errors and non-essential reporting should be easily user configurable to allow. As Bernt suggested above, this should include the ability to define configurable output filters to remove redundant information along with other reporting output configurations
4. Ability for 3rd party applications to easily parse the reporting output.
Automated QA and build systems should be able to easily and reliably parse the output of randgen to access proper functionality of the underling test target build.
5. Pre-defined output configurations
Ability to define and specify output configurations at run time. Examples, log4j runs quiet without redirection and sends reporting output to different files based on logging level (trace,
6. Ability to define custom reporting levels for specific types of output.
For instance, filter or highlight items based on error or warning categories from the underlying target database, cluster provider errors, etc...
Strategy:
- Next step: I am leaning towards standardizing all application output on Log::Log4perl and associated modules for log filtering, output adapters, etc... It is highly user configurable, already a dependency and it satifies the 'plug-in' nature of the solution.
