Phatch test suite 0.3

Registered by Nadia Alramli on 2009-10-06

TODO:
- fix crashing (on windows) if there is no USER_LOG_PATH yet
- make sure rename action does not 'delete' test image
- run actionlists using phatch functions rather than calling it as a script via subprocess. Needed for profiling -> don't know if this the best approach, maybe better add a --profile option to phatch
- use CONFIG_PATHS for all phatch paths
- try out with different user settings (stop for errors, skip existing files, ...)
- how to handle plugins (eg loslessjpeg with gif)
- testing workflow:
  - for each commit to the stable branch an extensive test has to pass first
  - for your own branch you must run a very lightweight with each commit, less leightweight at regular intervals
  - doctests must pass for every commit
- how to deal with old, existing code
- study possibilities for turning it into a robit gatekeeper for committing to trunk and send report to Stani when done with request to merge
- follow PEP8!
- integration with clone digger? http://clonedigger.sourceforge.net/
- integration with pylint/pep8/pyflakes or ...
- Stani likes to have an option only to rerun tests which failed previous time. So we can go to zero bugs more quickly and later run the full suite to see if no new errors were introduced with the fixes
- keep statistics of succes/failure of tests and maybe execution time, so we can maybe use that to develop time based test suites (eg suppose you only have one hour time)

Extra:
- add a command line option for a revision number (so we can compare performance across revisions)
- we need to guarantee that the executed action lists are comparable (eg we could have the polaroid benchmark)
-I found the action list naming somewhat cumbersome. It was difficult to find the one matching to the failed test due to the length of the filename. Perhaps it would be neater to construct a directory tree representing each choice and stash the action lists at the end of the branches or something similar... Alternatively it would be nice to automatically copy failed action lists to some predefined place.

QUESTIONS
- What is our strategy to testing in the bigger picture: pyflakes, pylint, ...

Blueprint information

Status:
Not started
Approver:
None
Priority:
High
Drafter:
Nadia Alramli
Direction:
Approved
Assignee:
Nadia Alramli
Definition:
Approved
Series goal:
Accepted for 0.3
Implementation:
Not started
Milestone target:
None

Related branches

Sprints

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.