Enhancements for the test suite

Registered by Yann Pouillon on 2017-11-22

The test suite available in the revno 646 of the trunk is a bit limited regarding automation and validation. Within the RETOS project, we propose a simple YAML-based enhancement of the output of SIESTA associated to a small set of post-processing Python scripts.

Blueprint information

Status:
Not started
Approver:
Alberto Garcia
Priority:
Undefined
Drafter:
Yann Pouillon
Direction:
Needs approval
Assignee:
None
Definition:
Drafting
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Whiteboard

Yann: I already have a very basic YAML dumper in lp:~pouillon/siesta/trunk. You can have a look at it to understand what I have in mind.

Nick: While I think YAML is probably a bit easier to use, why not simply use the XML file which contains the same (and more) information? Currently the XML file is heavily integrated with AiiDA to extract the same information for comparison and extraction. XML should be as easy to parse in python as YAML. No?

Yann: I'm a big fan of YAML because of its readability, its stricter design principles, and the absence of the "element vs. attribute" dilemma. Technically speaking, I don't see any major problem in using XML. However, my personal concerns about security push me towards YAML, which has a lot less vulnerabilities than XML and provides mechanisms for a safer use of the data.

Yann: Actually, the idea of using YAML came during our last RETOS meeting with Emilio, while discussing ways to improve the post-processing of test results, as part of the project.

Nick: I am also fond of YAML. However, I think for all practical purposes everything is already implemented with the XML output. So to reduce double work I think that would probably be the best way forward. But essentially I think this is up to Alberto, any thoughts Alberto?

Alberto: I think that a compromise might be reached: if we identify a (small) subset of output quantities that are worth looking at for checking purposes, then those might be output in a small YAML file (maybe even produced after-the-run by a small script that extracts them from the XML file, if the latter
contains all that is needed -- if not, it might be worth adding it). The smaller YAML file might be more manageable.
On the other hand, I am surprised by the fact that you are using raw fortran I/O to generate the YAML output. I am sure that there must be various wrapping packages. In particular, the BigDFT project has
one within its FLIB library, which is now released separately.

Yann: To summarize our discussions on Skype, YAML is not needed to emit YAML documents, which lets any code produce YAML data without introducing any new external dependency. OTOH, BigDFT is using YAML for its input files as well, which means it has to parse it, hence the Futile library (Flib has been renamed Futile). In any case, once the mechanism is in place, it will be quite easy to improve.

Yann: Another item that might be relevant is that the current subdirectories of Tests/ should be considered more as example use cases than fully-defined tests. We should thus select a representative subset of these examples to define the scope of the first iteration.

(?)

Work Items

Work items:
Write the Fortran module: DONE
Write the post-processing script: INPROGRESS
Discuss tests specifications: TODO
Select a subset of tests to upgrade: TODO
Hand over to Simune: TODO
Merge into the trunk: TODO

This blueprint contains Public information 
Everyone can see this information.