split serial.log in deploy, test and submission logs

To make this part work sensibly we need to track where the command execution is with regards to the job that was submitted.

This would also affect the way lava-dispatcher stores serial log attachments (perhaps we should just split that to multiple attachments, I'm fine with that solution).

The scheduler UI would need updates to track this properly. The dashboard would be mostly unaffected as it does not treat attachments specially based on their name.

Couldn't we look for something like the logger messages and use those as placeholders to roll up the text between them? This would make it much easier to browse the raw log

My own plans for this sort of thing have been to have the dispatcher communicate (using the --oob-fd thing I added for reporting the bundle url) which action it was executing, and then divide up the log by action. We could then present the output in the scheduler as an accordion type thing so you could instant zoom in on say the logs of the deploy action. It would also be nice to fold away the reboot logs by default as they really get in the way a lot of the time.

Any move from the current approach where we just dump unstructured data into a file would probably be quite difficult though -- I guess we could continue to do that, but also record a series of offsets into that file to know which bits to display? That sounds possible, but it would also be nice to display log messages, commands we execute and output from said commands differently, and that seems to push towards a more structured format -- maybe something a sequence of json things?

{'type': 'logmsg', 'content': '<INFO> executing linaro-media-create'}
{'type': 'command', 'content': 'root@linaro# [rc=0] linaro-media-create ...'}
{'type': 'output', 'content': 'Unpacking rootfs...'}

Maybe this is overkill though?

Apologies for the brain dump.

I think the new log view is better for this -- we probably need to tweak how the dispatcher logs to make best use of it though.