Create per-partition images

Registered by Данило Шеган

LAVA has a need to access individual partitions of the created image so it can more easily restore each one to a clean state. At the moment, this is done on the LAVA side (in a module called lmc_utils), but it would be much better for LAVA if linaro-media-create could produce partition images directly.

Blueprint information

Данило Шеган
Needs approval
Series goal:
Milestone target:
Completed by
Данило Шеган

Related branches



[2012-03-20, Danilo] Added the conversation with Zyga, will clean up and summarise later.
[2012-06-22, doanac] I might be missing something, but it seems low priority. Hopefully we'll have sd-mux to aliviate this need soon, and in the meantime we have something that's "good enough".
[2012-07-20, danilo] Killing this since it's a low priority item that we'll never get to.

<zyga> danilos, hi, could linaro-media-create output partition images instead of full card image given a special option?
<-- mertsas has quit (Ping timeout: 240 seconds)
<danilos> cjohnston, cool, I'll ping you later then (in say 4-5h)
<cjohnston> ok
<danilos> zyga, is that not easy to get from the full image anyway?
<zyga> well, harder
--> dlezcano (dlezcano@nat/ibm/x-wmdlwssdhjjcwooa) се прикључи #linaro
<danilos> zyga, perhaps we can create a small utility which makes that easy (if there isn't one already), I'd rather not add to the tiny set of command line options l-m-c already has
 zyga, unless this is something you find you need very frequently
--> virals (~viral@ се прикључи #linaro
<danilos> zyga, "you" as in you and everybody else :)
<fabo> danilos: changing mabac's approach to simply allow providing a hooks dir sufficient to solve immediate concerns. we can improve feature when more time allows it.
 fabo fabo_
<zyga> danilos, right now accessing the partition means I need to loop-mount with offset that I can get while talking to parted
<danilos> fabo: excellent, thanks, we'll get it done asap in time for 2012.03 release then
<zyga> danilos, lava could use that option
<fabo> danilos: thanks a lot!
<danilos> zyga, right, but for lava, I think it'd make more sense for it to be outside l-m-c, but I am can easily be convinced otherwise; can you please file a bug, and I'll talk it over with the team to see what they think
<-- hongbo has quit (Quit: Leaving)
<zyga> danilos, I'll start a conversation, lmc is very much essential to lava and we were hoping it could grow an option to make a master image
--> davelong (~<email address hidden>) се прикључи #linaro
<-- ryanharkin has quit (Remote host closed the connection)
<danilos> zyga, what do you mean with "master image"?
<-- omnium has quit (Remote host closed the connection)
--> mertsas (~martin@nat/cisco/x-zeuqixfdfkwdtapj) се прикључи #linaro
<zyga> danilos, the master image concept that is essential for all lava work
<danilos> zyga, also, we are about to get a fix for in, would that solve some of the LAVA needs for mounting the individual partitions inside the image perhaps?
<ubot2`> Launchpad bug 751531 in linaro-image-tools "Add support for rootfs post-processing in l-m-c" [High,In progress]
<zyga> danilos,
<-- Sunil_K has quit (Remote host closed the connection)
--> Sunil_K (~kvirc@ се прикључи #linaro
<zyga> danilos, perhaps
<danilos> zyga, thanks, definitely worthwhile to investigate having master image creation supported in l-m-c
<zyga> danilos, currently this process is manual and thus error-prone
--> pfalcon (~pfalcon@ се прикључи #linaro
<fabo> zyga: we talked about using Dave script to generate master image and create a ci job
<zyga> fabo: ci job?
--> ryanharkin ( се прикључи #linaro
<fabo> zyga: ci.l.o (jenkins) job
<zyga> fabo: dave's script is far from perfect at this stage, I'm looking at how things should work
 fabo: I don't see how CI is related to master images
<fabo> manual process should be avoided and master image creation automated
<zyga> fabo: anyway, master images require a "basic" image which in turns require all the knowledge l-m-c has about how to setup an SD card for a particular device
 fabo: I don't see CI as a part of that process
<fabo> "automated"
<zyga> fabo: unless we want to create another entity that knows all about all hardware
 fabo: the right place to keep this knowledge is lmc
 fabo: I'll take automated, thank you :-)
<fabo> nothing prevent lmc in the job
<danilos> zyga, indeed, agreed
--> davidlong (~<email address hidden>) се прикључи #linaro
<-- davidlong has quit (Remote host closed the connection)
--> rogerq (~<email address hidden>) се прикључи #linaro
<fabo> zyga: though, we need to evaluate Dave script. I admit I never seen it.
<zyga> fabo: !evaluate
 fabo: it's not a complete solution
 fabo: I know what the script does
<danilos> zyga, fwiw, while I agree master image creation support would be good to have in l-m-c, I think per-partition images would be a detour we don't really want in l-m-c; as I said, that's just me, though :)
<zyga> fabo: it's 1/3 of the process at most
--> jluisn (~quassel@ се прикључи #linaro
<zyga> danilos, having per - partition images would allow lava progress beyond unreliable master images
 danilos, but perhaps hooks could allow us to do that
--> davmor2 (~<email address hidden>) се прикључи #linaro
<fabo> zyga: haha :)
 it's making his way
<zyga> looking
 fabo: well I'm glad we follow pep8 more
<fabo> I know but it's the 1s time we get this kind of comments... which is good imo
<danilos> zyga, what makes master images unreliable? if it's the way they are created, having it done by l-m-c should improve the situation; if not, what lava does with such images and what could it do better with per-partition images?
<zyga> danilos, a long list of things that starts with them being read-write and the fact that any rouge/acciental test can mount and change the master
 danilos, if masters were always bit identical (on boot) then we'd solve a whole class of problems
 danilos, thus limiting the unreliability to connection issues and bootloader environment
<zyga> (and SD cards dying but we have no way of fixing that with software)
<danilos> zyga, right, I don't see how just using full master images wouldn't solve anything that using per-partition images would: are you suggesting LAVA could more easily then restore each partition to a clean state?
<zyga> danilos, because we'd mount master rootfs via NBD
 danilos, a single machine can happily serve that to a whole cluster of boards, with full snapshots and restore-on-disconnect
 danilos, so ideally we'd have a bootloader on the card and the rest would be written over with the test image 's root filesystem
<-- sumits has quit (Quit: Leaving)
<danilos> zyga, ah, I see; basically, you'd only need the rootfs split out then, right?
<zyga> danilos, we still need to copy the kernel over somehow but I believe that the master / test kernel could also be loaded from the network
 danilos, well the boot partition is also important as it has the initrd/kernel
<danilos> zyga, ok
<-- uweigand has quit (Quit: Ex-Chat)
<zyga> danilos, I think it would be best to keep the whole test image boot similar to normal use case (all from SD)
 danilos, just keep the immutable boot loader in a place that minimizes damage (for example, on upgrades)
 danilos, we still cannot copy the whole card image directly because of this
 danilos, so lava needs to post-process each image (even if we did not call l-m-c to create it ourselves)
--> c10ud (~c10ud@emesene/dictator/c10ud) се прикључи #linaro
<-- gesha has quit (Remote host closed the connection)
<danilos> zyga, right, then it definitely sounds like something that'd be very useful to have in l-i-t
<zyga> danilos, we don't need it tomorrow but just having it would unlock this path
<danilos> zyga, how do you post-process it today? extract partitions directly in lava?
<zyga> danilos, yes
 danilos, let me check and tell you though
<danilos> zyga, heh, you ain't getting it tomorrow either :)
 zyga, no need to be specific, just wondering
<zyga> danilos, yes, we even have a module called lmc_utils ;-)
 danilos, full with scripts that click "yes" for snowball
<danilos> zyga, I'll file a BP with reqs, and just paste our discussion in the notes for now, so we can expand on it later
<zyga> danilos, thanks, please subscribe me to this bp
<-- siam has quit (Ping timeout: 245 seconds)
--> cltang ( се прикључи #linaro
 gesha (~<email address hidden>) се прикључи #linaro


Work Items

This blueprint contains Public information 
Everyone can see this information.