Validation Lab Expansion and Maintenance

Registered by Paul Larson

We are planning to continue to add new test hardware to the validation lab, with a focus on low-cost, high-volume boards. We should discuss what additional infrastructure will be needed to support this such as: networking issues, whether non-arm support is needed, rack density solutions, backups, monitoring, and maintenance.

Blueprint information

Status:
Started
Approver:
Paul Larson
Priority:
Undefined
Drafter:
Dave Pigott
Direction:
Needs approval
Assignee:
Dave Pigott
Definition:
Discussion
Series goal:
None
Implementation:
Started
Milestone target:
None
Started by
Dave Pigott

Related branches

Sprints

Whiteboard

User Stories:

1. As a Linaro engineer, I would like to see test results on Snowball boards
 - Snowball boards shipped this week, need power and serial cables at a minimum
 - ACTION: Dave to order power and serial cables if they are not included
https://blueprints.launchpad.net/lava-dispatcher/+spec/linaro-platforms-o-setup-snowball-boards

2. As a Linaro engineer, I would like to have boards in the lab only support serial over usb
Chaining hubs should be ok, since we have low bandwidth needs
udev rules would need to be written to statically associate boards with usb ids
cable length could be a concern
ash said he has cheated and got 10 meters out of one before
A 49 port usb hub is made by a company in cabridge, ~ 400 UK
ACTION; Dave to get small hub in to test the implementation issues
https://blueprints.launchpad.net/lava-dispatcher/+spec/linaro-platforms-o-serial-over-usb

3. As a Linaro engineer, I would like more reliable access to the validation lab
 - Completely separate networks for Linaro office/val lab
 - Rack mount modem/router for Val lab (something like http://www.draytek.co.uk/products/vigor3100.html) - not expensive £200
     Lines should be distinct. Val lab line should be 5x9s SLA. Office network can be lower spec
ACTION: Dave to spec router, and dedicated line for the validation farm
ACTION: James T to advise on networking hardware
https://blueprints.launchpad.net/lava-dispatcher/+spec/linaro-platforms-o-lab-better-network

4. As a validation lab administrator, I would like to restore back to full operation quickly in the event of a drive/server failure.
 * Online/offsite backup
Noted that even if we lose everything, it can in theory be re-created, so local 2TB storage would be fine.
ACTION: James T and Dave to look at storage options
https://blueprints.launchpad.net/lava-dispatcher/+spec/linaro-platforms-o-lab-storage-backups

5. As a validation lab administrator, I would like the serial console server to "just work" so that it doesn't have to have stale connections cleared to maintain reliability.
 * Do something about cyclades reliability
ACTION: Dave to contact Avocent regarding firmware upgrade and reliability
**Possibly done already by a simple fix, need to recheck in a few months**

6. As a validation lab administrator, I would like to have ample supplies on hand so that new boards can be integrated into the validation farm quickly.
 * Detailed requirements list for board suppliers (e.g. power supplies, power on reset - more?)
ACTION: Dave and Paul to draw up check list so that we can kit up on missing parts before board delivery (particularly power supplies and serial cables.

--------
 * Co-hosting?
Probably not advisable at this time because of the constant support needs and detailed knowledge required to support

Not covered or not applicable:
 * Android support - can't comment, don't know enough about what that means in terms of the Val lab itself
 * Simplify new board deployment
 * Simplify fixing boards in Val lab cabinet(s)
 * Keep historic builds of all boards images roll over every x months

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.