Lab administrative tasks Q

Registered by Gema Gomez on 2012-01-27

This blueprint is created to as a way of tracking the work the Platform QA team is doing on the lab during the Q cycle.

Blueprint information

Status:
Complete
Approver:
Gema Gomez
Priority:
High
Drafter:
Larry E Works
Direction:
Approved
Assignee:
Canonical Platform QA Team
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Implemented
Milestone target:
milestone icon ubuntu-12.10
Started by
Gema Gomez on 2012-05-15
Completed by
Gema Gomez on 2013-11-11

Related branches

Sprints

Whiteboard

Magners Lab - Lexington
1 - Lab re-organization
    a - Re-arrange server racks and shelving units
    b - Install cable troughs
    c - Re-run all cables; match CAT-5 color scheme used by IS
    d - Label all cables
2 - Re-IP all hardware
    a - Move everything to the 10. IP range
    b - Sub-net for management devices (KVM, CDU, Serial Switch, Managed Switch(es), Network Camera, etc ...
    c - Seperate sub-net for physical testing hardware
    d - Seperate sub-ner for virtual machines; bridged network interface instead of NAT
    e - Everything should be accessible via VPN connection
3 - Identify additional hardware needed for test cases
    a - Desktop with High-end Nvidia card
    b - Desktop with High-end ATI card
    c - Laptop(s) for external monitor disconnetc/reconnect testing
4 - Install DMZ server
    a - Host VMs that require unfiltered access to the Internet for testing
    b - Serve as firewall for physical hardware that require unfiltered access to the Internet for testing
1SS Lab - Boston
 1 - Install additional servers; configure with Openstack to host future VMs for testing
 2 - Migrate existing testing VMs to the stack
 3 - As current VM hosting servers are freed up add them to the stack
 4 - Install DMZ server
     a - Server as firewall for systems/VMs requiring unfiltered access to the Internet for testing
5 - Reconfigure network
    a - Bridge Virtual networks
    b - Move everything to the 10. IP range
    c - Enable connectivity to all servers/VMs via VPN

Long term goals:

- To get LDAP authentication into the labs
- Add storage (SAN) in both labs for both site-to-site backups and VM
storage
- Re-work the networks in both labs to isolate different teams/projects
testing environment on their own sub-nets
- Add our own firewall server to each lab to route physical and virtual
machines that require direct internet access for testing
- Identify, purchase and install additional hardware (HBAs, managed
switches, UPS', etc...) to support the listed tasks

Ongoing tasks:
- And, of course, document everything that is done
- Identify hardware needed for additional testing

(?)

Work Items

Work items:
[larry-e-works] Figure out if installing a plugin for sub-views would be good for our purposes (i.e. make jenkins usage easier): DONE
[larry-e-works] Figure out how to structure tabs and what is a good reason to have a new tab and what is not. Job listing is growing fast and we need to figure out how to structure things so that they scale: DONE
[larry-e-works] Establish the workflow for creating jobs, removing jobs, modifying jobs. Make sure nobody adds new jobs to the production Jenkins without having tried it first and demonstrated it works in the pre-production environment. Use bazaar to store the jobs and jenkins configuration, people should submit to the branch and then you can catch up on the production servers for updating with the latest changes: POSTPONED
[larry-e-works] Document all available jobs and reasons for them, determine a policy regarding creating new jobs so that we only create new jobs when it makes sense. If there is a job without a reason to be, it should be removed: POSTPONED
[larry-e-works] Consider having more than one internal instance of Jenkins, so that dev teams can access their own and we can separate all our testing from theirs. Or any other logical differentiation that makes sense: DONE
[larry-e-works] Make sure notifications do not turn into spam: DONE
[larry-e-works] Deploy infrastructure monitoring system (Nagios, Ganglia, Centreon or similar): DONE
[larry-e-works] Set up an external SAN to back up jenkins: BLOCKED
[larry-e-works] Private instance: establish data and configuration back up policy, determine how often to back up Jenkins to ensure we can restore it safely and lose no work in the process: POSTPONED
[larry-e-works] Public instance: make sure IS has the same bases covered as with the private instance: POSTPONED
[larry-e-works] Test the backups (private and public), do a recovery trial regularly to make sure we are safe (every six months?): POSTPONED

Dependency tree

* Blueprints in grey have been implemented.