Juju Charm Unit Tests
Rationale:
This BP is a follow on to Automated Juju Charm testing work done in the Precise cycle. Further tooling is need to ensure automated Charm testing is accurately exercising Juju charms in all available providers.
Goal:
CharmTester is automatically running in all supported Juju providers.
Blueprint information
- Status:
- Started
- Approver:
- Antonio Rosales
- Priority:
- High
- Drafter:
- Ubuntu Server
- Direction:
- Approved
- Assignee:
- Mark Mims
- Definition:
- Approved
- Series goal:
- Accepted for quantal
- Implementation:
-
Beta Available
- Milestone target:
-
ubuntu-12.10
- Started by
- Kate Stewart
- Completed by
Whiteboard
charm unit tests
recap... we automatically run:
- installation
- graph test
we'd like to additionally run
- charm tests
read the spec lp:juju in the "docs" dir... look for something like charmtests.rst
http://
policy stuff?
tests required? just encouraged for now
additional testing plugins possible:
- nessus, etc against the running service
- app-armor complain against running service
- lint-type 'charm proof'
- upgrade-charm tests
- testing spread with add-unit
- coverage ???
- watching relation settings??????
please put all projects in one of:
charm-tools
juju-jitsu
charms
juju
environments - include all of them, including MAAS
support MAAS, other distro needs
enable manual kicks from ~charmers (charmbot, but we prob wanna do that through openid over the web) - parameterized jenkins tests, both official and against specific distro needs
charmtester is a *charm* in the charm store - test my custom stack of services is readily doable (contributions are welcome as always)
jenkins file bugs when charms fail (close them when they pass again?)
try for "maintainer" notification through launchpad
how do long-failing tests relate to "series" freeze???
coverage tools possible? check out next cycle
load testing for individual charms? for example, siege all charms with http interface
scale testing? weekly large runs? really a _juju_ test and less a _charm_ test
---
User Stories:
*Levi has just submitted and gotten his new Charm approved for inclusion in the Juju Charm store. Levi and other Juju community members would like to ensure his new Charm is staying functional with updates, and properly interacting with related services. Furthermore, Levi would like additional tools to give him further insights on how to improve his Charm.
*Andrew has submitted his charm and gotten it accepted in a previous Ubuntu release. Andrew would like to regularly confirm his charm is running appropriately in the current release under development.
*Naomi is interested in using MAAS/OpenStack/EC2 to deploy services via Juju. She would like to review charm testing results for a given Ubuntu release and have insights into how charms are performing in that release.
*Sarah is are regular charm and has a few charms she is maintaining. She would like to get Launchpad notifications when one of her charms fails in CharmTester. Sarah would like to easily look at a testing reporting interface and drill down to the logs to start the debugging process.
Test Plans:
*Run charm tester against new and current charms in supported Ubuntu releases against supported Juju providers.
Assumptions:
* Juju providers are functional to test in
Release Notes:
* Add availability notice and information on CharmTester.
Work Items
Work items:
documentation for unit tests: INPROGRESS
write examples of unit tests on "flag-bearing charms" (example charm): DONE
[jimbaker] make 'jitsu run-tests' aware of error codes (Jitsu watch): POSTPONED
[jimbaker] improve charmtester logging information: DONE
[hazmat] charm proof testing: DONE
keep everything green (ongoing task): DONE
[mark-mims] make charmtester easier to use one-off by anyone: DONE
[mark-mims] make charmtester whitelist: DONE
extend charm tools to help generate tests: POSTPONED
run charmtester against maas environment: TODO
Dependency tree
* Blueprints in grey have been implemented.
