Juju Charm Unit Tests

Registered by Jorge O. Castro on 2012-04-30

Rationale:
This BP is a follow on to Automated Juju Charm testing work done in the Precise cycle. Further tooling is need to ensure automated Charm testing is accurately exercising Juju charms in all available providers.

Goal:
CharmTester is automatically running in all supported Juju providers.

Blueprint information

Status:
Started
Approver:
Antonio Rosales
Priority:
High
Drafter:
Ubuntu Server Team
Direction:
Approved
Assignee:
Mark Mims
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Beta Available
Milestone target:
milestone icon ubuntu-12.10
Started by
Kate Stewart on 2012-07-10

Related branches

Sprints

Whiteboard

charm unit tests

recap... we automatically run:
  - installation
  - graph test
we'd like to additionally run
  - charm tests

read the spec lp:juju in the "docs" dir... look for something like charmtests.rst
http://bazaar.launchpad.net/~clint-fewbar/juju/charm-tests-spec/view/head:/source/charm-tests.rst

policy stuff?
tests required? just encouraged for now

additional testing plugins possible:
  - nessus, etc against the running service
  - app-armor complain against running service
  - lint-type 'charm proof'
  - upgrade-charm tests
  - testing spread with add-unit
  - coverage ???
  - watching relation settings??????

please put all projects in one of:
charm-tools
juju-jitsu
charms
juju

environments - include all of them, including MAAS

support MAAS, other distro needs

enable manual kicks from ~charmers (charmbot, but we prob wanna do that through openid over the web) - parameterized jenkins tests, both official and against specific distro needs

charmtester is a *charm* in the charm store - test my custom stack of services is readily doable (contributions are welcome as always)

jenkins file bugs when charms fail (close them when they pass again?)
try for "maintainer" notification through launchpad

how do long-failing tests relate to "series" freeze???

coverage tools possible? check out next cycle

load testing for individual charms? for example, siege all charms with http interface

scale testing? weekly large runs? really a _juju_ test and less a _charm_ test

---
User Stories:
  *Levi has just submitted and gotten his new Charm approved for inclusion in the Juju Charm store. Levi and other Juju community members would like to ensure his new Charm is staying functional with updates, and properly interacting with related services. Furthermore, Levi would like additional tools to give him further insights on how to improve his Charm.
  *Andrew has submitted his charm and gotten it accepted in a previous Ubuntu release. Andrew would like to regularly confirm his charm is running appropriately in the current release under development.
  *Naomi is interested in using MAAS/OpenStack/EC2 to deploy services via Juju. She would like to review charm testing results for a given Ubuntu release and have insights into how charms are performing in that release.
  *Sarah is are regular charm and has a few charms she is maintaining. She would like to get Launchpad notifications when one of her charms fails in CharmTester. Sarah would like to easily look at a testing reporting interface and drill down to the logs to start the debugging process.

Test Plans:
  *Run charm tester against new and current charms in supported Ubuntu releases against supported Juju providers.

Assumptions:
  * Juju providers are functional to test in

Release Notes:
  * Add availability notice and information on CharmTester.

(?)

Work Items

Work items:
documentation for unit tests: INPROGRESS
write examples of unit tests on "flag-bearing charms" (example charm): DONE
[jimbaker] make 'jitsu run-tests' aware of error codes (Jitsu watch): POSTPONED
[jimbaker] improve charmtester logging information: DONE
[hazmat] charm proof testing: DONE
keep everything green (ongoing task): DONE
[mark-mims] make charmtester easier to use one-off by anyone: DONE
[mark-mims] make charmtester whitelist: DONE
extend charm tools to help generate tests: POSTPONED
run charmtester against maas environment: TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.