CI for all click/appstore projects

curent list of tests: https://wiki.ubuntu.com/AppStore/Decisions/ContinuousIntegration
unit tests available for some projects already
 - appstore has some tests already, part of CI infrastructure
 - client (app scope + download service + click) already have some tests as well
 - do we need touch emulator to scale?

server bits are charmed, so could be autodeployed on test system

end to end user experience tests:
 - not available yet
 - appstore integration tests will need the CLI upload API first or we could possibly use Selenium Simple Test to run an integration test including uploading the click-app).

Example Test
 - ubuntu-sdk: builds manifest, builds click package (not on the phone)
 - click package verification (not on the phone)
 - appstore: login (SSO), upload to staging myapps and fill in required meta-data
 - appstore: review-approve app and publish on myapps (publishes the app to the separate click-package-index) (not on the phone)
 - click scope: list of applications which are available (could theoretically run on desktop unity8)
 - click scope: test installation/upgrade (needs touch setup)

selenium tests
autopilot tests for scope preview inside unity

testing errors require recreating errors on both client and server side

test everything seperately in a sandbox, while manually performing end to end testing
 - automated end to end tests are hard, but not impossible

get integration tests going, then construct the end to end testing

What tests need to be written?
tests need to be written in order
 - some tests require dependencies, ie scopes tests need

Possible blockers:
 - autopilot needs to be able to launch from upstart
 - Automated Uploads for click packages 2 months away
   - sst (simple selenium test) should be possible to fill in the gap?
     - need 1 user to upload, need another user to review, publishing package would then require the first user
       - reviewing step still has manual pieces
       - automated script exists for creating users
       - permissions are an issue
     - local server allows setting permissions easy enough, but not production

Unity scope components owned by Roberto Alsina (ralsina)
Software store components (SSO, myapps, click-updown, click-package-index) owned by Martin Albisetti (beuno)
Click owned by Colin Watson (cjwatson)
Security bits owned by Jamie Strandboge (jdstrand)
SDK owned by Zoltan Balogh (bzoltan)
Touch images owned by Sergio Schvezov (sergiusens)

Work items:
[alecu] help with testing installation, upgrading: TODO
[noodles] Ensure that staging click-package-index service is auto deployed on each commit: TODO
[noodles] Ensure that myapps staging is auto-deployed on each commit: TODO
[noodles] Back-door for test user where uploaded app gets auto approved and published: TODO
Write SST integration test for the app submission to myapps, review and publishing (vila or elopio or cgoldberg, maybe): TODO
Write autopilot tests for the preview components of the dash. (elopio, maybe?): TODO
Prepare the deploying of app servers from Jenkins to be usable by slaves so tests will run against local isolated and controlled servers. (vila maybe?): TODO
[fginther] Work with alecu and elopio to discuss needed integration tests and possible assist with autopilot training: TODO
[dholbach] start discussion on the mailing list about manual tests and ask where community people could help out with: TODO
[dholbach] set up follow-up call with noodles, fginther, elopio, alecu, vila to put together lists of actual tests to run and start filing bugs: DONE
document which projects their team works on and which tests are already there: TODO
figure out which high-profile tests are still missing and file bugs for them: TODO
bring up on the mailing list which integration points with other projects there are: TODO