Implement Karma for JS Testing

Registered by Matt Borland on 2015-05-13

Use Karma to run Jasmine tests, allowing for many features such as jUnit reporting, test coverage, live test running, and enhanced IDE features.

Summary
=======

Karma is a test runner framework that provides abstracted, pluggable features for running unit tests. As such, it can run tests from different frameworks (e.g. Jasmine, qUnit), but provides common features. This blueprint describes the advantages of using Karma over the standard Jasmine test runner, providing real-world examples.

Motivation
========

Test writers need useful tools to do their work, and Jasmine, although a very good test framework, lacks many useful tools such as coverage reports, output geared toward automated test runs, etc. Jasmine test automation right now does not have a good detailed reporting mechanism as run through Selenium; it basically passes/fails the entire set of tests with a large, undifferentiated output, making it hard to run useful analysis.

John Papa briefly summarizes the use of Karma quite succinctly here: https://github.com/johnpapa/angular-styleguide#style-y192

The following descriptions will also speak to the motivation behind using Karma.

Description
=========

History

An experimental patch has been created that, when a significant number of AngularJS tests needed to be written, allowed the test writers to very easily test their code coverage, ensuring that all branches were accommodated. Through this experimentation, the value of having a good coverage tool became evident. The patch used to enable this testing is: https://review.openstack.org/#/c/168152/

Reporters

A key feature of Karma is that it can use a set of "reporters" to provide output of the test run. For example, a single test run can produce screen/progress output, jUnit XML output, and coverage reports.

Coverage Reporter (karma-coverage)

Karma accepts a plugin called 'karma-coverage' that provides a test-coverage reporter. This plugin analyzes the test run and determines the coverage of the test run over a given set of source. The output of the report is a directory containing HTML (or other formats, like CSV) describing in character-by-character detail the source coverage (see included screenshots). It reports Statement, Branch, Function, and Line coverage, and summarizes these each at each file and directory level, so it is easy to identify areas that lack testing. This feature was invaluable when developing tests. Jasmine by itself does not provide this feature.

jUnit-formatted reporting

Karma accepts a plugin called 'karma-junit' that provides a junit-output reporter. jUnit, a leading test framework in Java, has established a language-agnostic way of reporting test results, and this is used commonly in various analysis software. This provides useful statistics that can be used as seen fit by a consumer. Jasmine by itself does not produce this output.

Auto-watching

Karma, when run manually, watches any input files (sources, specs, etc.) and immediately reruns test suites when it detects changes. This is very useful to have running in the background when developing. It is a configurable setting.

IDE compatibility/features

Many developers use python IDEs with standard debugging mechanisms. Just using PyCharm as an example: Karma tests can be run from inside PyCharm, but pure Jasmine tests can not be. PyCharm can report on the code coverage of the tests and debug your tests, but only if Karma is used. This will provide a more consistent environment for current Open Stack developers to transition into the Angular world.

Better configuration

Right now, Jasmine tests need to be explicitly configured: all sources, specs, and templates must be individually called out in order to run. This presents opportunities for error, and for tests not to be added or to be accidentally removed. The test runner does not present any errors if explicit files are not found, but continues on blithely. Karma, on the other hand, can use fileglobbing to find files, and is intelligent enough to recognize sources and specs. Configuration is therefore less tedious and error-prone.

Better feedback

In most cases Karma will better report compilation errors in the sources/tests than Jasmine's test runner; Jasmine will typically just not execute tests in this case, rather than fail.

Conclusion

All of these features make Karma a powerful tool that JavaScript developers should have in their toolkit. The coverage features alone have been shown to be critical in writing robust tests; the other features likewise surpass what Jasmine offers out-of-the-box.

Implementation

Karma is best implemented by establishing a package.json at the project root level. In this case, we could do this at the /horizon level. This establishes the Karma and plugin versions and any additional metadata for the project, with regards to npm.

NOTE: due to confusion, some people may believe Karma uses/requires Bower. This is not the case and the patch below does not (any longer) use it.

In terms of Karma configuration it may be easier, due to how dependencies are included, to have two separate karma.conf.js files, one for 'horizon' and one for 'openstack_dashboard.' These two karma configurations can be referenced and called together for the 'npm run test' target established in package.json.

These two files could be condensed into one file, the only issue being that it would be hard to determine how to route certain template requests in some cases due to overlapping website file structures within the horizon and openstack_dashboard components.

run_tests.sh would be modified to accept --karma or -k as a way to run 'npm run test'. It would also run 'npm install' to ensure that the specified versions of Karma and plugins are used.

A 'shim' JS file would need to be used to replicate the Django-served features providing internationalization, since that file does not exist independently within the source tree. This would only be necessary until we are no longer using raw gettext() and related i18n features from Django (until we use a wrapper).

Testing
======

Who tests the test runner? This blueprint speaks entirely to testing, as witnessed above. Without this framework, high test quality would be significantly harder to achieve.

UX
==

This is not a production/user-facing feature.

Outside Dependencies
==================

This relies on npm, which is already used for JavaScript-focused features in the run_test framework (jshint, JSCS). These are NOT production dependencies.

Requirements Update Required
=========================

Does not change features.

Doc Impact
=========

Documentation of the configuration and running of Karma needs to be included in the documentation of client-side (JS) features.

Blueprint information

Status:
Complete
Approver:
David Lyle
Priority:
Medium
Drafter:
Matt Borland
Direction:
Approved
Assignee:
Matt Borland
Definition:
Approved
Series goal:
Accepted for liberty
Implementation:
Implemented
Milestone target:
milestone icon 8.0.0
Started by
David Lyle on 2015-05-22
Completed by
David Lyle on 2015-06-05

Related branches

Sprints

Whiteboard

[david-lyle | 2015-05-22] Moved the meat to the top.

Gerrit topic: https://review.openstack.org/#q,topic:ModuleStructure,n,z

Addressed by: https://review.openstack.org/168152
    Get unit tests working with Karma

Gerrit topic: https://review.openstack.org/#q,topic:bp/karma,n,z

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.