Software codec optimization

Registered by Ilias Biris

Captures the requirements and dependencies for SW codec optimization, related to Multimedia WG

Blueprint information

Not started
Kurt Taylor
Kurt Taylor
Needs approval
Kan HU
Series goal:
Milestone target:

Related branches



Proposal and action points as registered in Multimedia mini summit (June 2011) -

= Optimization Forum Proposal =
* Wikipage to hold existing optimized software implementations (codecs, encryption, checksumming)
   * Also capture requests for software-optimizing
   * How to message it? NEON-optimizing usefully constrains optimization type; maybe ARM optimizing; take into account that OpenCL is another optimization avenue
   * Aggregate per domain? For instance, Codecs/formats available, and then noting what is optimized
      * Potentially grade optimization level of codecs
* Mailing list to hold conversation around site-specific optimization
* Potentially hold a day or set of sessions in Cambridge August sprint
* Look at libjpeg-turbo and the optimizations that can be done there
  * Mans, Tom, Mandeep to work with Darrell on -turbo

* ACTION: Study how Android does hardware capability selection to dynamically load NEON vs non-NEON components?
* ACTION: Look at OpenSSL on ARM; evaluate NEON optimization potential
* ACTION: Look at libjpeg Android loading: how is support for hardware codecs handled?
* ACTION: Investigate how libjpeg-turbo could cater for handling hardware decoding
   * Switching would need to be dynamic because for smaller images the latency overhead could negate the benefit
   * How would the hardware codec be exposed to libjpeg-turbo
   * What about decoding hundreds of images in a gallery
* ACTION: Investigate OpenCV implementation -- how much of it is vectorizable; what has been SSE'd; how hard will it be to add NEON routines (Mandeep: some Freescale work done)
* ACTION: Do a broad sweep of Ubuntu packages that are using SSE and missing NEON for candidates for work
   * pixman, skia probably already have NEON code
   * transcoding for MMS might be another use case
* ACTION: Look at use of composition/blending and color conversion in SurfaceFlinger; potential for NEON work or handled by the GPU already?
* ACTION: Investigate which upstream projects could accept NEON optimizations
* ACTION: NEON: study if we need to look at anything beyond what we have checked so far (AAC, VP8, JPEG)?
  * Skype SILK - is there a NEON version for that? (No, interesting to optimized by neon, low priority)
  * Low latency voice codecs (which ones are interesting to look at)? (Speex neon optimization, low priority)
  * MP3 - do we want to do the effort to optimize it (not low hanging fruit, and not big improvement expected).


Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.