big.LITTLE MP hotplug stress/performance tests

[asac Sep 10, 2012]: lacks headline, acceptance etc. might need some input from plars what is really wanted here and who is the driver.
[dzin, Oct 10, 2012] Blocked on https://blueprints.launchpad.net/linaro-android/+spec/cpu-hot-plug-latency which is in turn blocked.
[dzin, Nov 24, 2012] cpu-hot-plug-latency is no longer blocked, target for 12.12
[nkambo, nov 27, 2012] related bug has been reported and linked in this blueprint
https://bugs.launchpad.net/linaro-landing-team-arm/+bug/1068595
[dzin, Dec 20, 2012] Bug resolved but unreleased. To be moved to 13.01
[nkambo, Jan 21, 2012]
On Thu, 2013-01-03 at 12:10 -0500, David Zinman wrote:
> Does this issue need to be followed up?

Well, function tracing is still broken.

When I tried upstreaming my fix the ensuing discussion showed that my
understanding what I thought was going on was incomplete and the reason
for my fix appeared to work couldn't be what I thought it was. And the
discussion also seemed to conclude that the ftrace on ARM is doing
things that according to the ARM architecture specification isn't
defined.

Also, this week, when I tried adding the 'notrace' annotations Vincent
identified and Nicolas expanded on, then enabling ftrace still crashes
if cpuidle is running even when avoiding Thumb2 instructions.]

[nkambo Jan 25 2013]
<email address hidden> wrote:
> Hi Tixy/Nicolas/Vireshk/Amitk,
>
> I have tested attached cpu_hotplug_latency.sh file on android successful.
> test execution required following kernel configurations changes.
> --->vexpress_bL_mp_defconfig is the base kernel config
> --->Cpu idle has been disabled.
> --->Ftrace has been Enabled
> --->Ftrace Dynamic has been enabled
> --->Ftrace Graph has been enabled
>
> We do not have above config as default so i have modified and executed test.
> Do you have plan to make these config as default to
> vexpress_bL_mp_defconfig? Which would be helps me in testing release build
> rather than local builds.

[amitk] wrote:
Daniel is currently investigating a bug with hotplug/RCU in mainline
(https://bugs.launchpad.net/linaro-power-kernel/+bug/1102347).

We shouldn't mark this as done until we are able to reliably hotplug
_with_ cpuidle enabled.

[nkambo Jan 31 2013]
Attached bugs needs to be fixed to move forward on this blueprint.

[lorenzo-pieralisi Feb 12 2013] what's the status with attached bugs ? can the work items breakdown for 13.01 and 13.02 be made available ?

[nkambo, Feb 14 2013] the linked bugs still under investigation. I could not break work items available for 13.01 and 13.02.
Test case works if i modify kernel config options.
 --->Cpu idle has been disabled.
 --->Ftrace has been Enabled.
Which is not the appropriate way to test kernels. until these options become default in Vexpress_bL_mp_defconfig we can not close this blue print.

[nkambo, March 13 2013] the linked bugs still under investigation. this blue will be addressed or unblocked once after the above bugs fixed.

[asac, April 3 2013] bugs still blocked delivery; big.LITTLE PM should pay attention; moving over to 2013.04

[nkambo, April 12 2013] bugs linked on this blueprints have been updated by me with latest test information and kernel crash log.

Enabling function tracer kills the system on TC2
    https://bugs.launchpad.net/linaro-landing-team-arm/+bug/1068595
missing trace on function "cpu_hotplug_done"
    https://bugs.launchpad.net/linaro-big-little-system/+bug/1087149
exynos : rcu stall with 3.8-rc2
    https://bugs.launchpad.net/linaro-power-kernel/+bug/1102347

[nkambo, May 3 2013] All the attached bugs are fixed now, it is time to unblock the blueprint.
I have to work on test case verdict. which will be addressed in 13.05 cycle.

[2013/5/08 nkambo] Function tracer creating trace.dat file contains huge trace data. which can be decoded by using kernelshark or powertop. Currently I am exploring the options to read it on target and add pass/fail verdict.

[2013/05/20 nkambo]
vincent wrote:
Regarding the cpu hotplug tests, we need to define several latency
thresholds based on current measurements (have you got some results ?)
and target value:
-a low latency threshold which should be reached in idle system (and
at the end all the time) otherwise the test is failed. I have in mind
30ms (for tc2)
-an acceptable latency value which should never be reached even in a
loaded system otherwise the loaded test are failed: i have in mind
500ms (for tc2)

We also need to check the load that must be generated by the script
during the tests.

Vincent comments needs to be addressed.

[2013/05/20 nkambo]
Pass/fail verdict created.
python script from host machine will use test_load.txt as input and parse the results and prints pass fail.
Test integration on to lava will be handled in below BP:
https://blueprints.launchpad.net/linaro-big-little-system/+spec/integrate-cpu-hotplug-latency-tests-lava

See also: https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/Hotplug#CPU_hotplug_on_latest_kernel

Status:
Android team is working on porting cyclictest for 12.09 which is needed for the performance part of this testsuite.

Meta:
Roadmap id: CARD-196
Headline: CPU hot-plug latency test should be integrated to Android and Ubuntu builds.
Acceptance: CPU hot-plug latency test should run and integrate it in LAVA test.

Work items:
[pwlars] Create a stress test for hotplug cpu: DONE
Test new proposed test case on Android: DONE
Investigate the kernel config support for this test: DONE
Add pass/fail verdict on to test case : DONE
Investigate performance measurement scripts on wiki link above: DONE