Implement and upstream ARM 32 bit uprobes support

Registered by Deepak Saxena

Implement and upstream uprobes for ARM

Blueprint information

Jakub Pavelek
Deepak Saxena
David Long
Series goal:
Accepted for kernel-merge-window
Informational Informational
Milestone target:
milestone icon 3.13
Started by
David Long
Completed by
Jakub Pavelek

Related branches



Roadmap id: CARD-153
Headline: ARM uprobes support has been upstreamed
   * linux-linaro Ubuntu Engineering build and build spec have been updated to include kprobes and uprobes support (Naresh/Jakub can help)
   * Kprobes and Uprobes can be used in unison to trace system behaviour across the software stack (TODO - how does one prove that? Not a criterion, a requirement)
   * uprobes support for ARM is merged into upstream kernel
   * GIT commit link to kernel tree has been logged here
   * uprobes support for ARM is merged into systemtap sources
   * GIT commit link to systemtap tree has been logged here
   * linux master build also execute the tests developed in this card and succeeds (to prove that the code merged upstream still passes tests) (Andrey Konovalov/Jakub can help)

[jakub-pavelek 2013-03-28] moving from 3.10 to 3.11 as current upstreaming target.
[dave-long 2013-04-30] After discussion with tixy it appears the amount of redundant code in the current approach is likely to be an issue upstream. We came up with a modified approach which I am working on now.
[jakub pavelek 2013-06-17] Moving from 3.11 to 3.12 with a bit of optimism
[jakub-pavelek 2013-06-26] Naresh got system tap test suite running - Dave please contact Naresh and get the same test suite deployed locally for your testing. Naresh will deploy the test suite to LAVA (we will make a follow-up Blueprint for him) and will run the test suite there - we want to see it passing in our CI.
[frank-eigler 2013-06-26 Please involve <email address hidden> if any systemtap-side porting work is found to be required.]

[nkambo 2013-09-04] Systemtap install on snowball and run the a simple test. Here is the log which gives complete information about
- Build kernel with following config
     - kernel config file:
- Must install kernel headers and modules on the target
     - systemtap looks for .config and file under /lib/module/'uname -r'/
- Download systemtap-2.2.1 source
- Download elfutils source
- Download dependencies build-essential aptitude m4 dejagnu
    - apt-get install build-essential aptitude m4 dejagnu
    - cd systemtap-2.2.1
    - cp /usr/share/dejagnu/* . -a
- Configure systemtap
     - ./configure --prefix=/usr/local/ --with-elfutils=../elfutils-0.155
- Build and Install systemtap-2.2.1
     - make all; make install;
- systemtap example program
- # stap -ve 'probe begin { log("hello world") exit() }'
- Run " make installcheck"
- Complete systemtap log can be found at:

 === systemtap Summary ===

# of expected passes 770
# of unexpected failures 466
# of unexpected successes 1
# of expected failures 281
# of known failures 20
# of untested testcases 211
# of unsupported tests 5
runtest completed at Wed Sep 4 15:21:35 2013

- Observations While running this test suite on target, kill command has issues due to this reason, test process created are not killed and the inserted modules not removed from the kernel space due to this reason kernel reported "Out of memory: " many times.
- The tests keeps on adding new modules to kernel space and finally kernel space was full and stalled. I have manually killed many test processes and removed kernel modules to free the memory to continue test execution.
    - "Out of memory:" dmesg log:

- After the test execution test results will be available in file : systemtap.sum
    - file printout :
- Test results notations
    - PASS
    - XFAIL
    - FAIL
    - KFAIL
    - XFAIL

[jakub-pavelek 2013-09-05] Bumping from 3.12 to 3.13 - the merge window is open, do not see this being pushed in. Also we do not have Thumb implemented yet and the testing is just starting. I see many failures in the testing - do we need some SystemTap work do be done to get this working? Frank has been offering help couple of months ago - please consider keeping him in the loop on testing.

[William Cohen <email address hidden> 2013-09-05]
I have been working on making sure that SystemTap functions reasonably well on ARM. Frank Eigler mentioned that you are working on uprobes for ARM and pointed me at . I would be happy to help get ARM uprobes systemtap support working.

I can't see the details on the pages, but the number of failure looks unusually high and the number of tests run looks low on

 === systemtap Summary ===

# of expected passes 770
# of unexpected failures 466
# of unexpected successes 1
# of expected failures 281
# of known failures 20
# of untested testcases 211
# of unsupported tests 5
runtest completed at Wed Sep 4 15:21:35 2013

One thing thing that might prevent the modules from unloading is not having the following in the kernel config:


For comparison I have a samsung arm chromebook set up with Fedora 19 and a locally built linux-3.11 kernel. systemtap from a git repo checkout yielded the following results:

                === systemtap Summary ===

# of expected passes 1815
# of unexpected failures 89
# of expected failures 283
# of known failures 10
# of untested testcases 364
# of unsupported tests 10

The details on chromebook's locally built kernel are on


[William Cohen <email address hidden> 2013-09-06]
There has been some work at to include uprobes in the arm kernel:

I built a local version of a the uprobes branch kernel with uprobes enabled from:;a=summary

The tests took several hours on a samsung arm chromebook
                === systemtap Summary ===

# of expected passes 3011
# of unexpected failures 168
# of expected failures 283
# of known failures 40
# of untested testcases 179
# of unsupported tests 4

The complete results are posted on dejazilla:

I didn't see any regressions in a comparison between a stock linux-3.11 vs. the linux-3.11-rc5 with uprobes on the same arm machine:


[nkambo 2013-09-11]
latest systemtap results on snowball

  === systemtap Summary ===

# of expected passes 2034
# of unexpected failures 425
# of unexpected successes 1
# of expected failures 281
# of known failures 27
# of untested testcases 152
# of unsupported tests 6

[Will 2013-09-17]
On 09/10/2013 04:08 PM, William Cohen wrote:
> Last week Frank mentioned there were some patches for uprobes on ARM ( I have taken a look at the patches and built a local kernel for my Samsung Chromebook that included those patches. I thought this is cool! These allow probing user-space applications on ARM. But then the question was "What would be an interesting user-space application to probe?" Then I thought about another new feature that is available on ARM, kvm hardware acceleration for virtualization. With some more tweaking of the 3.11 kernel I was able to get a kernel that supports KVM on arm. I also needed very new versions of elfutils, qemu, and systemtap rpms installed on the machine.
> With the host set up I was able to set up a guest Fedora 18 vm based on the instructions at With this guest vm running I can run the qemu_count.stp example. Things are not perfect (some args locations can't be parsed), but at least to the point of getting some interesting information on ARM:

A bit more playing around with hardware virtuatlization acceleration on arm. I got a stock fedora 19 image guest image set up on the chromebook. The kvm support is working well enough to build systemtap and test systemtap within the guest. The results look pretty respectable. This is the stock fedora arm 3.9.5-301.fc19.armv7hl kernel that doesn't support uprobes and a older elfutils that may not find argument in some cases (

                === systemtap Summary ===

# of expected passes 1744
# of unexpected failures 85
# of expected failures 280
# of known failures 10
# of untested testcases 367
# of unsupported tests 12

More detailed breakdown of the results are at:


[nkambo 2013-Sept-18]

I built a local version of a the uprobes branch kernel with uprobes enabled from:;a=summary

Systemtap translator/driver (version 2.2.1/0.155, non-git sources)
Thread model: posix
gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1)
Linux linaro-developer 3.11.0-rc5 #1 SMP PREEMPT Tue Sep 17 18:46:23 UTC 2013 ar
mv7l armv7l armv7l GNU/Linux

Results logs:
- systemtap.log
- systemtap.sum
- stap-report-sept-18-2013

snowball target systemtap test results summary
               === systemtap Summary ===

# of expected passes 2089
# of unexpected failures 381
# of unexpected successes 1
# of expected failures 281
# of known failures 32
# of untested testcases 151
# of unsupported tests 6

[dave-long 2013-Sept-18] I've created another repo at:


...which contains a "uprobes" branch with my uprobes patches. This repo is based on linaro-stable-kernel, instead of upstream. This may be more useful for Linaro testing.

Note, the following are the config differences from the default config to enable uprobes/kprobes testing:


[nkambo Oct-2-2013]
Latest systemtap (2.4) and elfutils (0.157) test execution results on snowball.

kernel version: 3.11.0-rc5
systemtap version: version 2.4/0.157

  === systemtap Summary ===

# of expected passes 2309
# of unexpected failures 390
# of expected failures 280
# of known failures 41
# of untested testcases 195
# of unsupported tests 7

[jakub-pavelek 2013-10-03] Naresh/others - how should the test results look like in order to day "uprobes32 are tested and passing as expected". I'm not familiar with what issues are part of systemtap and which part of the implementation we did. Someone please spell it out here so that we have a target to go after please.

Another issue - I will be moving this blueprint from Launchpad to JIRA - as this is the last artifact that kernel working group has left behind on Launchpad. I won't be able to subscribe all the people subscribed here unfortunately. Will keep you posted.

[jakub-pavelek 20013-10-30] Moving active tracking over to Engineering card in JIRA at and its sub-tasks. This should be public info accessible to anyone. Subscription and active participation possible but requires JIRA login.


Work Items

Work items for 13.02:
Contact patch originator (Feb 22) : DONE
Build and test patch on 3.6 using originators test suite (Feb 25): DONE

Work items for 13.04:
Review existing uprobes patches (April 3) : DONE
Uplevel and test existing patches (April 3): DONE
Extract common uprobes and kprobes code into shared functions (April 29): DONE
Separate uprobe and kprobe parsing data and function into their own files (April 29): DONE
Internal review (new approach discussed) (4/30): DONE

Work items for 13.06:
Design and implement vector table for parser output actions (6/12): DONE
Recreate patch set for public review (June 19): DONE
Get QEMU enviroment set up and working (6/4): DONE
Uplevel work to v3.10-rc3 (6/6): DONE

Work items for 13.08:
Fix regressions (August 1): DONE
Ask Naresh to help setup systemtap locally for own testing the same way it will run in our continuous integration: DONE

Work items for 13.09:
Add thumb support for uprobes: TODO
Upstream uprobes patches to 3.12: TODO
After patchset review, ping Andrey Konovalov and Naresh to take your topic branch and setup testing in our CI - we will need that one working as acceptance criterion: TODO
Review thumb-supporting patchset: TODO
Upstream thumb-supporting patchset: TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.