RPM

Finding Useless Buildrequires at build time using URPM ecosystem

Registered by Pirouette Cacahuète

Experimentations around discovering useless buildrequires directives in specs files. First experimenting in Mandriva's URPM context.

Code is available in branches at :
 - http://git.mandriva.com/projects/?p=users/alissy/jurt.git;a=summary for Jurt build tool
 - http://git.mandriva.com/projects/?p=users/alissy/iurt.git;a=summary for Iurt build tool

Some code is available to run a massive rebuild, and grab the logs, then do some plots:
http://git.mandriva.com/projects/?p=users/alissy/jurt-useless-buildrequires.git;a=tree;h=refs/heads/massive_rebuild_v2;hb=refs/heads/massive_rebuild_v2

Overview results on Cooker's python-* SRPMs packages, as of 2011, dec:
$ screen -L sh -c 'for pkg in rpm/SRPMS/python-*; do jurt-build -c $pkg && for fs in `grep jurt /proc/mounts | cut -d" " -f2`; do sudo umount $fs; done && sudo rm -rf /var/spool/jurt/chroots/*/*; done;'

[user@cooker ~]$ cat /home/jurt/builds/logs/*/*/check-useless-buildrequires.log
==== started log at Wed Dec 28 11:09:13 2011
Found the following useless BuildRequires: ncursesw-devel (lib64ncursesw-devel), . Took a total of 77 secs.
build_packages_files: 54.1022000313; prepare_deps: 0.284519910812; solve_provides: 22.7554020882; hit_on_file: 0.0244071483612; prepare_inotify: 0.400624036789;
==== started log at Wed Dec 28 11:10:49 2011
Found the following useless BuildRequires: python-twisted (python-twisted), . Took a total of 29 secs.
build_packages_files: 25.4602980614; prepare_deps: 0.306715011597; solve_provides: 3.60064482689; hit_on_file: 0.00183415412903; prepare_inotify: 0.626179218292;
==== started log at Wed Dec 28 11:42:25 2011
Found the following useless BuildRequires: python-twisted (python-twisted), . Took a total of 30 secs.
build_packages_files: 27.0377569199; prepare_deps: 0.270758867264; solve_provides: 2.38583803177; hit_on_file: 0.00105571746826; prepare_inotify: 0.584053993225;
==== started log at Wed Dec 28 11:48:13 2011
Found the following useless BuildRequires: gnupg (gnupg), . Took a total of 18 secs.
build_packages_files: 15.4470729828; prepare_deps: 0.273365974426; solve_provides: 2.43638300896; hit_on_file: 0.00109314918518; prepare_inotify: 0.60448884964;
==== started log at Wed Dec 28 11:51:29 2011
Found the following useless BuildRequires: gtk-doc (gtk-doc), automake (automake), . Took a total of 47 secs.
build_packages_files: 37.3697249889; prepare_deps: 0.371140003204; solve_provides: 8.60318803787; hit_on_file: 0.0108880996704; prepare_inotify: 0.99089884758;
==== started log at Wed Dec 28 11:55:03 2011
Found the following useless BuildRequires: gtk-doc (gtk-doc), automake (automake), . Took a total of 31 secs.
build_packages_files: 28.2455551624; prepare_deps: 0.351963996887; solve_provides: 3.04018807411; hit_on_file: 0; prepare_inotify: 0.0965831279755;
==== started log at Wed Dec 28 12:04:04 2011
Found the following useless BuildRequires: libGConf2-devel (lib64GConf2-devel), . Took a total of 37 secs.
build_packages_files: 30.5932838917; prepare_deps: 0.298672914505; solve_provides: 6.06628394127; hit_on_file: 0.00474119186401; prepare_inotify: 0.799780130386;
==== started log at Wed Dec 28 12:09:31 2011
Found the following useless BuildRequires: png-devel (lib64png0-devel, lib64png-devel), . Took a total of 78 secs.
build_packages_files: 65.0466279984; prepare_deps: 0.531357765198; solve_provides: 12.2173159122; hit_on_file: 0.00566053390503; prepare_inotify: 0.943692922592;
==== started log at Wed Dec 28 12:59:32 2011
Found the following useless BuildRequires: ipython (ipython), python-configobj (python-configobj), libwxPythonGTK-devel (wxPythonGTK-devel), . Took a total of 101 secs.
build_packages_files: 61.7893910408; prepare_deps: 1.14847803116; solve_provides: 24.8998289108; hit_on_file: 0.0489292144775; prepare_inotify: 13.7504940033;
==== started log at Wed Dec 28 13:17:06 2011
Found the following useless BuildRequires: tcl (tcl), tk-devel (lib64tk-devel), mesaglut-devel (lib64mesaglut3-devel), swig (swig), tk (tk), . Took a total of 45 secs.
build_packages_files: 33.9580700397; prepare_deps: 0.316027879715; solve_provides: 9.31832289696; hit_on_file: 0.00342464447021; prepare_inotify: 1.45233917236;
==== started log at Wed Dec 28 13:42:38 2011
Found the following useless BuildRequires: tetex-latex (texlive-collection-latexextra), tetex-dvipdfm (texlive-dvipdfm), . Took a total of 38 secs.
build_packages_files: 32.3348870277; prepare_deps: 0.273763179779; solve_provides: 4.80337095261; hit_on_file: 0.00590991973877; prepare_inotify: 0.663867950439;
==== started log at Wed Dec 28 13:44:45 2011
Found the following useless BuildRequires: libpython-devel (lib64python-devel), . Took a total of 14 secs.
build_packages_files: 12.4180939198; prepare_deps: 0.254108190536; solve_provides: 1.1321451664; hit_on_file: 0; prepare_inotify: 0.563099145889;
==== started log at Wed Dec 28 13:47:36 2011
Found the following useless BuildRequires: python-dbus (python-dbus), dbus-glib-devel (lib64dbus-glib-1-devel), gtk+2-devel (lib64gtk+2.0-devel), . Took a total of 27 secs.
build_packages_files: 20.3509891033; prepare_deps: 0.271450042725; solve_provides: 5.85151481628; hit_on_file: 0.00205206871033; prepare_inotify: 0.866218090057;
==== started log at Wed Dec 28 14:09:52 2011
Found the following useless BuildRequires: python-simplejson (python-simplejson), . Took a total of 20 secs.
build_packages_files: 16.4130871296; prepare_deps: 0.278006076813; solve_provides: 3.66475605965; hit_on_file: 0.00200033187866; prepare_inotify: 0.613239049911;
==== started log at Wed Dec 28 14:13:03 2011
Found the following useless BuildRequires: python-zope-interface (python-zope-interface), . Took a total of 16 secs.
build_packages_files: 13.456182003; prepare_deps: 0.326878786087; solve_provides: 2.36840796471; hit_on_file: 0.00270390510559; prepare_inotify: 0.614902019501;

$ echo "package build_packages_files prepare_deps solve_provides hit_on_file prepare_inotify"; for f in /home/jurt/builds/logs/*/*/check-useless-buildrequires.log; do PKG=`echo $f | cut -d'/' -f7`; VALUES=`grep build_packages_files $f | cut -d' ' -f2,4,6,8,10 | sed -e 's/;//g'`; echo "$PKG $VALUES"; done;package build_packages_files prepare_deps solve_provides hit_on_file prepare_inotify
python-2.7.2-3 54.1022000313 0.284519910812 22.7554020882 0.0244071483612 0.400624036789
python-axiom-0.6.0-2 25.4602980614 0.306715011597 3.60064482689 0.00183415412903 0.626179218292
python-epsilon-0.6.0-2 27.0377569199 0.270758867264 2.38583803177 0.00105571746826 0.584053993225
python-GnuPG-Interface-0.3.2-10 15.4470729828 0.273365974426 2.43638300896 0.00109314918518 0.60448884964
python-gobject-2.28.6-2 37.3697249889 0.371140003204 8.60318803787 0.0108880996704 0.99089884758
python-gobject3-3.0.2-1 28.2455551624 0.351963996887 3.04018807411 0 0.0965831279755
python-gtksourceview-2.10.1-4 30.5932838917 0.298672914505 6.06628394127 0.00474119186401 0.799780130386
python-imaging-1.1.7-5 65.0466279984 0.531357765198 12.2173159122 0.00566053390503 0.943692922592
python-matplotlib-1.1.0-1 61.7893910408 1.14847803116 24.8998289108 0.0489292144775 13.7504940033
python-opengl-3.0.1-2 33.9580700397 0.316027879715 9.31832289696 0.00342464447021 1.45233917236
python-OpenSSL-0.12-1 32.3348870277 0.273763179779 4.80337095261 0.00590991973877 0.663867950439
python-psyco-1.6-7 12.4180939198 0.254108190536 1.1321451664 0 0.563099145889
python-pyatspi-0.4.1-2 20.3509891033 0.271450042725 5.85151481628 0.00205206871033 0.866218090057
python-translate-1.9.0-1 16.4130871296 0.278006076813 3.66475605965 0.00200033187866 0.613239049911
python-twisted-core-10.1.0-3 13.456182003 0.326878786087 2.36840796471 0.00270390510559 0.614902019501

Can be transformed to the following gnuplot data file:
# package build_packages_files prepare_deps solve_provides hit_on_file prepare_inotify
python-2.7.2-3 54.1022000313 0.284519910812 22.7554020882 0.0244071483612 0.400624036789
python-axiom-0.6.0-2 25.4602980614 0.306715011597 3.60064482689 0.00183415412903 0.626179218292
python-epsilon-0.6.0-2 27.0377569199 0.270758867264 2.38583803177 0.00105571746826 0.584053993225
python-GnuPG-Interface-0.3.2-10 15.4470729828 0.273365974426 2.43638300896 0.00109314918518 0.60448884964
python-gobject-2.28.6-2 37.3697249889 0.371140003204 8.60318803787 0.0108880996704 0.99089884758
python-gobject3-3.0.2-1 28.2455551624 0.351963996887 3.04018807411 0 0.0965831279755
python-gtksourceview-2.10.1-4 30.5932838917 0.298672914505 6.06628394127 0.00474119186401 0.799780130386
python-imaging-1.1.7-5 65.0466279984 0.531357765198 12.2173159122 0.00566053390503 0.943692922592
python-matplotlib-1.1.0-1 61.7893910408 1.14847803116 24.8998289108 0.0489292144775 13.7504940033
python-opengl-3.0.1-2 33.9580700397 0.316027879715 9.31832289696 0.00342464447021 1.45233917236
python-OpenSSL-0.12-1 32.3348870277 0.273763179779 4.80337095261 0.00590991973877 0.663867950439
python-psyco-1.6-7 12.4180939198 0.254108190536 1.1321451664 0 0.563099145889
python-pyatspi-0.4.1-2 20.3509891033 0.271450042725 5.85151481628 0.00205206871033 0.866218090057
python-translate-1.9.0-1 16.4130871296 0.278006076813 3.66475605965 0.00200033187866 0.613239049911
python-twisted-core-10.1.0-3 13.456182003 0.326878786087 2.36840796471 0.00270390510559 0.614902019501

Which then can be plotted with:
set title "Useless BuildRequires verification additional build time"
set term png size 1920,1080 enhanced
set out "buildrequires.png"

set ylabel "Time (secs)"
set y2label "Total time (secs)"
set xtics out rotate by -90
set y2tics auto
set logscale y
set grid

plot "buildrequires.txt" using 2:xtic(1) axes x1y1 with linespoints title "build-packages-files", \
        "buildrequires.txt" using 3:xtic(1) axes x1y1 with linespoints title "prepare-deps", \
        "buildrequires.txt" using 4:xtic(1) axes x1y1 with linespoints title "solve-provides", \
        "buildrequires.txt" using 5:xtic(1) axes x1y1 with linespoints title "hit-on-file", \
        "buildrequires.txt" using 6:xtic(1) axes x1y1 with linespoints title "prepare-inotify", \
        "buildrequires.txt" using ($2+$3+$4+$5+$6) axes x1y2 with linespoints title "Total"

Blueprint information

Status:
Started
Approver:
Jeff Johnson
Priority:
Low
Drafter:
Pirouette Cacahuète
Direction:
Approved
Assignee:
Pirouette Cacahuète
Definition:
Pending Approval
Series goal:
Accepted for 5.4
Implementation:
Needs Code Review
Milestone target:
None
Started by
Pirouette Cacahuète

Related branches

Sprints

Whiteboard

Whiteboard cleared by moving text into the summary.

I'd like to see this functionality in RPM itself: there are many usage
cases for notify(7), not just in detecting unused BuilRequires:

The methods needed aren't hard, and an rpm "object" with
reference counts and mutex's is fairly simple wrapping, and
arranging for bindings onto a C object would permit other
more efficient (than python dicts and perl ties) to be used
for what are essentially set membership operations of
file paths and dependency assertions. (I'm thinking of
Yet Another Use for Bloom filters here).

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

Nearby

This blueprint contains Public information 
Everyone can see this information.
Edit subscription

Subscribers

No subscribers.