Merge into db-devel : support-timeframe-information : Code : Launchpad itself

Reviewer	Review Type	Date Requested	Status
Michael Nelson (community)	code	2010-03-25	Approve on 2010-04-14
Michael Vogt (community)			Needs Resubmitting on 2010-04-13
William Grant	code	2010-04-01	Approve on 2010-04-01
Review via email: mp+22118@code.launchpad.net

Revision history for this message

Michael Vogt (mvo) wrote on 2010-03-25:

#

If it can be run on dogfood first to ensure it works properly in the LP environment, that would be most appreciated.

Revision history for this message

Aaron Bentley (abentley) wrote on 2010-03-29:

#

The description does not give me enough context to do a review.

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-01:

#

Download full text (9.8 KiB)

Hi Michael!

Thanks for getting in there and improving Launchpad code!

As Aaron suggested, it would be really helpful to include more info with your merge proposal. A template we often try to use is:

https://dev.launchpad.net/QuickCoverLetterTemplate

or if you want lots of details ;) :

https://dev.launchpad.net/DetailedCoverLetterTemplate

Especially important for branches like this without tests is a QA plan for how we can verify on dogfood that this does exactly what you intend. Tests would be wonderful, but I know that it's fair to ask you to add them now (If you want an example of how other cron scripts are tested, see lib/lp/archivepublisher/tests/test_generate_ppa_htaccess.py)

OK, I've got a bunch of small changes below, but more important for me is the summary of what this branch does (after reading it I've now understood most of it, but it would still be helpful to clarify) and steps for us to QA this on dogfood.

As I don't have lots of experience with germination etc. (I'm still learning ;) ), and Julian is away, I'll ask William to take a look to and provide feedback.

Thanks!

> === modified file 'cronscripts/publishing/maintenance-check.py'
> --- cronscripts/publishing/maintenance-check.py 2010-01-22 13:57:45 +0000
> +++ cronscripts/publishing/maintenance-check.py 2010-03-25 11:57:28 +0000
> @@ -59,9 +63,57 @@
> # germinate output base directory
> BASE_URL = "http://people.canonical.com/~ubuntu-archive/germinate-output/"
>
> +# hints dir url, hints file is "$distro.hints" by default
> +# (e.g. lucid.hints)
> +#HINTS_DIR_URL = "http://people.canonical.com/~mvo/maintenance-check/%s.hints"

Is there a reason for leaving the above comment eg. in?

> +HINTS_DIR_URL = "http://people.canonical.com/~ubuntu-archive/seeds/platform.%s/SUPPORTED_HINTS"
> +
> +# we need the archive root to parse the Sources file to support
> +# by-source hints
> +#ARCHIVE_ROOT = "file:/srv/launchpad.net/ubuntu-archive/ubuntu"

Same above?

> +ARCHIVE_ROOT = "http://archive.ubuntu.com/ubuntu"
> +
> # support timeframe tag used in the Packages file
> SUPPORT_TAG = "Supported"
>
> +def get_binaries_for_source_pkg(srcname):
> + """
> + get all binary package names for the given source package name
> + """

Just a few style comments (based on the Launchpad python style guide at https://dev.launchpad.net/PythonStyleGuide). I realise the rest of this file
doesn't follow the style-guide, but it'd be nice to clean it up bit-by-bit
(but not necessary, up to you).

So the above docstring could be formatted along the lines of:
""" Return all binary names for the given source name."""

If you are really keen, you could add the :param: and :return: types as outlined under "Docstrings" in the above styleguide, but it's not necessary.

> + pkgnames = set()
> + recs = apt_pkg.GetPkgSrcRecords()
> + while recs.Lookup(srcname):
> + for binary in recs.Binaries:
> + pkgnames.add(binary)
> + return pkgnames
> +
> +def expand_src_pkgname(pkgname):
> + """ expand a given pkgname if prefixed with src: to a list of
> + binary package names, if not prefixed, just return the pkgname
> + """

""" Expand a package...

Hi Michael!

Thanks for getting in there and improving Launchpad code!

As Aaron suggested, it would be really helpful to include more info with your merge proposal. A template we often try to use is:

https://dev.launchpad.net/QuickCoverLetterTemplate

or if you want lots of details ;) :

https://dev.launchpad.net/DetailedCoverLetterTemplate

Especially important for branches like this without tests is a QA plan for how we can verify on dogfood that this does exactly what you intend. Tests would be wonderful, but I know that it's fair to ask you to add them now (If you want an example of how other cron scripts are tested, see lib/lp/archivepublisher/tests/test_generate_ppa_htaccess.py)

OK, I've got a bunch of small changes below, but more important for me is the summary of what this branch does (after reading it I've now understood most of it, but it would still be helpful to clarify) and steps for us to QA this on dogfood.

As I don't have lots of experience with germination etc. (I'm still learning ;) ), and Julian is away, I'll ask William to take a look to and provide feedback.

Thanks!

> === modified file 'cronscripts/publishing/maintenance-check.py'
> --- cronscripts/publishing/maintenance-check.py 2010-01-22 13:57:45 +0000
> +++ cronscripts/publishing/maintenance-check.py 2010-03-25 11:57:28 +0000
> @@ -59,9 +63,57 @@
>  # germinate output base directory
>  BASE_URL = "http://people.canonical.com/~ubuntu-archive/germinate-output/"
>
> +# hints dir url, hints file is "$distro.hints" by default
> +# (e.g. lucid.hints)
> +#HINTS_DIR_URL = "http://people.canonical.com/~mvo/maintenance-check/%s.hints"

Is there a reason for leaving the above comment eg. in?

> +HINTS_DIR_URL = "http://people.canonical.com/~ubuntu-archive/seeds/platform.%s/SUPPORTED_HINTS"
> +
> +# we need the archive root to parse the Sources file to support
> +# by-source hints
> +#ARCHIVE_ROOT = "file:/srv/launchpad.net/ubuntu-archive/ubuntu"

Same above?

> +ARCHIVE_ROOT = "http://archive.ubuntu.com/ubuntu"
> +
>  # support timeframe tag used in the Packages file
>  SUPPORT_TAG = "Supported"
>
> +def get_binaries_for_source_pkg(srcname):
> +    """
> +    get all binary package names for the given source package name
> +    """

Just a few style comments (based on the Launchpad python style guide at https://dev.launchpad.net/PythonStyleGuide). I realise the rest of this file
doesn't follow the style-guide, but it'd be nice to clean it up bit-by-bit
(but not necessary, up to you).

So the above docstring could be formatted along the lines of:
       """ Return all binary names for the given source name."""

If you are really keen, you could add the :param: and :return: types as outlined under "Docstrings" in the above styleguide, but it's not necessary.

> +    pkgnames = set()
> +    recs = apt_pkg.GetPkgSrcRecords()
> +    while recs.Lookup(srcname):
> +        for binary in recs.Binaries:
> +            pkgnames.add(binary)
> +    return pkgnames
> +
> +def expand_src_pkgname(pkgname):
> +    """ expand a given pkgname if prefixed with src: to a list of
> +        binary package names, if not prefixed, just return the pkgname
> +    """

""" Expand a package name if it is prefixed with src.

If the package name is prefixed with src it will be expanded
       to a list of binary package names. Otherwise the original
       package name will be returned.
       """

> +    if not pkgname.startswith("src:"):
> +        return [pkgname]
> +    return get_binaries_for_source_pkg(pkgname.split("src:")[1])
> +
> +def create_and_update_deb_src_source_list(distro):
> +    """
> +    create sources.list with deb-src entries for a given distro release
> +    and update to make sure we are current
> +    """

Similar docstring formatting here (and I think we use distroseries instead of distro release now?)

More importantly, without an explanation, I don't understand why this function is being added to update the src sources list... I assume it's to do with the hints, but I'm not sure how.

> +    # apt root dir
> +    rootdir="./aptroot.%s" % distro
> +    sources_list_dir = os.path.join(rootdir, "etc","apt")
> +    if not os.path.exists(sources_list_dir):
> +        os.makedirs(sources_list_dir)
> +    sources_list = open(os.path.join(sources_list_dir, "sources.list"),"w")
> +    for pocket in ["%s" % distro,
> +                   "%s-updates" % distro,
> +                   "%s-security" % distro]:

Small style-guide differences:

for pocket in [
        "%s" % distro,
        "%s-updates" % distro,
        "%s-security" % distro,
        ]:
(or if you prefer, just define the list first.

> +        sources_list.write("deb-src %s %s main restricted\n" % (
> +                ARCHIVE_ROOT, pocket))

sources_list.write(
            "deb-src %s %s main restricted\n" % (
                ARCHIVE_ROOT, pocket))

> +    sources_list.close()
> +    cache = apt.Cache(rootdir=rootdir)
> +    cache.update(apt.progress.FetchProgress())
>
>  def get_structure(name, version):
>     """
> @@ -150,6 +202,8 @@
>     parser.add_option("--source-packages", "", default=False,
>                       action="store_true",
>                       help="show as source pkgs")
> +    parser.add_option("--hints-file", "", default=None,
> +                      help="use diffenrt use hints file location")
>     (options, args) = parser.parse_args()
>
>     # init
> @@ -160,6 +214,17 @@
>             sys.exit(1)
>     else:
>         distro = "lucid"
> +
> +    # make sure our deb-src information is up-to-date

So this is related to my comment above. Why is this necessary now when it wasn't before. It's probably obvious to you, but not to me :)

> +    create_and_update_deb_src_source_list(distro)
> +
> +    if options.hints_file:
> +        hints_file = options.hints_file
> +        (schema, netloc, path, query, fragment) = urlparse.urlsplit(hints_file)
> +        if not schema:
> +            hints_file = "file:%s" % path
> +    else:
> +        hints_file = HINTS_DIR_URL % distro
>
>     # go over the distros we need to check
>     pkg_support_time = {}
> @@ -175,20 +240,57 @@
>         else:
>             support_timeframe = SUPPORT_TIMEFRAME
>         get_packages_support_time(structure, name, pkg_support_time, support_timeframe)
> +
> +    # now check the hints file that is used to overwrite
> +    # the default seeds
> +    try:
> +        for line in urllib2.urlopen(hints_file):
> +            line = line.strip()
> +            if not line or line.startswith("#"):
> +                continue
> +            try:
> +                (raw_pkgname, support_time) = line.split()
> +                for pkgname in expand_src_pkgname(raw_pkgname):
> +                    if support_time == 'unsupported':
> +                        try:
> +                            del pkg_support_time[pkgname]
> +                            sys.stderr.write("hints-file: marking %s unsupported\n" % pkgname)
> +                        except KeyError:
> +                            pass
> +                    else:
> +                        if pkg_support_time.get(pkgname) != support_time:
> +                            sys.stderr.write("hints-file: changing %s from %s to %s\n" % (pkgname,  pkg_support_time.get(pkgname), support_time))

Lint should tell you this is too long. Actually, there's lots of lint here to clean up if you've time:

http://pastebin.ubuntu.com/407510/

> +                            pkg_support_time[pkgname] = support_time
> +            except:
> +                logging.exception("can not parts line '%s'" % line)

s/parts/parse?

> +    except urllib2.HTTPError, e:
> +        if e.getcode() != 404:
> +            raise
> +        sys.stderr.write("hints-file: %s gave 404 error\n" % hints_file)
>
>     # output suitable for the extra-override file
>     for pkgname in sorted(pkg_support_time.keys()):
> -        # go over the supported arches, they are divided in
> -        # first-class (PRIMARY) and second-class with different
> -        # support levels
> -        for arch in SUPPORTED_ARCHES:
> -            # full LTS support
> -            if arch in PRIMARY_ARCHES:
> -                print "%s/%s %s %s" % (
> -                    pkgname, arch, SUPPORT_TAG, pkg_support_time[pkgname])
> -            else:
> -                # not a LTS supported architecture, gets only regular
> -                # support_timeframe
> -                print "%s/%s %s %s" % (
> -                    pkgname, arch, SUPPORT_TAG, SUPPORT_TIMEFRAME[0][0])
> +        # special case, the hints file may contain overwrites that

s/overwrites/overrides ?

> +        # are arch-specific (like zsh-doc/armel)
> +        if "/" in pkgname:
> +            print "%s %s %s" % (
> +                pkgname, SUPPORT_TAG, pkg_support_time[pkgname])

Could this not be done below with something like:

if "/" in pkgname:
    pkgname_and_arch = pkgname
else:
    pkgname_and_arch = "%s/%s" % (pkgname, arch)

That would make your diff a bit smaller.

> +        else:
> +            # go over the supported arches, they are divided in
> +            # first-class (PRIMARY) and second-class with different
> +            # support levels
> +            for arch in SUPPORTED_ARCHES:
> +                # ensure we do not overwrite arch-specific overwrites
> +                pkgname_and_arch = "%s/%s" % (pkgname, arch)
> +                if pkgname_and_arch in pkg_support_time:
> +                    break
> +                if arch in PRIMARY_ARCHES:
> +                    # arch with full LTS support
> +                    print "%s %s %s" % (
> +                        pkgname_and_arch, SUPPORT_TAG, pkg_support_time[pkgname])
> +                else:
> +                    # not a LTS supported architecture, gets only regular
> +                    # support_timeframe
> +                    print "%s %s %s" % (
> +                        pkgname_and_arch, SUPPORT_TAG, SUPPORT_TIMEFRAME[0][0])
>

review: Needs Information (code)

Revision history for this message

William Grant (wgrant) wrote on 2010-04-01:

#

The functionality seems good. Apart from the stylistic issues that noodles identified, this looks fine.

review: Approve (code)

Revision history for this message

Michael Vogt (mvo) wrote on 2010-04-12:

#

Many thanks for the review. I updated the branch now to cover most of your points.

Revision history for this message

Michael Vogt (mvo) wrote on 2010-04-12:

#

The cover letter:

Summary:
Add ability to override seeds to make support information more accurate

Pre-implementation notes
For some package the seeds are not expressive enough, e.g. kvm is something we want to support for 5y, but the multimedia stack that is part of kvm and is not required to run kvm as a headless service (like libsdl) we only want to support for 3y.

Implementation details:
The implmentation adds the ability to override specific source or binary package support information.

Tests:
There are no unitests (yet). One test is running it and verifing that the output changes from:
libsdl1.2debian/amd64 Supported 5y
to 3y

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-13:

#

Thanks for the changes Michael. I spoke with Julian and he's ok without the tests this time, but would like to start always adding tests for this kind of thing. We'll need to run it on Dogfood before landing it.

review: Approve (code)

Revision history for this message

Julian Edwards (julian-edwards) wrote on 2010-04-13:

#

Hi

This can go in as it is, provided that it's tested on dogfood first. However, before any more non-urgent changes go in you should write unit tests please. If you need help with that, we'll be glad to assist.

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-13:

#

Tested initially on df with:

https://pastebin.canonical.com/30476/

Michael will update the script so it doesn't depend on a newer version of python-apt.

Revision history for this message

Michael Vogt (mvo) wrote on 2010-04-13:

#

I fixed this now to work with the python-apt version in hardy. Please give it another go.

review: Needs Resubmitting

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-13:

#

Great, thanks Michael!

Here's the output, please check if it's as you expected:

{{{
$ ./codelines/current/cronscripts/publishing/maintenance-check.py lucid > ./ubuntu-archive/ubuntu-misc/more-extra.override.lucid.main_after_mvos_change
hints-file: changing x11-common from 5y to 3y
hints-file: changing libsdl1.2debian from 5y to 3y
hints-file: changing libsdl1.2debian-alsa from 5y to 3y
hints-file: changing libpulse0 from 5y to 3y
hints-file: changing libsndfile1 from 5y to 3y
hints-file: changing libogg0 from 5y to 3y
hints-file: changing libvorbis0a from 5y to 3y
hints-file: changing libvorbisenc2 from 5y to 3y
hints-file: changing libhal-storage1 from 5y to 3y
hints-file: changing libhal1 from 5y to 3y
hints-file: changing hal from 5y to 3y
hints-file: changing hal-info from 5y to 3y
}}}

But doing a diff on the before/after seems to just have *all* packages changed to 5y?
https://pastebin.canonical.com/30498/

review: Needs Information

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-13:

#

OK, after chatting with mvo, I've run the script but *without* the server-ship change of r9154.

{{{
$ ./codelines/current/cronscripts/publishing/maintenance-check.py lucid > ./ubuntu-archive/ubuntu-misc/more-extra.override.lucid.main_after_mvos_change_but_without_server_ship
hints-file: changing x11-common from 5y to 3y
}}}

and:
{{{
$ diff more-extra.override.lucid.main_before_mvos_change more-extra.override.lucid.main_after_mvos_change_but_without_server_ship > /home/michaeln/more-extra.override.lucid.main.mvo.diff.3
}}}

which gives
{{{
$ less more-extra.override.lucid.main.mvo.diff.3
903a904,906
> crypto-modules-2.6.32-20-generic-pae-di/i386 Supported 5y
> crypto-modules-2.6.32-20-generic-pae-di/amd64 Supported 5y
> crypto-modules-2.6.32-20-generic-pae-di/armel Supported 18m
3867a3871,3873
> kernel-image-2.6.32-20-generic-pae-di/i386 Supported 5y
> kernel-image-2.6.32-20-generic-pae-di/amd64 Supported 5y
> kernel-image-2.6.32-20-generic-pae-di/armel Supported 18m
16272a16279,16281
> nic-shared-modules-2.6.32-20-generic-pae-di/i386 Supported 5y
> nic-shared-modules-2.6.32-20-generic-pae-di/amd64 Supported 5y
> nic-shared-modules-2.6.32-20-generic-pae-di/armel Supported 18m
17667a17677,17679
> ppp-modules-2.6.32-20-generic-pae-di/i386 Supported 5y
> ppp-modules-2.6.32-20-generic-pae-di/amd64 Supported 5y
> ppp-modules-2.6.32-20-generic-pae-di/armel Supported 18m
19182a19195,19197
> serial-modules-2.6.32-20-generic-pae-di/i386 Supported 5y
> serial-modules-2.6.32-20-generic-pae-di/amd64 Supported 5y
> serial-modules-2.6.32-20-generic-pae-di/armel Supported 18m
20773,20774c20788,20789
< x11-common/i386 Supported 5y
< x11-common/amd64 Supported 5y
---
> x11-common/i386 Supported 3y
> x11-common/amd64 Supported 3y
}}}

Are those additional entries expected? If not I can re-run script without your changes to confirm.

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-14:

#

Approving after the above tests on dogfood.

review: Approve (code)

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-14:

#

I meant to mention, although it's unrelated to your branch, when running the maintenance check for karmic the following happens on dogfood:
https://pastebin.canonical.com/30562/

To find out why, I added the following:
{{{
=== modified file 'cronscripts/publishing/maintenance-check.py'
--- cronscripts/publishing/maintenance-check.py 2010-01-22 13:57:45 +0000
+++ cronscripts/publishing/maintenance-check.py 2010-04-14 10:10:45 +0000
@@ -68,6 +68,8 @@
     get structure file for named distro and distro version
     (e.g. kubuntu, lucid)
     """
+ url = "%s/%s.%s/structure" % (BASE_URL, name, version)
+ print url
     f = urllib2.urlopen("%s/%s.%s/structure" % (BASE_URL, name, version))
     structure = f.readlines()
     f.close()
}}}

and the result is:
{{{
http://people.canonical.com/~ubuntu-archive/germinate-output//ubuntu.karmic/structure
http://people.canonical.com/~ubuntu-archive/germinate-output//kubuntu.karmic/structure
http://people.canonical.com/~ubuntu-archive/germinate-output//edubuntu.karmic/structure
http://people.canonical.com/~ubuntu-archive/germinate-output//netbook.karmic/structure
}}}

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-14:

#

JFTR: the above is expected by Michael:

12:28 <mvo> hello
12:28 <mvo> sorry for the slow reply, I was at lunch
12:29 <mvo> it needs to run post release for -updates and friends
12:30 <noodles> Great, as long as it's expected behavior.

Revision history for this message

Michael Nelson (michael.nelson) wrote on 2010-04-14:

#

I created bug 562938 for the above.

Launchpad itself

Merge lp:~mvo/launchpad/support-timeframe-information into lp:launchpad/db-devel

Commit message

Description of the change

Preview Diff

Subscribers

 === modified file 'cronscripts/publishing/cron.germinate'
 --- cronscripts/publishing/cron.germinate	2010-03-02 09:58:34 +0000
 +++ cronscripts/publishing/cron.germinate	2010-04-13 12:58:27 +0000
@@ -127,7 +127,7 @@
  echo " done."
  # now generate the Supported extra overrides
--$MAINTAINCE_CHECK $suite > "$MISCROOT/more-extra.override.$suite.main.supported"
++$MAINTAINCE_CHECK $suite > "$MISCROOT/more-extra.override.$suite.main.supported" 2> maintenance-check.stderr
  if [ $? -eq 0 ]; then
      cat "$MISCROOT/more-extra.override.$suite.main.supported" >> "$MISCROOT/more-extra.override.$suite.main.new"
  fi
 === modified file 'cronscripts/publishing/maintenance-check.py'
 --- cronscripts/publishing/maintenance-check.py	2010-04-12 15:13:53 +0000
 +++ cronscripts/publishing/maintenance-check.py	2010-04-13 12:58:27 +0000
@@ -6,9 +6,19 @@
  #  https://code.edge.launchpad.net/~mvo/ubuntu-maintenance-check/python-port
  # (where it will vanish once taken here)
++# this warning filter is only needed on older versions of python-apt,
++# once the machine runs lucid it can be removed
++import warnings
++warnings.filterwarnings("ignore","apt API not stable yet")
++import apt
++warnings.resetwarnings()
++
++import apt_pkg
  import logging
++import os
  import sys
  import urllib2
++import urlparse
  from optparse import OptionParser
@@ -59,24 +69,98 @@
  # germinate output base directory
  BASE_URL = "http://people.canonical.com/~ubuntu-archive/germinate-output/"
++# hints dir url, hints file is "$distro.hints" by default
++# (e.g. lucid.hints)
++HINTS_DIR_URL = "http://people.canonical.com/~ubuntu-archive/seeds/platform.%s/SUPPORTED_HINTS"
++
++# we need the archive root to parse the Sources file to support
++# by-source hints
++ARCHIVE_ROOT = "http://archive.ubuntu.com/ubuntu"
++
  # support timeframe tag used in the Packages file
  SUPPORT_TAG = "Supported"
--
--def get_structure(name, version):
--    """
--    get structure file for named distro and distro version
--    (e.g. kubuntu, lucid)
--    """
--    f = urllib2.urlopen("%s/%s.%s/structure" % (BASE_URL, name, version))
++def get_binaries_for_source_pkg(srcname):
++    """ Return all binary package names for the given source package name.
++
++    :param srcname: The source package name.
++    :return: A list of binary package names.
++    """
++    pkgnames = set()
++    recs = apt_pkg.GetPkgSrcRecords()
++    while recs.Lookup(srcname):
++        for binary in recs.Binaries:
++            pkgnames.add(binary)
++    return pkgnames
++
++def expand_src_pkgname(pkgname):
++    """ Expand a package name if it is prefixed with src.
++
++    If the package name is prefixed with src it will be expanded
++    to a list of binary package names. Otherwise the original
++    package name will be returned.
++
++    :param pkgname: The package name (that may include src:prefix).
++    :return: A list of binary package names (the list may be one element long).
++    """
++    if not pkgname.startswith("src:"):
++        return [pkgname]
++    return get_binaries_for_source_pkg(pkgname.split("src:")[1])
++
++def create_and_update_deb_src_source_list(distroseries):
++    """ Create sources.list and update cache.
++
++    This creates a sources.list file with deb-src entries for a given
++    distroseries and apt.Cache.update() to make sure the data is up-to-date.
++    :param distro: The code name of the distribution series (e.g. lucid).
++    :return: None
++    :raises: IOError: When cache update fails.
++    """
++    # apt root dir
++    rootdir="./aptroot.%s" % distroseries
++    sources_list_dir = os.path.join(rootdir, "etc","apt")
++    if not os.path.exists(sources_list_dir):
++        os.makedirs(sources_list_dir)
++    sources_list = open(os.path.join(sources_list_dir, "sources.list"),"w")
++    for pocket in [
++        "%s" % distroseries,
++        "%s-updates" % distroseries,
++        "%s-security" % distroseries]:
++        sources_list.write(
++            "deb-src %s %s main restricted\n" % (
++                ARCHIVE_ROOT, pocket))
++    sources_list.close()
++    # create required dirs/files for apt.Cache(rootdir) to work on older
++    # versions of python-apt. once lucid is used it can be removed
++    for d in  ["var/lib/dpkg",
++               "var/cache/apt/archives/partial",
++               "var/lib/apt/lists/partial"]:
++        if not os.path.exists(os.path.join(rootdir,d)):
++            os.makedirs(os.path.join(rootdir,d))
++    if not os.path.exists(os.path.join(rootdir,"var/lib/dpkg/status")):
++        open(os.path.join(rootdir,"var/lib/dpkg/status"),"w")
++    # open cache with our just prepared rootdir
++    cache = apt.Cache(rootdir=rootdir)
++    cache.update(apt.progress.FetchProgress())
++
++def get_structure(distroname, version):
++    """ Get structure file conent for named distro and distro version.
++
++    :param name: Name of the distribution (e.g. kubuntu, ubuntu, xubuntu).
++    :param version: Code name of the distribution version (e.g. lucid).
++    :return: List of strings with the structure file content
++    """
++    f = urllib2.urlopen("%s/%s.%s/structure" % (BASE_URL, distroname, version))
      structure = f.readlines()
      f.close()
      return structure
  def expand_seeds(structure, seedname):
--    """
--    expand seed by its dependencies using the strucure file
--    returns a set() for the seed dependencies (excluding the original seedname)
++    """ Expand seed by its dependencies using the strucure file.
++
++    :param structure: The content of the STRUCTURE file as string list.
++    :param seedname: The name of the seed as string that needs to be expanded.
++    :return: a set() for the seed dependencies (excluding the original seedname)
      """
      seeds = []
      for line in structure:
@@ -150,6 +234,8 @@
      parser.add_option("--source-packages", "", default=False,
                        action="store_true",
                        help="show as source pkgs")
++    parser.add_option("--hints-file", "", default=None,
++                      help="use diffenrt use hints file location")
      (options, args) = parser.parse_args()
      # init
@@ -160,6 +246,17 @@
              sys.exit(1)
      else:
          distro = "lucid"
++
++    # make sure our deb-src information is up-to-date
++    create_and_update_deb_src_source_list(distro)
++
++    if options.hints_file:
++        hints_file = options.hints_file
++        (schema, netloc, path, query, fragment) = urlparse.urlsplit(hints_file)
++        if not schema:
++            hints_file = "file:%s" % path
++    else:
++        hints_file = HINTS_DIR_URL % distro
      # go over the distros we need to check
      pkg_support_time = {}
@@ -175,20 +272,60 @@
          else:
              support_timeframe = SUPPORT_TIMEFRAME
          get_packages_support_time(structure, name, pkg_support_time, support_timeframe)
++
++    # now check the hints file that is used to overwrite
++    # the default seeds
++    try:
++        for line in urllib2.urlopen(hints_file):
++            line = line.strip()
++            if not line or line.startswith("#"):
++                continue
++            try:
++                (raw_pkgname, support_time) = line.split()
++                for pkgname in expand_src_pkgname(raw_pkgname):
++                    if support_time == 'unsupported':
++                        try:
++                            del pkg_support_time[pkgname]
++                            sys.stderr.write("hints-file: marking %s unsupported\n" % pkgname)
++                        except KeyError:
++                            pass
++                    else:
++                        if pkg_support_time.get(pkgname) != support_time:
++                            sys.stderr.write(
++                                "hints-file: changing %s from %s to %s\n" % (
++                                    pkgname,  pkg_support_time.get(pkgname),
++                                    support_time))
++                            pkg_support_time[pkgname] = support_time
++            except:
++                logging.exception("can not parse line '%s'" % line)
++    except urllib2.HTTPError, e:
++        if e.getcode() != 404:
++            raise
++        sys.stderr.write("hints-file: %s gave 404 error\n" % hints_file)
      # output suitable for the extra-override file
      for pkgname in sorted(pkg_support_time.keys()):
--        # go over the supported arches, they are divided in
--        # first-class (PRIMARY) and second-class with different
--        # support levels
--        for arch in SUPPORTED_ARCHES:
--            # full LTS support
--            if arch in PRIMARY_ARCHES:
--                print "%s/%s %s %s" % (
--                    pkgname, arch, SUPPORT_TAG, pkg_support_time[pkgname])
--            else:
--                # not a LTS supported architecture, gets only regular
--                # support_timeframe
--                print "%s/%s %s %s" % (
--                    pkgname, arch, SUPPORT_TAG, SUPPORT_TIMEFRAME[0][0])
++        # special case, the hints file may contain overrides that
++        # are arch-specific (like zsh-doc/armel)
++        if "/" in pkgname:
++            print "%s %s %s" % (
++                pkgname, SUPPORT_TAG, pkg_support_time[pkgname])
++        else:
++            # go over the supported arches, they are divided in
++            # first-class (PRIMARY) and second-class with different
++            # support levels
++            for arch in SUPPORTED_ARCHES:
++                # ensure we do not overwrite arch-specific overwrites
++                pkgname_and_arch = "%s/%s" % (pkgname, arch)
++                if pkgname_and_arch in pkg_support_time:
++                    break
++                if arch in PRIMARY_ARCHES:
++                    # arch with full LTS support
++                    print "%s %s %s" % (
++                        pkgname_and_arch, SUPPORT_TAG, pkg_support_time[pkgname])
++                else:
++                    # not a LTS supported architecture, gets only regular
++                    # support_timeframe
++                    print "%s %s %s" % (
++                        pkgname_and_arch, SUPPORT_TAG, SUPPORT_TIMEFRAME[0][0])