support shipping verbatim files in the exported tarballs

Currently, /usr/share/gnome/help has 103 MB of language specific files, which gzip -9 into 72 MB.

Some typical sizes of a particular translation (all gzip -9 compressed):
  es: 9.2 MB
  de: 4.4 MB
  fr: 5.5 MB
  ar: 780 KB

This would be a lot of work. It would not solve the problem fully (e.g. we couldn't generate those PNG files), and the intermediate step of exporting XML files and such untranslated wouldn't be all that helpful.

For this use case, it should definitely be done outside Launchpad. Where we find a real need to do something like this inside Launchpad, it'd be a blueprint rather than a bug.

Rosetta doesn't actually need to interpret any of those files, or change them. The problem is that we need some hook in Launchpad to export the files which we stripped during the build to some place where we can handle them. That could be as easy as making the uploaded _translations.tar.gz files available somewhere (right now they just disappear into a black box).

Please close again if even this simple solution is too much work. If so, do you have another idea?

I don't even think we ever keep the uploaded tarballs per se in Librarian. A lot relies on us treating the individual translation files inside the tarball as separate files, and we don't have a single other reason to keep the tarball around. This is not a generic file-sharing application!

Danilo points to Soyuz as a possible place to look for the stripped data.

Hang on, let's keep this open. I asked Adam and Celso a while ago, but did not get a response so far.

On UDS I had a discussion with Danilo about how to move GNOME help file translations to language packs [1], since they are extremely big.

There doesn't currently seem to be a way to do this through the translations.tar.gz tarball and Rosetta.

So I wondered if it is possible to build another raw-gnome-help or similar tarball during package build and stuff that to the _arch.changes. Would that file land in the librarian somehow, so that langpack-o-matic can pick them up and unpack them into the langpacks? If not, would that be something easy to do? Or am I totally on crack
with the idea?

[1] https://wiki.ubuntu.com/DesktopTeam/Specs/JauntyGnomeHelpLangpacks

Pitti,

Yes, implementing a new custom format is fairly easy, the uploaded files will end up in librarian.

You should decide how is the way to expose them. From what I can see they don't need to end up in the repository (as d-i or ddtp-tarballs), so we could provide web UI and/or APIs to facilitate browsing and download.

I'm assuming the changes required in pkgbinarymangler are feasible, I can't really evaluate them.

Hey Celso,

Celso Providelo [2009-01-27 17:00 -0000]:
> Yes, implementing a new custom format is fairly easy, the uploaded files
> will end up in librarian.

Glad to hear it! Assuming the .changes would have a raw-*
tarball, how could I find these tarballs in the librarian?

> You should decide how is the way to expose them. From what I can see
> they don't need to end up in the repository (as d-i or ddtp-tarballs),

Confirmed, I just need to be able to find all of them mechanically,
i. e. the latest version for each source package per distrorelease.

> so we could provide web UI and/or APIs to facilitate browsing and
> download.

That would be awesome.

> I'm assuming the changes required in pkgbinarymangler are feasible, I
> can't really evaluate them.

They are, I just set it to incomplete because I don't know yet how
soyuz needs to get them fed. Would something like this be okay?

  123DEADBEEF 123456 raw-translations-static - gnome-mount_0.8-1ubuntu1_i386_translations-static.tar.gz

Thanks, Martin

Pitti,

The .changes snippet suggested looks fine. 'raw-translations-static' identifier is fine too, although it may cause some confusion with the current 'raw-translations' one, but I don't mind as long as it's clear to you.

Regarding the look-up mechanism I'm not entirely sure. Custom uploads are directly related to build records but in most of the cases they are only generated on the i386 attempts (nominatedarchindep), resulting in source->[builds]->[custom] that can be easily aggregated and exposed via the API as:

{{{
>>> ubuntu = launchpad.distributions['ubuntu']
>>> firefox = ubuntu.getSourcePackage('firefox')
>>> for custom_file in firefox.custom_files:
.... print custom_file.filename, custom_file.custom_type
firefox_1.0_i386-translations.tar.gz RAW-TRANSLATION
firefox_1.0_i386-translations-static.tar.gz RAW-TRANSLATIONS-STATIC
...
}}}

Or something similar. Obviously custom file would allow you to get the file contents via the API as well.

On the web UI side, we could simply present 'custom files' everywhere we present the source and binary files, I guess.

That launchpadlib workflow looks suitable for me, that would be great.

The solution design seems clear to me now, thus marking as Triaged. Unassigning since the pkgbinarymangler fix is blocked by getting the soyuz feature.

I've some work in the same are related with the debug-symbols support scheduled to 2.2.3. Let's keep this bug around, maybe I can do something yet in this cycle, but no guarantees.

Any chance we could get this implemented soon? We have to start throwing out some serious features from the desktop CD soon otherwise (killing GIMP, for example).

Celso, let's try and get this done in 2.2.6, we can speak about it in the week.

After talking to pitti, here's what we're going to do:

1. Add a new custom upload file type, RAW-TRANSLATIONS-STATIC
2. Store these files in the librarian, hanging off the usual PackageUploadCustom table entries.
2. Expose a new webservice method: IDistroSeries.getStaticTranslations(since="<date>") which will return a collection of URLs to the files in the librarian.

The files are easily associated with the original package because it's part of the file name.

Optionally, we can add some webservice calls like Celso suggests above.

The buildds also need to be changed to strip these files out and put them in the build, of course.

Job done.

Fixed in devel, RF 8772

Julian Edwards [2009-07-23 10:32 -0000]:
> ** Changed in: soyuz
> Status: Fix Committed => Fix Released

Wohoo, thanks muchly!

So what do I have to do now to make this work? Can pkgbinarymangler
just start to upload RAW-TRANSLATIONS-STATIC files? And how to get
them back through launchpadlib?

Thanks! Martin

Yes, there's still some work on pkgbinarymangler of course, which needs to be
integrated in a new chroot for the buildds.

To retrieve the files on launchpdlib you need to use the getPackageUpload()
method on the DistroSeries object and pass it a param custom_type='raw-
translations-static'. The returned objects have a custom_file_urls list
property that specify librarian URLs where you can get the files.

Julien,

I tested it on dogfood in https://dogfood.launchpad.net/~pitti/+archive/gnome-help-test and gnome-applets built fine:

  pkgstriptranslations: preparing static translation tarball gnome-applets_2.27.3-0ubuntu3test2_i386_translations_static.tar.gz...done (403 files)

But it failed to upload:

2009-08-03 22:02:20 WARNING Upload was rejected:
2009-08-03 22:02:20 WARNING Upload has more architetures than it is supported.

Perhaps it stumbled over the raw-translations-static .changes part? Celso will fish out the .changes.

The upload is on https://chinstrap.canonical.com/~cprov/20090803-230207-1064973-2323117/

Reopening the bug for now. Thanks in advance for having a look!

cprov-afk| pitti: aha, we have a problem in the upload processor
cprov-afk| pitti: we cope with <arch>_translation but not with <arch>_translation_static

As this bug is still in progress, I'm re-targeting it to 2.2.8.

Assigning back to 2.2.7 and closing. Please open a new bug with the relevant and specific problems if necessary Martin.

Ah, after the n-th iteration of naming schemas it works now. Thanks for your help!

pkgbinarymangler (61) karmic; urgency=low

  * pkgstriptranslations: Extract GNOME help translations into a new tarball
    and add them to the upload as raw-translations-static component. Do not
    strip the files from the .debs just yet, since the upload/download
    infrastructure in Soyuz still needs to be tested, and the language pack
    integration bits need to be written. See LP #123020
  * debian/compat, debian/control: Bump to debhelper compat 5, 4 is
    deprecated.
  * debian/control: Bump Standards-Version to 3.8.2.
  * debian/control: Drop long-obsolete Replaces.

 -- Martin Pitt <email address hidden> Mon, 03 Aug 2009 12:59:52 +0100

From the PPA, I get

$ ./static-translations
upload: https://api.dogfood.launchpad.net/beta/ubuntu/karmic/+upload/1085128
display name: gnome-applets, gnome-applets_2.27.3-0ubuntu3test5_i386_translations.tar.gz, gnome-applets_2.27.3-0ubuntu3test5_static_translations.tar.gz
display version: 2.27.3-0ubuntu3test5
custom files: ['https://librarian.dogfood.launchpad.net/27630559/gnome-applets_2.27.3-0ubuntu3test5_i386_translations.tar.gz', 'https://librarian.dogfood.launchpad.net/27630560/gnome-applets_2.27.3-0ubuntu3test5_static_translations.tar.gz']

which looks fine.

pkgbinarymangler (62) karmic; urgency=low

  * pkgstriptranslations:
    - Also store /usr/share/omf/ files in static translation tarball.
    - Remove GNOME help translation files now, and replace them with symlinks
      to /usr/share/omf-langpack/* and usr/share/gnome/help-langpack/*.

 -- Martin Pitt <email address hidden> Sun, 30 Aug 2009 12:39:06 +0200

pkgbinarymangler (61) karmic; urgency=low

  * pkgstriptranslations: Extract GNOME help translations into a new tarball
    and add them to the upload as raw-translations-static component. Do not
    strip the files from the .debs just yet, since the upload/download
    infrastructure in Soyuz still needs to be tested, and the language pack
    integration bits need to be written. See LP #123020
  * debian/compat, debian/control: Bump to debhelper compat 5, 4 is
    deprecated.
  * debian/control: Bump Standards-Version to 3.8.2.
  * debian/control: Drop long-obsolete Replaces.

 -- Martin Pitt <email address hidden> Mon, 03 Aug 2009 12:59:52 +0100

langpack-o-matic now downloads the tarballs and installs them into the langpacks. Just tested a recent build, about to upload them.