Store/read tags inside photos

To be a bit more precise, existing photo collections' tags (i.e. keywords) are normally stored in IPTC and/or XMP data (EXIF can't really hold keyword data). The trunk build of Shotwell can now read IPTC/XMP keywords from photo files; these become Shotwell tags after an import. So this feature will be present in 0.6.

It's true that we have a ticket (#1290) for storing Shotwell tags in photo files. We're currently debating, however, whether Shotwell will do this on the fly or only when the user exports photos explicitly. We're currently leaning toward writing tags out only during an export operation, because if we always store tags in photo files then tagging becomes a relatively expensive operation. A user might quite reasonably want to apply a tag to 100 photos, but if we need to write that tag to 100 10 Mb photo files then now we need to write 1 Gb of disk data in order to apply the tag (since we may need to rewrite an entire photo file in order to insert a new tag into it).

Some other photo programs (e.g. Picasa) do, however, write tags out on the fly; others (e.g. iPhoto, Aperture) do not. F-Spot gives the user the option whether to do this.

Have you considered a background task to update the tags?
Tagging represents a lot of work by the user and this can be easily lost. (e.g. a user backing up their files to CD and reinstalling the OS without realising that the Shotwell database has not been backed up).
If the performance is too critical I'd recommending having the option like F-Spot - we would deliver it enabled and more advanced users may consider disabling it as a speed up.

If we do decide to keep tags up to date in photo files, we will absolutely update them via a background task: we certainly don't want to make users wait while that happens! :)

There's another question here: if we do keep tags in photo files, we need to decide whether to update master photo files when photos are tagged, or to create a modified copy of each master solely for the purpose of holding the tags, even if there are no changes to the image itself. It seems wasteful to copy the files just to hold tags: this will double the library size if every file is tagged. But, again, some users might think that the master files are sacred and might not want even tags written there.

:) I mean, what is the cost of writing 1G? As long as it's done in the background it shouldn't matter to the user (aside from using energy and generating heat). There is a small risk of the user trying to access the file until it's written but as long as that's visually indicated in Shotwell then they've been warned. I am sure if the tags are not written there will be enough people who will forget/not know to use Shotwell to do the export.

I'd say if you want to cater to both classes of users then you'll need the checkbox in the preferences.
a) Pristine mode to keep all info in the database only, generating a "(modified)" file for changes
b) Modify mode to update file metadata
And we'd default to the latter

Another factor to consider here: starting with Shotwell 0.6 the user can import RAW photos from cameras. The exiv2 library can't write metadata into most RAW formats (run "man exiv2" to see a list of formats and the metadata which can be read/written into them). Writing metadata into a RAW file is considered slightly risky even when using programs such as Aperture; see http://aperture.maccreate.com/2010/02/10/write-iptc-metadata-to-masters/ for a discussion, for example.

So if a photo is RAW, we can't realistically keep up-to-date metadata in the photo file; we can only store it in the Shotwell database and/or a parallel JPEG file generated from the RAW photo.

Perhaps someday camera manufacturers will converge on a single RAW format where software can easily update metadata; DNG is the best hope for that. That day is probably far off, though.

As long as Shotwell is "best effort" this should be fine. The 99% use case I'm thinking of is:
- JPEG files taken from a point-and-shoot camera/phone
- Some existing management (either filenames/directories and/or tags)
- Sharing via web services (Facebook and twitter).
- Printing on a printer
- Copying to USB stick

So I'd be happy that users who use things like RAW files will be able to understand the limitations of that.

The single most important function of a photo management app (for me, personally) is the ability to organize and search for photos by keyword. If Shotwell does not allow at least the *option* of saving tags/keywords to the masters, the program will be of no use to me, and I'll have to stick with F-Spot.

I completely understand that some users don't want their applications messing, in any way, with their original files. I, however, feel that the editing and application of metadata for photos should be non-destructive and completely transparent. I don't want to *have* to have JPEG versions of undeveloped RAW files lying around, just to store tags; I don't want .xmp files clogging up my filesystem; and I don't want to be locked into some application's proprietary database. The capability to store metadata inside the files themselves exists, as it has for mp3 files for decades, and I should be able to rely on that, as I do for mp3s.

As an example--if we have to create a JPEG version of a RAW file to store tags, how, when we go back to that RAW file to properly process it, do we re-associate the tags from the Shotwell created JPEG with the (raw editor of choice)? Manually? Likewise if the keywords & other Shotwell managed metadata is stored in the database.

If the metadata is IN the RAW file, our (raw editor of choice) should be able to export our newly processed JPEG with all metadata intact, with no extra effort on the part of the user.

I was content to watch Shotwell develop (and you guys are making crazy, awesome progress), but if Shotwell is to replace F-Spot as the default photo management app on Ubuntu desktop, then you guys really have to get the basic functionality in place before that happens. And if you can't properly manage metadata, you're not a management app, you're just a browser.

Apologies for going on a bit of a rant there.

@Adam,
What about making it configurable? Give the user the following options:
- always store tags in original files (and possibly with a sub-option that says "in RAW files too")
- never store tags in original files
- only store tags in files on export

@coffeemonk,
The bug is under review and will be addressed. However, it needs to take into consideration users who may have different requirements. Also, one aspect which is essential, as Adam pointed out, is to not accidentally bork RAW files in doing this, considering that writing metadata to some RAW file formats is considered risky. I'm sure you would be a lot more annoyed if you lost your RAW files than if they didn't contain metadata.
Note that whichever solution is used, this does not prevent you organising and searching your photos by keyword.

@coffeemonk,
Thanks for your feedback, and no worries - we like hearing what our users really want. :) Clearly you'd like Shotwell to be able to store tags inside RAW files. I think that would be great, though as I mentioned above there are some technical challenges above. In particular, the exiv2 library used by Shotwell can't yet write metadata into many RAW formats. That may change, though; for example, there's a feature request to extend exiv2 to write metadata into Canon RAW CR2 files and it looks as if it may be implemented soon (see http://dev.exiv2.org/issues/show/635). In any case, I hope we can add this capability to Shotwell at some point.

By the way, F-Spot can't write metadata into RAW files either (at least not into CR2 files, which I just tried).

@Bruno,
I agree: it would be great to give this user this option. When we begin development of Shotwell 0.7 in a couple of weeks we'll think about this as a possibility.

@Adam and @Bruno,

Thanks for understanding my intent to offer a user's perspective, and passionate but I hope ultimately constructive feedback.

I think Bruno's suggestion for implementing a user preference, even down to the RAW sub-option, is absolutely what should be done. If this feature makes it into 0.7, I will definitely put Shotwell through it's paces again up against F-Spot to see what works best for me.

For what it's worth, I shoot Canon (and thus get CR2 files), but my import workflow includes running the CR2 files through the dngconverter from kipi-plugins. I made the choice a while back to go with a more open standard for my RAW files, for the express purpose of enhanced compatibility with other software. I see that exiv2 has ReadWrite support for DNG, so it looks like that was an appropriate choice.

I would also prefer metadata to be stored in files, if only to save a problem the next time that I move photo programs.

More important than my "me too", I have an additional use-case. We currently have a server with our shared photos and we access them over the network. Any changes to a local database would not be picked up by the other machine. This would also make it slower to update file-based tags, but a background task should be okay.

One additional thought that is probably worth considering is an option to strip out some of the tags on export, though I can't immediately think of too many sensitive tags. It may not be a bad idea generally, though, if people start uploading "picture of my garden" photos to the internet with GPS tags (inserted automatically by some cameras).

It would seem to make sense to follow the guidance of the Metadata Working Group:
http://www.metadataworkinggroup.org/pdf/mwg_guidance.pdf
http://www.metadataworkinggroup.org/

@Aaron: Thanks for your comments. Yes, the network sharing use case is interesting. As for stripping out tags on export, we're potentially open to that if people want it for privacy reasons. Feel free to file a ticket for that at Yorba Trac (trac.yorba.org) if you personally would want to use this feature. If you only have a vague feeling that other people might want it, though, let's wait until someone who has a concrete need for the feature chimes in.

Thanks also for the link to the Metadata Working Group recommendations - we hadn't seen these before.

I share Aaron's use-case of storing the photos on my server at home. The photos are presently tagged in Shotwell on my laptop. If my wife wants to browse them in their organised state on her laptop, then I either need to find a way of sharing Shotwell's db file or to host in on the server. Alternatively, having the info stored in each photo might do the trick but less efficiently. I'd like the option to occasionally 'update all changed meta-data' in order to ensure that my sorting isn't lost in the future upon reinstalling etc...

Thanks for the great software.

Is it possible to also store the title as Picasa does?

@Dominik:

Yes - if we store keywords in photo files then it would certainly make sense to store the title there too, as well as other metadata such as the rating. We've added a comment to the upstream ticket (http://trac.yorba.org/ticket/1290) to reflect that.

Great. Thank you.

I found this thread while looking for information whether or not shotwell store the keywords in the files "presently".

Just to add my "me too, i want this feature"...

Thx for the efforts.

Florian

I also want this feature, but would also echo what themuddler said. As task to sync metadata on demand would be perfect
This way those who do not want to update the master copy will not need to, and I can tag files on my eee on wireless, then plug in a cable to run the real file transaction (All photos are on my NAS).

I just switched from F-Spot to shotwell. And the only thing I really miss is to persist my tags in the image file. I agree with the Keith, and would at least have the option to store them on demand. I work in the exactly same scenario, small eee pc for browsing and tagging and a NAS to hold the collection.

+1 for both the "photos shared over network" and "my RAW files are CR2" use-cases.

Played with this a little. It doesn't seem to write to DNG files. Is there any hope to get metadata into RAW formats? Isn't this is supposed to be one of the points of using DNG over manufacturer formats?

@Todd: Yes, there is hope. :) See http://trac.yorba.org/ticket/2622 .

@ Adam Dingle

Chiming in on the request for an option to strip tags on export. It's all too easy to tag a bunch of photos with names, locations, etc., and then export them to who-knows-where without thinking. It would also be good if it always showed up as a visible option on any export dialogs -- it can really be a serious security and safety concern; and many people may not even consider where their sensitive personal information is going if the option is hidden deep in preferences. Additionally, it's a preference which would generally need to be considered per-export, as the location to which it is being exported is normally the deciding factor.

@cometdog: Thanks for the suggestion, which is ticketed upstream at http://trac.yorba.org/ticket/2556 , by the way. I've upped that ticket's priority to high since a few people have asked for this. Not yet sure whether this will make 0.8, however.

@Adam: Thanks for the update :)

I can't use Shotwell just because that the tags aren't saved inside the image. I have to use DigiKam instead and I'm not a KDE-person. I would really like to use Shotwell for all my images, not only for fast editing and viewing.

Couldn't you create and option for the user to either save the tags inside the image or not?

Daniel: Yes, we've implemented this capability in the trunk and it will be present in Shotwell 0.8, which we're planning to release in November. If you want to try this in the trunk build, you'll need to run Shotwell with the --autocommit-metadata command-line option, which tells Shotwell to save tags inside image files (since we haven't built the preferences checkbox to enable this yet). Or you can just wait for 0.8. :)

Adam: That is good news, thank you! And thank you for the tip.
I think I can wait until the 0.8 release ;)

Perhaps a bit of topic, but what will happened with the current (external) tagged photos? Will the keywords get stored in the file afterwards in 0.8 or will I have to tag them again?

The plan is when you first run 0.8 it will detect that your files are dirty and begin updating the metadata in the background. If you've added tags, changed titles, etc., they will be written to disk. If no changes have been made to a photo, the file is untouched.

-- Jim

Thank you Jim! That is also just lovely!

Please make this optional.

I sometimes tag photos temporary while searching for information and after I'm finished I delete these tags. The problem here is when the files are actually changed the backup program will notice that and backup all those files (and maybe twice!). So for me it would be great if I had an option to actually write the information to the image files on demand.

Thanks.

@Peter-Alexander: This feature has been implemented in trunk, is optional, and is off by default. It can currently be enabled via a command-line option, and in the next few days we'll be adding a preferences checkbox you can use to enable the feature if you want.

By the way, in Shotwell 0.9 we're planning to add the ability to flag photos. Perhaps you'll be able to use these flags instead of temporary tags. See http://trac.yorba.org/ticket/2756 .

shotwell 0.8.1 is now in Natty which should fix this issue

I'm just curious if this was fully implemented. I am currently using F-Spot mainly because it stores meta-data in the image files. According to the following article, there are three formats (Exif, IPTC, and XMP) for storing meta-data in the image file:

http://blogs.sitepoint.com/2006/08/24/open-source-image-archiving-exif-iptc-xmp-and-all-that/

According to the article XMP (standard BSD license) is the "holy grail" for storing meta-data in the image file.

Which of these formats does Shotwell support?

If XMP is supported, someone may want to update the following Wikipedia page to indicate that Shotwell is a supporting tool:

http://en.wikipedia.org/wiki/Extensible_Metadata_Platform

Yes, it has been implemented. Shotwell supports EXIF, IPTC, and XMP metadata formats. However, there was a design issue/bug in 0.8 where we only stored metadata in a particular format if it was already available in the image file (i.e. if there was no XMP in the fiel, no XMP data was written). This has been fixed and will be available in our upcoming 0.9 release: http://trac.yorba.org/ticket/3044

Michael, I've updated the Wikipedia page you mentioned to include Shotwell as you suggested.