Consilience

While I haven’t fully figured out the answer to my previous photography related question (on the archiving, not simply backing up, of photos) I am getting closer.

I have a question about the archiving of RAW files.

One of the commenters, Damon Lynch, provided some insight to a collection of best practices (“DPBestFlow”) developed by the Library of Congress (an organization that I trust when it comes to the process of archiving materials). [He also linked to a Free Software tool he wrote to quickly download and backup photos from your digital camera, Rapid Photo Downloader.]

Just your daily photo of Rowan.

The DPBestFlow site introduces me to a file format that I was previously unaware of, DNG, or Digital Negative. The DNG format, created by Adobe, is a ‘universal’ RAW format with an open (ie: not patent encumbered) specification. This is in stark contrast to the slew of proprietary formats that are camera maker specific (closed formats that might include encrypted portions) such as NEF (Nikon’s) or CRW/CR2 (Canon’s).

There was even an initiative to garner wider adoption of open standard RAW formats back in 2005 started by Juergen Specht called OpenRAW (Wikipedia).

Now, I am a huge proponent of open formats and standards (obviously?) but my question really is:

• Is it worth it?
• Should I convert all of my Nikon RAW files to DNG?
• If I do, should I save the original NEF files “just in case?”

Also, why am I not simply reading the many discussions already had on this topic? Because I want to get the uniquely free software view on the issue.

Thoughts?

I recently (end of November) bought myself an early Christmas present: a Nikon D3100. I did this mostly because our first child was due on December 10th (he came December 11th! – photos).

Now, as you imagine, as a crazy happy new dad, I’ve been taking a lot of photos. And I’ve been especially good about making sure they are all backed up. I have them on my laptop, my external RAID array, and now on my in a different state colo’d server.

But, since that D3100 is a DSLR I’ve been playing around with RAW images, and those can be pretty big. So far, just since I got this camera around November 30th I have about 17 gigs of photos already. My harddrive, which is an SSD, is going to fill up soon enough and it would be full now if I had imported all of my previous photos from my previous camera(s).

So, I’ve been looking for a smart way of dealing with large photo collections where part of it is on my laptop and part is ‘archived’ on an external harddrive or in the cloud or wherever.

What I see as a perfect work-flow for this is:

1. Take photos with camera
2. Import photos to Photo Management Software on your laptop
3. Process, tag, export, publish, etc
4. Repeat 1-3 many times
5. Use a ton of your computer’s harddrive space
6. Archive all photos except the last 60 gig/3 months (whatever) to an external harddrive, or nfs share, or cloud storage, etc
7. The Photo Management Software knows where those photos are, their metadata, and has a small thumbnail for them as placeholders in the timeline

The killer feature here is the Archive button. Admittedly, Google revolutionized webmail with that button (while getting tons of other things wrong with GMail*), who is going to revolutionize photo management with it?

Do any photo management applications out there do this without having to do stupid painful things like this (which makes me remove photos from one library and add them to a ‘backup library’ where I can’t see them all in the same Shotwell session).

* Like non-threaded email, non-standard IMAP server, etc

I had a great time at the Michigan LoCo team’s event for the Natty Ubuntu Global Jam. We had about 20 people show up, including the famous Jason Smith. We all gave Natty a spin, made things crash, reported bugs, triaged others, and gave user testing feedback (loosely).

Also, a big thanks to Ben Rousch for bringing the Founders, Arcadia, and New Holland beer from the west side of Michigan; we loves those breweries!

The rest of my photos, some from Jorge Castro, and more from Craig Maloney. If you were there and took photos, let me know!

Another Ubuntu Global Jam means another Michigan LoCo success.

From left to right is: Gib, Craig, Bob, myself, Rick, and Jim.

I think we hit every single proposed topic during the Jam. We talked about how to use the Ubuntu area of StackExchange and even had a few members post answers and gain some reputation points. We did some ISO testing (and ran up against bug 608382). We reported some bugs (bug 625989 and bug 626003). And we even played with some new toys (Gib’s 16 megapixel(?!) $10(?!) digital video camera).

And of course, since the two awesome co-hosts of Lococast (Rick and Craig) were in attendance, we chatted a bit about the future of their podcast including ideas for interviewee guests. If you have any ideas/suggestions for interview subject for Lococast, send them along to Rick and Craig at feedback@lococast.net. But I would highly recommend you to check out Lococast right now because they already have some great features like the Lococast Screencasts with 2 on VIM already and the always enjoyable Rick’s Rants.

AND! If you listen to Lococast you’ll notice they utilize really awesome CC-licensed metal for their intros/outros/transitions. If you want more of that, check out Craig’s Open Metal Cast. He just released the first episode. If you have some suggestions for Craig of great CC-licensed metal, send him a message at craig@openmetalcast.com (yes, Severed Fifth has already been highlighted :) ).

With all the talk of podcasts and Ubuntu we got a bit hungry and ordered some pizzas. There was only one problem; we needed to report a bug against Papa John’s! All of our pizzas and even the cheese bread weren’t sliced!

Does anyone know if there is a Launchpad project for Papa John’s?

I just got back from a great day at the Ubuntu Michigan LoCo edition of the Global Jam were we tested Lucid on a ton of different hardware. It was a great time. See the photos.

But, what I want to share with you right now is a quick script I whipped up to backup my Google Calendars nightly. This is one of the steps in my on-going process of making sure all of my personal data is backed up by me on machines I control with an eye to migrating to self (or friend) hosted services. Yes, I want services I use to follow the Franklin Street Statement.

Until the day that all of the services I use follow the Franklin Street Statement recommendations, I will just have to make sure I make personal backups of my information. So tonight, I finally did that for Google Calendars. It was pretty simple, really:

#!/bin/sh
# Backup my Google Calendars

WORK="/home/greg/backup/google/work-`date +%F`.ics"
PERSONAL="/home/greg/backup/google/personal-`date +%F`.ics"
OPENMICHIGAN="/home/greg/backup/google/open_michigan-`date +%F`.ics"
MILOCO="/home/greg/backup/google/miloco-`date +%F`.ics"

wget private_url_for_work_calendar -O $WORK
wget private_url_for_personal -O $PERSONAL
wget private_url_for_otherwork -O $OPENMICHIGAN
wget private_url_for_the_loco -O $MILOCO

# Remove files that are older than 1 week
find /home/greg/backup/google/*.ics -mtime +7 -exec rm -f {} \;

That’s it. Create the filenames for the various calendars I’m backing up, including today’s date. Then wget them. Then, delete any .ics file that is older than a week. Not sure why I need 7 days of backup, but better safe than sorry, I guess.

Here is my predicament:

I take photographs with my new shiny digital camera (thanks to all your input on my previous post) and I use F-Spot to manage my photos. I love F-Spot; it does almost everything I want it to do. It tags photos, it takes care of organizing the files on my harddrive (folders are not a method of categorization for photos, tags are), and it even uploads them to my flickr account.

The thing that it doesn’t do is let me embed my choice of license into the photos. It does let me embed the tags into the metadata, so the base functionality is there.

I am a huge supporter of Open content licenses, specifically Creative Commons licenses [ed: Greg is employed by Creative Commons] and I would love to see the ability to set the license of a photo to something of my choice (eg: CC:BY-SA) and have that license info be a part of the image no matter where it goes. And hey, there is even a bug report/feature request for this: here.

My proposed Use Case:
Jane is your typical photographer who uses Linux. As a Linux user she is more likely to know about open content licenses such as Creative Commons licenses and also more likely to use such a license for her photographs.

Jane takes a bunch of photos and imports them into F-Spot. F-Spot lets her set the license for a group of photos to CC:BY-SA. She then uploads those photos to her website. She makes it clear on her website that all her photos are licensed under CC:BY-SA.

Now, when Yahoo or Google go indexing her website their spiders can programatically discern that those photos are licensed under CC:BY-SA from the metadata. The benefits of this information can be seen when searching flickr for Creative Commons licenses: no more worrying about infringing on other’s copyright when you use a photo. Another example is search.creativecommons.org which searches across multiple sites at once for CC licensed material.

All kinds of cool new automated things can be thought of with this information available within the file. flickr automatically displaying the correct license for a photo without you having to specify it; the same thing with personally hosted photo galleries; users of sites like wikipedia can more easily find images for their uses; etc. Also, now that license information will always be in that file so others who find it will know what they can and can not do with it without asking.

Are you also interested in something like this? Take a look at the information on this wikipage which outlines what you need to do to support this in the XMP metadata standard. The how of doing it is most likely not that difficult since F-Spot is already supporting the metadata standard. Or just subscribe to this bug report so you are kept up to date on its progress.