Digital Asset Management – My Workflow

Version 1.  Paul Pavlinovich www.steamengine.com.au

All photographers store images. The challenges of organising and keeping track of these stored images can keep you awake at night.

The Beginning

I used to have a very simple workflow that most amateurs probably use; I simply copied the photos off the camera cards into a series of folders named for the activity I photographed. This reflected how I used to manage my film based photography. Stick the negatives into bags in a folder and organise the prints in boxes. This suited me fine in those days simply because the cost of film photography made you choose your images carefully. Any one activity would probably generate only four or five images.

Around 10 years ago I converted exclusively to digital photography. My image habits have changed from four or five images per activity to hundreds and sometimes thousands. Most professional photographers will shoot between 500 and 1500 images per activity. I don’t (generally) earn an income from my images, nor do I intend to but l follow professional practices and also generate huge numbers of images.

This was simple while I only had a few thousand images. Once I reached 100,000 (you’ll be surprised how quickly this happens!) I had generated a nightmare for myself. It became very hard to find images I wanted (that I remember taking) for some particular purpose and keeping backups in sync became more and more problematic. I made the situation even worse by managing pictures on more than one computer.

It took nearly 10 years of digital photography I have over a terabyte of image files. I have three different camera RAW formats and have JPEGs ranging from lowly 320 by 160 from my first digital camera to the 21 megapixel images from my current camera of choice my Canon 5D.

Modern photographic techniques such as High Dynamic Range where you take 3 to 20 (or more) images from a tripod with various bracketing, white balance and other settings to be merged into a single image worsen the problem.

The problem really came to my attention this year when I simply could not find an image that I needed. I knew it was there, but fruitlessly scanning through hundreds of folders left me drained and wasted a great many hours with no outcome at all.

I decided that I needed a management plan for my images so I started to seek what others had done, I read some blogs and personal web sites of photographers, I researched other online sources such as www.dpbestflow.org but they only scratched the surface. I am a member of the Australian Photographic Society. These organisations regularly distribute information and member only offers and include magazine subscriptions. One of the offers was for Peter Krogh’s “The DAM Book”. I had read the information on Peter’s website previously but wasn’t all that convinced the book would help me, this time it came with a complimentary license to Phase One’s Expression Media 2 (formerly a Microsoft product).  This swayed me because I knew I would need some decent catalogue software. The available free products like Microsoft Pro Photo Tools and Bridge that came with Photoshop CS3 really didn’t cut the mustard. Not to mention that Microsoft have recently dumped their image meta tagging, geo tagging and image management software.

The book arrived and I read it from cover to cover – I skipped some parts such as the explanations of how computer storage works and the available options because, well, I do that for a living. I digested Peter’s suggestions and constructed a workflow that is part way between what I had before and the extremely rigorous practices suited to a professional who is earning their bread and butter from their photography. In generating my workflow I took into account that storage is extremely cheap and that external drives can be purchased for around $80 per terabyte at the time of writing. This has lead me to the point of deciding not to backup to digital offline media such as DVD-W or BluRay. With hard disk storage being extremely cheap and reliable and archival quality media still being expensive and an unknown quantity for lifespan I have built my model around multiple online and offline hard disk storage. Update 2011: Ironically I no longer use EM2, I outgrew it very quickly. Today I use Adobe Lightroom. The cataloguing and editing facilities are excellent. It is now rare for me to need to visit Photoshop because I can do it all in LR.

Will my workflow work for you? Only you can answer that question. All I can say is that it works for me and that I’m writing this article to make it available to other people.

My Workflow

workflow

I have tried to keep my workflow as simple as possible while still following the principals of good digital asset management. I have balanced being anal retentive about backups and keeping copies and the costs of time and money in doing so.

To construct my workflow I determined what my basic requirements would be:

  1. Need to be able to create images in any format supported by my cameras both today and into the future
  2. Need to be able to ensure my images are available in the future
  3. Need to be able to view, edit and update my images in the Microsoft Windows platform
  4. Need to be able to manage my images both in a master repository and on the move
  5. Need to keep a backup of my images as they came from the camera before I work on them
  6. Need to be able to find my images easily and quickly
  7. Need to be able associate metadata with the images to interpret them for the future
  8. Need to be able to surface my best work
  9. Need to support post processing of my images
  10. Need to support publication of my images
  11. Need to support delivery of my images to other people

Your requirements may vary so you should start the process of determining your own workflow by writing down your requirements and thinking about how you will measure your success when you have “finished” creating your workflow.

Each of my requirements has dictated elements of my workflow. Some elements satisfy one requirement, some satisfy several at once. Let’s take them one at a time and interpret what they mean.

Need to be able to create images in any format supported by my cameras both today and into the future

Over my digital image taking experience I’ve had several cameras, initially they either had a proprietary version of an open standard (e.g. Casio had their own JPEG) or supported the open standard directly (e.g. Sony FD Mavica supported JPEG) or had both their own RAW format and JPEG support (e.g. Canon 300D and my current 40D).

Those of you who have been around computing for a while will be aware that file formats change over time. It can become challenging to handle and old format in a modern computing environment, particularly if that format is proprietary and requires tools from the manufacturer to work properly.

I don’t intend to enter the RAW vs. JPEG debate, but for my purposes I have changed my position – I used to shoot exclusively in JPEG because it was easy and open. Today I shoot in RAW because my photographic talents and my requirements for post processing have outgrown JPEG – I simply demand higher quality of my own work than I used to. The problem that this causes is RAW formats are proprietary.

Because I need to be able to use my images in the future and I choose to use a RAW format I don’t want to be locked into to any manufacturer or have to keep out-dated software on my modern computer that might impact its ability to work properly I choose to convert my images to the “open” image format known as Digital Negative (DNG) by Adobe. While this is still a proprietary format in that it is owned by a commercial organisation, Adobe has promoted it as a public archiving standard and has published the file content and format to allow other software vendors to utilise it without having to use Adobe’s software.

DNG conversion appears to be faithful, but to mitigate the risks involved in file format conversion I back up the camera RAW files prior to conversion. A secondary reason for retaining the original RAW files is to enable use of new software that appears from time to time that requires the original RAW format to work correctly. Examples would include updates to Canon’s RAW utilities to reduce noise reduction.

Need to be able to ensure my images are available in the future

This requirement both crosses over into the previous requirement and dictates that I need rigorous backup of the images. To ensure my images are available I have to consider:

  • Protect from software change
  • Not be locked into any one software vendor
  • Not be locked into any one operating system
  • Not be locked into any one camera platform
  • Protect from destruction by fire or other natural event
  • Protect from theft
  • Protect from hardware failure
  • Protect from software failure
  • Protect from myself (problem between brain and keyboard)

Use of DNG covers the first four points and my insurance and backup strategy covers the others. My backup strategy is a little anal (but does not go as far as some people do) and would work well in any enthusiast or studio setup where there is only one main photographer.

My desktop computer is my primary or master image store. I have it configured to use two drives together so that if either of them fails the data is intact on the other one and it is a simple matter of replacing the failed unit and the computer itself will copy the data to the new unit. This configuration is called Raid 1 (mirror) and is managed in hardware.

During ingestion the images are backed up, converted to DNG then copied to a working area.

Post ingestion I backup my images to both a Linux based file server and to offline portable drives. Note if you’re not a computer geek you could purchase network based file storage or use an additional portable drive in place of the server. The point is to have your master and THREE backups to cover you from various problems that could occur. I’m not going to go into why three here as there are plenty of articles on the internet you could read.

The portable drives are kept offsite. I have two that I use in rotation and perform a full backup weekly. At least one of the drives is always offsite.

 

Need to be able to manage my images both in a master repository and on the move

In its simplest form I want to be able to access my images at home or at work. I also travel around and often want my images to be with me. With 1.2 terabytes of images this is not really practical – even with the affordable drive storage available today.

However, what is practical is the use of cataloguing software in which thumbnails are stored. This same cataloguing software has other uses that meet other requirements.

I currently use Adobe Lightroom but keep an eye on the improvements in other available software. Don’t get too wed to any one software tool, just make sure your choice has the ability to export and import in common formats and you’ll be ok with any choice.

Need to keep a backup of my images as they came from the camera before I work on them

ii

This requirement is met by the backup strategy of keeping the virgins, both of the camera RAW and of the converted DNG. Keeping both means that I can edit my DNG files to my heart’s content and do not have to be concerned if I use an editor that is not parametric (in other words it changes the actual image file) because I always have the originals to go back to if I need to (and I have needed to from time to time).

Keeping all these backups would be onerous if done by hand, so I use Image Ingestor II to manage this process (along with metadata).

Need to be able to find my images easily and quickly

It should be noted that I now use Lightroom 3 rather than EM2. EM2 became too limiting especially once I started to bring in the last ten years of digital images and scans of 25 years of prints. EM2 simply could not cope.

There are people working on being able to search for an image by describing the image or by giving the search tool a similar image. These tools are a way off. In the meantime text based searching is all we have. But where is the text in an image? That is where metadata comes in.

Metadata is the secret to finding anything when you have a lot of things. Metadata is defined as being data about data, in my case data about images, when, what and how they were taken. There are four sources of metadata that I care about in my quest to quickly find images and there is other metadata that is useful during image use.

  • Metadata recorded by the camera with the image – e.g. the camera records details such as the date and time, the camera settings, the file name, and sometimes the attachments used
  • Metadata recorded during the ingestion process by the computer – e.g. File dates and file location
  • Metadata recorded in bulk against a set of images by me – e.g. import of a GPS log to geotag, bulk metadata such as the location, the event or activity being recorded
  • Metadata recorded both individually and in small groups by me – e.g. metadata particular to the image such as people’s names, a description of the image, a rating (out of 5)

I use Light Room 3 for these tasks. I keep a single catalogue for all my images. I have to keep my computing hardware up to date to manage this successfully. 

Need to be able associate metadata with the images to interpret them for the future

After searching, the next use of metadata for me is to interpret the image for the future. This usually means recording the names of the people and places in the image. It can also mean describing the image so that you know what it is later in life. As an example, if you take a series of abstract macro photos and want to refine or reproduce some element of them again in a few years’ time then you need to know that the source of the yellow abstract was in fact a daffodil.

Need to be able to surface my best work

Another (and final in my workflow) use for metadata is to make it simple to locate your best work on a particular subject when you need to. You might be searching for an image to enter into a competition or you might be fulfilling a family request for an image from an event. In my case it also supports my www.steamengine.com.au website so that I never need to purchase stock images from someone else to go with my articles.

The ability to locate your best work from an image stream cuts down on the amount of time you spend on images in post processing. Why waste time on poor images that you may never use. If it comes about that you do use one you have not lost anything but not immediately post processing – just do it later.

To achieve both, I use a system of ratings – most cataloguing software supports ratings, whether it be numeric or by stars. For me:

Star Rating

Meaning

1star

The image has been rated – this is the minimum for any of my images

2star

The image is mediocre but useable

3star

The image is good – the composition, lighting and subject are ok

4star

The image is exceptional – the composition, lighting and subject are all great

5star

The image is amazing – everything is just right – it could not be better

Generally, out of 100 images at my current level of proficiency I expect that:

Star Rating

Percentage

Reason

1star

5

Mistakes, accidental shoots, wrong settings, equipment failures

2star

40

A lot of images fall into this, these are what the average amateur would describe as a snapshot

3star

50

These are the images where you’ve taken a degree of care and are better than snapshot but are not exceptional

4star

4

These are the ones that are better than most of your images – they are the stretch of your abilities

5star

1

Didn’t I say it has to be “amazing” – it has to make you catch your breath

Why do I have the bulk of my images falling into two and three stars - am I that poor a photographer? No I’m not but I would be kidding myself if I said most of my shots were exceptional or amazing. Even top notch professionals have plenty of mediocre photographs. I will do detailed post processing putting possibly hours into perfecting a four or five image. I put in a few minutes on the others and nothing on the 1’s. Some of my images don’t even make it to 1’s. If during rating I think an image is so bad that it would never have a use then out it goes. Remember this is not gone – I still have the virgins hiding in the wings if I ever really need it back. It is probably worth nothing that the post processing could promote an image in the stars rating. If I think it is deserving after PP up it goes - it could also go down when I discover it isn't as sharp as I initially thought. There are some things you don't see in thumbnails!

Need to support post processing of my images

I choose to post process all the images that get two stars or better. I won’t spend much time on two’s, each three gets a little more individual attention and the fours and fives get much more.

I keep the working copy and manage all of the post processing on that copy – either through scripts or manually. The better the rating the more manual the post processing gets.

This is where my workflow departs from the really anal retentive ones; they will talk about derivative images and will take great pains to not modify the original working copy making only parametric changes. This means the original copy stays true and that the post production is a set of adjustments to be applied to the image and not an image in its own right. This work method does not suit my practices. It is rare for me to generate a true derivative that I would keep alongside the original – not just as an adjustment. When I do, that new image is manually copied into the virgin backup manually. I also cannot see the point of keeping the working copy true when you have at least two copies of it in the virgin backup anyway.

Need to support publication of my images

I publish a great many of my images on my website www.steamengine.com.au and more trivially on community pages like Facebook.

My images are huge. I’m not going to put huge images up on the internet – even if you want to, most upload image systems resize it on the way – all you do is waste your bandwidth.

To publish my images I resize them, (generally) strip the meta data, apply a watermark (I’ve had images stolen and used commercially – the watermark IS necessary) and save them as an 80-90 quality JPEG depending on the use. Remember when I said I don’t make derivative works? Ok, so I lied. Generally speaking I do NOT keep these derivatives which are why I don’t consider them to be derivative works.  Once they are uploaded to whatever gallery they are going into I consider that they’re no longer managed.

I’ve been publishing on the internet and in magazines for many years and while I’ve had occasion to replace lost images from the website after a server corruption the workflow to create them is automated and quick. Having to do some rework also forces me to consider what I’m publishing – is it still current – do I have better images of the same object/activity/event? Could I do a better job post processing them again from the virgins now that my skills have improved? And of course… is it actually worth my time to republish? Some articles on my site are a given – yes, I’d always replace them and republish but the little news blurbs that are only useful for a couple of months (e.g. locomotive G42 returns to service after an overhaul) are simply not momentous enough to keep!

Need to support delivery of my images to other people

From time to time I deliver images to other people for various uses and these images I do want to keep a copy of. I normally deliver them on write once media so I will burn two of them – one for the delivery and one for me to keep as a record of what I gave them. My cataloguing software needs to support to copying a selected set of images to the archive\delivery folder then burning them to disc. On the same disc I will burn a copy of a boiler plate license agreement that I’ve filled out for their use. I also make a 2nd copy of the burnt disc. This 2nd read only copy is very important as a read only record will stand better stead in court than one you can easily change. None of us ever want to attend court over Copyright.

Examples of the situations where I will give out images:

  • My model wants a copy – if I’m photographing a human I will always give them a copy of the images for their own use (e.g. their portfolio or on social networking sites) if they ask. This could be a professional model or just someone I’ve casually photographed.
  • I’m photographing an event run by a not for profit organisation (e.g. a preserved railway) and they ask for a copy of the images.
  • Someone sees me taking an image or images and wants a copy – e.g. the property owner or someone with a related interest.

To reduce the hate mail I’ll get from professional photographers who charge for their images: I’m not reducing the work available to you by handing out images. Puffing Billy Railway for example, has a professional photographer who they engage for their major events but they simply could not afford to engage someone to work on the railway every single day capturing every little moment. They depend on people like me. They may never use my images or they may use them in a book supporting the railway. This is one of the ways I support them.

My workflow does not need to support

When you read about workflows you’ll discover that a lot of them talk about archiving the working copy of an image. I don’t do this – I keep all my images online (both the master and the backups) and I expect that this will not change. Too many people fail to recognise that storage is becoming cheaper and more reliable over time. My first hard disk was 2MB and that was only about 25 years ago!

Why put yourself through hours of effort and pain to burn media and remove images from your master collection into a secondary collection with the risks inherent in doing so when you can simply buy more storage? Both the Windows and Macintosh platforms make it simple to move the entire system to larger storage when you need to.

If you do choose to do this then you have several considerations

  • Digital write once media (e.g. DVD and BluRay) have an unknown life span – even the archival quality ones may only last a few years. They can claim anything they like, but the extent of their warranty remedy is to give you another disc if they’re still in business when you make your claim. But what about your images – they’re gone!
  • Moving files from one location to another is dangerous. All sorts of failures in hardware and sometimes even those found between your brain and your fingers can cause you to lose the files or store them someplace you would not expect to find them. This means you need a tool to verify that the move is complete. They usually work by doing a copy, verify then delete.

I don’t require support in the workflow for Copyright. In Australia, all works are automatically copyright to the author. You do not need to register them. It is advisable to place a copyright notice and use instructions in the metadata or within a watermark on the image when publishing – this is covered during my publishing. If you live in a country where it is advisable to register your images to ensure that you do hold the copyright then you should include this step in your workflow.

Backup

In my main photo workflow computer I have two RAID 1 (Mirror) arrays. One set is 6GB/S SATA 3 drives. This is my main working repository. My ingestion, archive storage and working storage are on this raid set. The second set is 3GB/S Sata 2 drives. This set has my primary backups, my temporary image storage (during publication) and MS Windows itself. For insurance purposes I have a pair of USB 3 cases with (currently) 2TB drives in each. I keep one off site away from my workflow computer and the other is connected. At the end of every major session I backup my catalogue using LR and my images using Microsoft SyncToy to both the internal backup and the external backup drive. Six monthly I format and do a complete copy. Roughly once a month I swap the external drives - ensuring the offisite copy is always reasonably up to date. If I do a big editing session or work on images particularly important to me I will swap the drives sooner. When I swap I take the onsite drive off site and bring the other one back so I am never in a situation where all copies of my images are in the same place. As my catalogue grows I ensure the storage keeps pace. Storage is cheap. At approximately AU$50/TB you can afford to have plenty.

When in the field on an extended trip I use a netbook and pair of USB 2 portable drives for the backups. The primary copy is on the netbook and this is synced to both of the portable drives. If I need to ensure a copy is safe I will send it home.

Tools

I use a number of tools in my workflow. The tool, its URL and what I use it for are documented in the table. This was a full set at the time of writing, but I am always evaluating new tools and adopt them when they both do something cool and are affordable. I often use old versions of tools because they are more affordable than the current versions and you can often buy old stock at greatly reduced prices – as an example I purchased CS3 for $100 in a legit unopened unregistered package from old stock that was being disposed. This is a far cry from the $$$ Adobe want from CS5 and the functionality simply isn’t that much different. I keep an eye out on the disposal bins at software shops and the clearance section of software websites.

Tool

What I use it for

Adobe Bridge

I really only use this for browsing folders directly and for checking what Metadata other tools are sticking into the files and that metadata isn’t becoming corrupted.

Adobe Camera Raw 4.6 and 6.2 updater

Interpret Canon RAW images for use within CS3 and Bridge

http://www.adobe.com/support/downloads/detail.jsp?ftpID=4040

Note to install plugins for Photoshop geared to current versions (i.e. CS5) all you have to do is download them and drop them into the plugins folder directly and they seem to work fine.

Bridge C:\Program Files (x86)\Adobe\Adobe Bridge CS3\Plug-Ins

Photoshop C:\Program Files (x86)\Adobe\Adobe Photoshop CS3\Plug-Ins\File Formats

These are free plugins providing you have a licensed copy of Photoshop.

Adobe CS3

My photo editing software of choice. It does everything I need.

Adobe DNG Converter

Converts from various camera raw plugins to DNG. Free from http://www.adobe.com/products/dng/index.html

Download it and place into C:\Program Files (x86)\Adobe

Image Ingester II

This is great software allowing me to automate the ingestion process, the backups, the dng conversion, the application of metadata of my choice, the folder structure of my choice. It certainly doesn’t look pretty but it performs fantastically.

http://basepath.com/ImageIngester/

Adobe Lightroom

I use this software to catalogue the images, apply individual metadata and to rate the images as well as searching and production of delivery discs and via extensions uploading to social media sites such as Google+, Facebook, etc..  This is my workflow home for image manipulation.

Fast Picture Viewer Codec Pack

This is a very cheap pack of codecs that understand loads of the camera RAW formats and make them natively viewable within Windows. Without this I could not see my raw files as thumbnails in Windows, Bridge, etc. http://www.fastpictureviewer.com/codecs/

Note for those with 32 bit Windows, ignore the (x86) in the paths above.

I used to have a lot of other tools that I would use, but they have all been supplanted by native Windows functionality so I simply don’t need them anymore.


 

Folder Structure

I recently changed the location where I keep my images – I use the Windows platform and used to keep my images at the root of a drive dedicated to them. Now that Windows has improved the embedded handing of images and has improved the built in tools to transfer user data from one computer to another during a windows refresh or upgrade or hardware upgrade I have moved them to the Pictures library. Currently this is in Libraries\Pictures or in actuality C:\Users\paul\Pictures\.

To differentiate between images that have appeared on my workstation where I am not the photographer (e.g. someone else sending me images to publish on www.steamengine.com.au) my root folder for MY work is C:\Users\paul\Pictures\photography. Under that I have the structure that supports my workflow.

 folders

Most of the structure is self-explanatory, there are two folders that have uses outside the general workflow. Printing and scratch.

Printing: I have occasion to use special purpose printing software that uploads photos to a lab bureaux printing service. I don’t consider this a delivery and only keep the images until I get satisfactory prints back then I delete them.

Scratch: This is exactly that, a scratch pad. If I’m playing with a software package such as a trial version of something then I put the images I’m working on here. This minimises the danger to the rest of the working set from untried software and makes it clear to me that the images don’t need to be retained. I don’t consider it fair to publish something from a trial version of software as it is not rewarding the author.

Caveat Emptor – Choose your own path

Please don’t implement your workflow based on my say so – it works fine for me, or at least it does today. I’m sure I’ll change it again. Change is constant. Change is one of the reasons I wrote this document – while it will help other people to make choices, it also helps me to remember why I made particular choices to help me evaluate if a change to my workflow will be beneficial.

If you follow my workflow and lose images then all you will get from me is sympathy! Don’t come chasing me because you followed my practices verbatim without first choosing if they meet your needs. When you implement a workflow make it yours. Put your own stamp on it.