The Lightroom Catalog

Hello, everyone and welcome to another Lightroom Tutorial Podcast. I'm George Jardine, pro photography evangelist from Adobe Systems. As people are setting out to learn Lightroom, the questions I hear over and over again are: Where are my pictures? Just what am I looking at here? Lots of us are having a blast with the new capabilities. But then again, lots of us are still wondering: What in the heck is going on with this new Catalog Model? So in this tutorial, I'm going to try to illustrate some of the basic concepts that Lightroom uses to store, process, and output your photos. At the same time, I'm going to try to give you a glimpse into how the Lightroom Catalog concept is different than the File Browser model that's used by Bridge. Each model has its own strengths, for sure, but they are very different-- and it's useful to understand those differences. So starting with that first question, the true location of your photos, is actually very easy. Any time you want to know precisely where a specific photo is on you computer, all you have to do is hold down the Control key on a MAC, click, and choose Show in Finder from this menu. On Windows, you just right click and choose Show in Explorer. Lightroom will tell your computer to pop that Finder or Explorer window and it will select the actual file for you--and there it is, right there. It's as simple as that. Okay--back to Lightroom. I can also do the very same thing with the folder entries over here in the Folders panel. Each entry in this list corresponds to just one folder somewhere in my file system. And if I want to see that folder in the operating system, I can just Control, click; and again, choose Show in Finder and my computer will pop that window in the Finder and highlight that folder for me. And just like any other MAC folder window, if I want to see the complete path to it I just hold down the Command key this time and click up here on the folder name in the Title bar-- and I'll see the actual path to this folder. So here you can see that this particular folder is on an external hard drive named "Demo Library." Okay, that's easy enough--back to Lightroom. So what's the answer to the question: Exactly where are my photos? The answer is, they can be just about anywhere. And this is a luxury you should be taking advantage of-- Lightroom can access photos that you've already copied to your internal hard drive or it can access them from just about any device you might attach to your computer-- like an external hard drive or a CD or DVD. It can even access photos that might be out on your network somewhere. So the answer is: your photos can be anywhere. You do import them into a catalog, but don't let that word "import" confuse you. A better phrase would probably be to "reference" them or to "link" them to a catalog. When I click the Import button and navigate to some photos on a hard drive somewhere-- like this--and then click the Choose button, Lightroom will throw out this dialogue with a few choices. For the purposes of this tutorial, there is just one important option and that's this one right here--the File Handling choice. Unless you're importing from a camera or from a card reader, this will be the default choice: Import photos at their current location. Hmm--I agree, this seems a bit like improper English but it's worded this way just to drive home the point that Lightroom is going to leave your photos right where they are and not touch them. So when you import, you're not really moving your photos anywhere-- unless you specifically tell Lightroom to do so by choosing this option to move your photos to a new location.. If you're importing photos directly from a camera memory card your only option will be to copy them to a new location somewhere. But again, after Lightroom is done copying them, it will then still just be pointing to them as a source of information. The original files never really become part of the catalog. And so when you click Import, Lightroom starts reading the files-- and this is important--it reads the files. As we just learned, it doesn't move them and as Lightroom's importing them, it doesn't change them either--it just reads them. You'll hear a lot about Lightroom's nondestructive work flow, and this is a key part of it-- except for a couple of very minor exceptions that we'll go over later, Lightroom never modifies the photos that you are working with. It's always completely nondestructive. Okay, if it's not really moving your photos around when you do an import then what's it doing? Good question--Lightroom basically looks for three things when it reads a raw fille and, by the way--just a little side bar here-- most of what we're going to be talking about for the rest of this tutorial has to do with Camera Raw files. There are a few minor differences when working with RGB files, but not that many. I'll try to point them out later. Lightroom is looking for three things in your raw files: A preview, some metadata about the photo, and the raw image data itself. And as Lightroom is reading all this data from your files, it's storing its own version of most of it. Let's take these three things, one at a time. First is the Preview. When you're shooting raw format in your camera, the camera goes ahead and makes an RGB version of the photo, whether you ask it to or not. And then it generally applies a bunch of JPEG compression to this RGB file and stuffs it into the raw file, as it's writing the picture to the card. This RGB version of the photo is the JPEG preview that is displayed on the back of the camera, and it's the preview that you'll see for a moment when you first import a raw photo into LIghtroom. I've often said that there's no free lunch, when it comes to raw processing. Raw files do take a bit of processing time. So, while LIghtroom is busy decoding the raw image to create and cache a preview of its own for you, it temporarily shows you this embedded JPEG preview. That's why you might see a slight color or contrast change occur a few seconds after you import a raw file, when Lightroom swaps out this embedded preview for a preview that's been generated by the Lightroom application itself. Okay, so the camera-embedded preview is the first thing Lightroom reads from your raw file, and then it's disposed of pretty rapidly. The camera-generated preview is really not all that useful to us once we have the real image data, so we basically just toss it. On to the second thing Lightroom reads--the metadata. During the import process, Lightroom reads all the metadata it can from your camera's raw file. This metadata contains all sorts of information that's recorded by the camera at the moment of exposure. The really useful stuff for you, as a photographer, are the Xof entries that describe the camera's settings, the camera type, the lens type, and so on. Lightroom basically takes this info and stores it in the catalog, and once it's in there, the metadata gives Lightroom the ability to help you find your photographs in a lot of really cool ways. A great example of that would be the metadata browser right here in the Lightroom Library. The metadata browser lets you sort your entire catalog by lens type or by camera type--or even by camera serial number. Being able to search across your entire catalog, no matter where your original files might be, is one of the key benefits of the catalog. And we'll look at that in a little more detail later. Other bits of this metadata that Lightroom is reading from your raw files-- such as the white balance of them--can also help Lightroom properly decode the raw data iIn order to create an RGB version for you. So that's next--the raw image data. And this is what Lightroom is all about-- the rest is basically just a bunch of housekeeping. The picture data is what we're here for and Lightroom uses this data in two different ways. First, it takes the raw picture data and runs it through the raw processing engine to create an RBG rendering of your photo. We frequently call this rendering a Preview. Once Lightroom has the photo in an RGB space, it can then push it up to your display and it can also, then, cache various sizes of this Preview in the library-- which it does for things like thumbnails in the grid and in the filmstrip, and for the loop views in the library, and so on. It caches these previews to make it really fast and easy for you to work with your pictures. So Lightroom reads the raw picture data and then processes it in order to show it to you on your display. It's important to understand that Lightroom must process the raw data to store it or to display it in an RGB space at all. And this is where the nondestructive part comes in again because decoding the raw data from your camera is sort of a one way street. Once you've processed the mosaiced raw file and encoded it into an RGB space, you really don't want to push those changes back into the raw file. The RGB previews are useful for helping you look at your photos so that you can manage them, make further processing decisions, and so on. But they inherently contain much less information than the raw data did originally. Think of your raw file as your negative. Just like the film world, your negative has almost unlimited potential for the various interpretations you may wish to create from it. But saving an RGB rendering in favor of the raw data is no different than keeping your prints and throwing away your negatives. So again--Lightroom reads your raw files, processes them, and stores those previews in the Lightroom Catalog, for you to work with. And that's about it. It never changes your original files. Okay--now that we have that much straight, let's talk a bit more about the Lightroom Catalog. When you first start Lightroom on your MAC, it creates a folder right here in your Pictures folder, and it simply names this folder: Lightroom. This whole folder makes up one catalog. On Windows, the very same Lightoom folder will be created in your My Pictures folder. Your Pictures folder is the default location for this first catalog but you can build a new catalog almost anywhere, and you can have as many catalogs as you want. Lightroom is very flexible that way. But we're going to talk about all that in a minute. Right now I just want to peek in at this folder for a minute and talk about what's inside. Open it up, and you'll find two files: the catalog.lrcat file and the previews.lrdata file The Previews file is where Lightroom stores those RGB previews we've been talking about and the Catalog file is where it stores all the other information about your photos-- the metadata. Okay--now that we have the basic building blocks in place, let's take a look at the flow of things. It starts with your computer. It doesn't matter if it's a MAC or a PC, it has a file system with a Pictures folder. And as we mentioned before, when you first start Lightroom it creates its default catalog in your Pictures folder. Next, comes your photos. Now--again, it doesn't matter where your photos are. They might be out on your network somewhere or they might be on a RAID or they might be on a portable external hard drive. When you import those photos into Lightroom, a link to them is created-- and nothing more. Lightroom just references those photos, so to speak. It reads the metadata, as we talked about, and it stores it in the catalog. Lightroom also reads and processes the raw image data and then it stores that processed image data as Previews in the Preview cache. The important thing to understand here is that Lightroom creates links in the catalog that point to the originals. The originals are never really part of the catalog-- or associated with it in any other way. They're just pointed to as a source for raw data. Okay--we've seen how your originals can be stored almost anywhere but the same is also true for your Lightroom Catalog. Here we're showing it in the default location. But if you pull down the File menu in the library, you'll see that you can create a new catalog or open an existing catalog. We even built a list for you to choose from of the catalogs you've opened or created recently. If you choose to create a new catalog, you can put it in the same location as your source files, which is what I like to do. For Version I Catalogs this means just about anywhere, except on Network storage. But in any case, the end result is the same. The catalog simply links to the locations of the source files. The reason I like to put a catalog on the same disc as the source files that I import into it is because then I know those photos will never be offline. Lightroom will always be able to find them.. Even if I can't always work this way, the Lightroom Catalog is pretty flexible and deals with offline photos pretty well. If your originals are offline for some reason you can still open up the Lightroom Catalog, view your photos, and work with your key words and sorting, an so on. Most functions downstream from the library, such as the Develop Module, will be disabled if your photos are offline, but at least you can open the catalog and view them. At this point, the previews are coming right out of the Preview cache, and you can't modify them or generate New Previews as long as the source files aren't there. But for many photo work flows, this is still a key advantage of the catalog over a file browser where you can't view your photos at all if they're offline. Okay--next, let's take a look at what happens when you make changes to your raw photos. We just saw that Lightroom processes your raw photo in order to show you a preview and here in the Develop module we have a lot of really cool controls to change the basic settings or parameters that Lightroom uses during that processing. But remember, we also said that Lightroom is completely nondestructive. This means we never change a source file that you imported into Lightroom. So as I make changes to this raw photo, for instance, these processing settings are stored in the Lightroom Catalog. I'm not actually changing the raw file, but only the settings that are used during the conversion from raw to RGB. So how do I know that I'm not really changing my original? Well, there are a couple of interesting points here. First, Lightroom always caches the default rendering of a raw file in the catalog when you import it. so you can always very easily go back to the original and see what changes you've made. You'll find the default rendering of your photo right here in the History panel. It's the import state. And you can always get back to the default state by clicking the Reset button-- right here in the Develop module. Okay--what other evidence do I have? Well, let's take a look at it from another angle. First I'm going to undo that reset, by hitting Command + Z, which will take me back to the corrected state. Then I'll go up to my Photo menu and I'll choose Show in Finder or hit Command R. This reveals the actual raw file in the finder. Note that it still has the original 2006 shoot date here in the Finder. Lightroom is treating it as a Read Only file because it's a camera-generated raw file. Now, if I take that raw file and drag it onto the Photoshop icon--like this-- Photoshop will open it up in Camera Raw, using the default. So we're back to the dark rendering, which tells us two things: One, it confirms absolutely that the file has not been changed and two, as I move back over here to Lightroom, it confirms that the changes we made in Lightroom are stored in metadata--in the Catalog. That metadata was not stored in the file system where Photoshop would be able to see it. And the same is true for all the other metadata you might apply in the Lightroom apply in the Lightroom library as well. So key words:Copyright information, captions-- all of this metadata is just stored in the Lightroom Catalog and is not yet accessible to other programs. This is the default behavior for Lightroom. All those settings and key words and so forth are stored in the Catalog until you explicitly choose to make it accessible to other programs by pushing it back out to the file system, in the form of XMP metadata. We'll look at exactly how you tell Lightroom to do that in just a minute. But first, let's take a quick look at where Lightroom actually writes the XMP. In the case of your Camera Raw files, Lightroom will not write anything directly into the Camera Raw file because we're treating the raw files as Read Only files--remember? Instead, Lightroom writes XMP metadata into a sidecar file for each picture. These sidecar files are just small text files that contain the XMP metadata and we write them directly to the same folder that contains the Camera Raw file. In the case of DNG files, and for RGB files like TIFF, JPEG and PSD, we write the XMP metadata directly into the file. Okay--there are two simple methods for pushing your Lightroom metadata out to XMP. One is automatic, the other's manual. If you don't want to have to think about it, and you simply want Lightroom to always write out all of your changes to XMP, then just set this preference. It's not really a global preference but rather, it's a catalog setting. So just go up to your File menu and choose Catalog Settings and then--over here in the metadata tab-- turn this one on right here: Automatically write changes to XMP, and then close the dialogue. Now I'm not going to turn this option on, myself, because I prefer to write out my XMP manually, and only when I need to. Besides, there's a certain performance penalty you pay if you leave this baby turned on--at least on everything but the very fastest systems. Now, if you want to write your changes out manually, it's very easy--it's just like Saving. What do you do in any other program when you want to save your changes? You hit Command + S, right? Same with Lightroom. As you go along, Lightroom is always saving your settings, your key words, and other metadata automatically--right into the Catalog. In other words, if I quite Lightroom right now and reopen it later the exposure changes I've made to this photo of Helga will be right there--just as I left them. And the history of every move I've made in the Develop module is also saved here in the History panel. So in that sense, there's no need to save the Catalog or anything--it's all automatic. But if I have this photo of Helga selected and I hit Command + S or, from my Photo menu, choose Save metadata to file, Lightroom will write out the XMP to the file system. So I'll do that and then I'll just choose, Show in Finder once again, and now--here in the Finder, right next to the raw file, we see the new XMP sidecar file. Note that the XMP sidecar has the very same name as the photo file except for the extension, which is now .xmp. Also notice that it's tiny--just a few kilobytes. Besides your key words and other descriptive metadata, it contains the processing instructions for the Adobe Camera Raw engine which is used in both Lightroom and Camera Raw plugin for Photoshop and Bridge. It's the same code in each case which means, if I now take this same raw file and drag it onto Photoshop once again, this time Camera Raw will read the XMP sidecar file and will pick up the Lightroom settings that are in it. As the Camera Raw plugin pops open, notice that the color balance and exposure changes that I previously made in Lightroom are now being honored in Camera Raw. Nice, huh? Now the next question you should be asking is: Does it go the other way? And the answer is yes. Just to make a fairly obvious example, if I were to surf over here to the HSL Panel and convert this photo to black and white and then hit the Done button, Camera Raw would write those changes back into the XMP sidecar file. You see, Camera Raw doesn't rely on a catalog like Lightroom does. By default, Camera Raw looks to the file system for the latest settings for any photo. Okay, back to Lightroom. Ahh--the photo is still in color, and that makes perfect sense. Remember we said that Lightroom's default behavior is to read and write its settings to the catalog. We had to force Lightroom to write those changes out to the file system so that Camera Raw could see them--and the same thing is true going the other direction. LIghtroom will always assume that the metadata in the catalog is up to date and it will use that unless we tell it otherwise. We call that model, "the truth is in the database" or in the catalog. Camera Raw, on the other hand, always relies on what it find in the file system. We call that model, "the truth is in the file" and these two models are fundamentally different. We don't have all day here to talk about the difference between file browsers and databases, but the bottom line is that each model has its place in a professional work flow. Camera Raw relies on the file system and Lightroom looks to the catalog. So back to the problem at hand-- how do we get Lightroom to read the settings from the file system? Easy--just go back up to that menu and choose Read metadata from file this time. In the library, it's under the Metadata menu. And now Lightroom matches the black and white interpretation that we created in Camera Raw--cool!. And so, Lightroom and Camera Raw will always create identical interpretations of your raw file--as long as they are both synchronized to the XMP metadata. Cool, huh?--Okay, we're almost done. The last thing I wanted to touch on is the subject of output. And this is because, after showing all this nondestructive catalog-based work flow stuff to larger groups, I almost always get the question: So how do I get my changes out of Lightroom? And the answer is that just about any output from Lightroom bakes in the changes you've made. So when you create a web gallery, for instance, the JPEGs that Lightroom creates will all be fully processed and will contain your copyright or any other metadata that you've applied to the photos. They'll be perfectly ordinary JPEGs that can be viewed in an other program or browser. In the case of exporting photos from Lightroom, if you choose one of these RGB options--JPEG, PSD, or TIFF-- then, of course, LIghtroom's going to bake in all of your settings. And again, the resulting files will be completely normal RGB files that can be used in any other program. If you choose to export DNG your processing changes are not baked into the pixels, but a representative preview is made, and the Camera Raw settings are written right into the file. If you choose to export originals, we're back to the previous behavior meaning, if the original is an RBG or DNG file Lightroom writes the settings directly into the newly created export files. If the original is a Camera Raw file, Lightroom simply duplicates the file and writes the settings into a sidecar. So that's it. I hope this tutorial has helped answer some of the basic questions we asked at the beginning: Where are my files? What am I looking at? We've glossed over some of the intricacies of output--I admit it. But, honestly--I think that stuff is better saved for another tutorial later. So until then, please feel free to send your comments and suggestions to me at: GeorgeJ@adobe.com And always remember Rule No. 5.