OpenRAW

Uwe, you speak for many of us when you say: "Letting the raw converter designers work on the decryption of all existing raw file formats is such a waste of their talent! I want them to compete on image quality and workflow, not on hacking".

I believe that some of those products use Dave Coffin's DCRAW, so the problem, while bad enough, may not be as bad as it could be. Dave does the heroic job of reverse-engineering on behalf of some others.

The target is surely to have a single documented standard, (or at most a small set of standards), that all conform to. Whether it is DNG or something else is a different question - but at least DNG is a good "proof of concept". (As long as I can convert my DNG files to whatever the target standard is, I will be happy!)

But what happens if somebody decides to go after Dave Coffin with a large scale lawsuit, and/or perhaps criminal charges for violating the DCMA? His efforts would cease in a heartbeat because he doesn't have the wherewithal defend himself in such a situation.

The it is in our best interest, should that happen to collectively violate the DMCA and render it unenforceable.

Uwe, exactly my thoughts. I use third party software in my workflow (raw conversion, archiving) and I think that the same "reverse engineering" is done in everyone of them. Besides, what happens to my archived NEFs after 10, 20 years? What software is able to read them?

You speak as if a RAW format image has to archived as a RAW image -- why? Surely the point of a RAW image is to post process it to give it a natural appearance with correct white-balance etc., and then to save those changes in a more 'open' universal, non-lossy format such as TIFF. So any concerns about being able to access the file in the future are unfounded. Converting RAW to TIFF after post-processing is something we all have to do anyway. It isn't carved in stone that a RAW file must be archived for eternity once it has served it's initial purpose.

It is very valuable to archive the original raw image data. Then this can be reworked at a later date, perhaps using new interpretation, new skills, new tools, etc, to achieve even better, or at least different, results.

DNG is the best known way of achieving this. It is a genuine raw format, containing the original raw image data, but far more likely to exist as a file format into the future.

That TIFF may be a useful intermediate step. It may not be the long-term end result from that original raw image. It is probably not a good idea to omit archiving the original raw image data. DNG is good for that purpose.

It is totally irrelevant that SOME people may be content to discard the original raw image. It is absolutely clear that very many people don't want to do that, and so a solution HAS to be found. DNG.

I enjoy being able to revisit and reinterpret raw files, so I disagree with Philip Corcoran's comments above. It is analogous to reinterpreting an old film negative. If we throw out our digital "negatives" after processing them to Tiffs (or Jpgs), or can no longer access them because the raw converter is no longer available, we are left with only the initial interpretation, if any at all. We may not decide to "develop" a particular image until many years later. The raw file should still be accessible then. Photographers like Ansel Adams have made different interpretations of the same negatives over the course of many years. It should be the same with raw files in the future.

Unfortunately, saving a document in .dng, .tif w/ sidecards, &c. is that image improvement software can only reach its fullest potential if it has access to the truly raw data. Any conversion, even to .dng or .tif w/ sidecar always loses context/information.

So zero-conversion of the data is required. The image improvement software also needs to have well characterized information of the camera, lenses, exif, distance, .... So the goal of OpenRaw is necessary, but not sufficient to achieve an archivist's goals.

OpenRaw's tasks is big enough, so I don't think they can/will take on the whole problem. But we also need to create public camera & lens analysis algorithms & software; and a repository to store the results of said analysis.

At the end of the day, manufacturers of hardware and software can achieve success in the marketplace by providing the best analysis, user interface, correction and adaptation software. Some of this will be proprietary, some of this will also be within the camera itself. Hasselblad's digital APO, Dxo's software are commercial and proprietary, potentially worthy of patents. But a lot of the methods will originally be done as phd dissertations, and even if it is patented some/all of the techniques will be published into the public domain.

People who are motivated by profit can have their cake (for a few years), and the needs of the archivist's are still met. So the manufacturers still can reach competitive advantage with innovation, without precluding our needs. Essentially they succeed on their merits, not on their ability to obfuscate, lobby and beat up others in a court of law. In other words win by being competitive, not nasty with a big war chest. If you read the US constitution, this was the purpose of patents in the first place.

David, can you supply evidence for "Any conversion, even to .dng or .tif w/ sidecar always loses context/information"?

Adobe claim that for certain cameras, nothing is lost when converting to DNG. (And the situation improves release by release). Do you have evidence to the contrary?

What does Adobe is let you embed the original file in the .DNG. From a technical point of view you're not losing a single bit. But you're obviously still unable to extract all the metadata information as you were with the original file...

Fabrizio, Adobe claim that for certain cameras, nothing is lost when converting to DNG, even without embedding. They say this applies to NEFs, CR2s, PEFs, and some others.

In fact, they say that the reason why they sometimes lose data for other cameras is because they don't know the format of the files. If they had published specifications from the camera manufacturers, they could solve that problem. Or someone else could implement a DNG converter that did.

I believe David's statement "Any conversion, even to .dng or .tif w/ sidecar always loses context/information" is incorrect. There is no reason why it has to be the case.

Hi Barry. Adobe's statement puzzles me. I've being researching extensively with NEF, PEF, MRW, CRW, CR2, SRF and inside their makernotes there are lots of stuff whose meaning is not known. When you perform a DNG conversion, all that stuff is not copied into well-known DNG tags, but as a block of uninterpreted bytes in a specific section.

The point is that probably Adobe thinks that for the cited formats the 'unknown' metadata is not relevant to the final quality of the image. My basic question is: if they don't know what information is there, how they can be so sure that is not relevant?

PS Can you please point me to that Adobe sentence?

Fabrizio, it is copied by the DNG Converter as a block from Makernote to DNGPrivateData, for those cameras. It is therefore just as accessible afterwards as before.

ACR doesn't use it for raw conversion while it is in Makernote. Neither does it use it when it is in DNGPrivateData.

In fact, ACR has started to use some of it, but only marginally. For example, Pentax Makernote contains the lens-model, and Bridge metadata can now show this. It shows it for both PEFs and DNGs from those PEFs. But there have been problems, because the values are not properly documented, so it sometimes gets it wrong. That is the problem with trying to do something useful with undocumented data!

This illustrates an important point. The DNG Converter started to copy that data from Makernote to DNGPrivateData in 3.1. (That is why I started to discard my PEFs soon afterwards). Then it started to use the data, in either place, in 3.3 beta. So preserving the data in 3.1 has long-term value, even though there was no immediate benefit. That shows that the most important initial task is to preserve the data in the DNG file, which is why we need the best DNG conversion we can get. We can worry about what to do with it later.

Any raw converter that wants to use it while it is in Makernote can still use it when it is in DNGPrivateData. If a case can be made to have separate tags for some of the data, perhaps Adobe will add them to the DNG specification. But camera manufacturers tend to want to "do their own thing", and it may be hard to identify new common tags.

(If a camera manufacturer wants to use DNG, but continue to use Makernote, then they can ensure that it is in a safe format in Makernote, and use MakerNoteSafety).

Here is a link, but there are others too. I'm trying to find out more on this topic.
photoshopnews.com/2005/05/23/dng-workflow-part-i/

Ok, I do know that the DNG copies makernote as a block. I now understand what you means by saying that nothing is lost in the process - basically that you can convert all of your photos into DNG and throw away the original RAWs; but the point is while I could imagine that processing my NEFs with Nikon Capture Editor makes some good use of that metadata, even if I'm not aware of, for sure Adobe doesn't and can't do the same things with converted DNGs as is not able to understand it. I think that when David said "lose information" intended this fact. Ok, the bits are there but if a manufacturer drops a model and never discloses details about its format, you'll stay with a block of unknown bits forever, just as with the original RAW. That's why I say that DNG doesn't solve the problem at the root.

DNG combined with OpenRAW solves the problem.

OpenRAW will, hopefully, lead to an understanding of the content of the raw files from camera manufacturers.

Converting those raw files to DNG can then be a reliable process, because the contents will be fully known. And things puts in to special fields such as DNGPrivateData will be understood.

Without OpenRAW, we still have a partial solution in DNG, depending on the degree to which the writers of DNG converters understand the raw formats. Adobe's understanding ranges from "pretty good" for some, to "a bit lossy" for others. It is vastly better than nothing. It will become better if OpenRAW can cause formats to be published.

Adobe, of course, are just one company able to write DNG converters. Hasselblad-Imacon also have a DNG converter for some of their raw formats, and my guess is that they get it right. I don't know whether or not they publish their raw formats, but I expect the DNGs out of their DNG converters are very good indeed, perhaps perfect.

We want OpenRAW to help achieve the most comprehensive and best documented DNG files, and also to ensure that raw handling products support DNG properly. Then we will have solved the problem of the proliferation of raw formats.