OpenRAW

You have just described DNG!

DNG is an open raw file format. It was launched nearly two and a half years ago, and is currently supported in some way by more than 150 products from more than 140 companies. (This includes several cameras and digital backs. The number increases all the time). There are a number of DNG converters that convert from existing formats to DNG.

DNG supports lossless compression. I believe it can be read by UFRaw. (I've been using it successfully for years).

DNG has excellent archival characteristics. It is identified by the US Library of Congress as the preferred alternatives to other raw formats such as NEFs, CR2s, etc. You can read a lot about it here:
http://www.barrypearson.co.uk/articles/dng/

But it is a lossy format. So much for your lossless compression.

;)

J.

DNG is a lossless format

DNG files are capable of holding all the raw image data from cameras with CFA-based sensors, including those with offset sensors such as Fujifilm cameras. They are capable of holding all EXIF data, including Makernotes. They are capable of holding lots more, for example TIFF/EP metadata and XMP metadata.

Whether they actually hold all the data depends on how well the software used, (for example DNG converters, of which there are several), copies it across. The claim is that for NEFs, CR2s, PEFs, and probably some others, it is all copied across. Anyone claiming differently should provide suitable evidence, instead of exploiting "fear, uncertainty, doubt".

Two main types of problem with DNG conversion are known:

One problem is that the developer of the DNG converter doesn't know enough about the original raw file format to be able to copy all the data across. Typically, the raw image data is copied across, so high quality conversions can still be performed. ("Digital Image Preservation" is achieved). But some non-essential data may be missed, for example previews, or some information that isn't actually needed by most mainstream raw converters. (A problem with ORFs was identified with version 3.4 of the DNG Converter).

Another problem is that data that IS copied across into the DNG isn't understood, other than by the company that created the original raw file. As above, high quality conversions can still be performed without knowing this information. Examples include such added-value as "picture styles", "curves", perhaps lens-identification information, etc. In this respect, DNGs are obviously no worse than the original raw files, and since the rest of the DNG file is understand, and also the DNG Converter puts EXTRA data into the DNGs, DNGs are a much better archival format than the originals.

The solutions to both of these problems need better information about the original raw file formats. In the first case, that will result in better DNG converters being written. In the second case, simply documenting the Makernote automatically provides documentation for the copy of it in the DNGs. Since Adobe software and some other mainstream raw converters don't use that data anyway, solving these problems would actually make no difference to the resultant image.

DNGs generated from documented original raw files will be a better archival format than the documented original raw files themselves. This is for two main reasons:

First, the DNG files contain extra camera details so that future raw converters don't have to have camera details built in. In (say) 20 years time, while documentation of an original raw file format may enable a raw converter to READ the file, without being able to test a camera of that model it may be hard to determine how to PROCESS that data. Raw files, other than DNGs, typically don't contain all the necessary information for a high quality raw conversion, so merely documenting every detail of the raw file format is insufficient.

Second, DNG largely solves the problem of the proliferation of raw file formats and variations of these. Therefore, it is easier to achieve critical mass. One good implementation will support DNGs for images captured by many camera models, making it far more likely that a company will bother to implement it. Software companies support cameras according to their business case, not just because their raw file formats are documented.

When OpenRAW was launched, there were about 25 non-Adobe products that supported DNG in some way. Today there are more than 150, including several cameras and digital backs, plus some software products from some camera manufacturers. There is also more known about the contents of Makernotes, hence of copies of those in DNG files.

So you just confirmed its lossy ;)

Barry, nobody takes you seriously if you just sounds like a sales guy for DNG. You should just stop your crusade.

DNG beats all other raws

Let's clear. DNG is better than ANY other raw file format for archival purposes. The DNG specification provides a lossless raw file format. Anyone with evidence to the contrary should provide it, instead of resorting to "fear, uncertainty, and doubt" to further their own agenda.

Any losses are the result of policies of camera manufacturers, not the DNG specification itself. Major formats such as NEFs, CR2s, PEFs, are believed to be converted to DNG without loss, because they are understood well enough, and are well-formed enough, that the conversion can copy everything across. (Indeed, the DNG file contains a SUPERSET of the original information). Some other formats are not (yet) understood well enough. But at least the raw image data is copied across, which is what matters to serious photographers. Anyone with an obsession about every last detail of metadata, for example perhaps which lens was used, or perhaps the JPEG preview used in-camera, may not like this. Do they expect that information to improve their results?

If camera manufacturers don't publish their formats, DNG is better than the camera manufacturer's own format, because, except for those last details, the result is openly documented. (As in "Digital Image Preservation Through Open Documentation"). If the camera manufacturers DO publish their formats, then for the cases where all the data is copied across, we will understand what all that data means. And where it isn't, the DNG Converter can be improved, and the result will still be that DNG is the better format.

It is obvious that the problem isn't just about undocumented raw file formats, but also about their proliferation. The evidence is that software developers don't support formats just because they are documented, but rather based on the business case of supporting a particular camera. Therefore, mere documentation of vast numbers of raw file formats won't enable people to be sure of being able to process their images in the future with the tools of their choice. They may have to use older tools that can't get the best from the raw image data concerned.

What OpenRAW should have done from the start was tackle the "proliferation" problem as well, and promote the need for a common raw format. I supplied a useful definition of "common raw format" to OpenRAW that could have been used. It is interesting that OpenRAW used to say: "Many have suggested (and Adobe has created) a common, open file format for RAW image files for all camera makers to use as a solution to the RAW problem". Then the bit about "Adobe has created" was deleted. Was DNG getting in the way of OpenRAW's own agenda?

If you go back through posts from OpenRAW's supporters, you will find that a proportion of them think that OpenRAW is trying to devise such a format, or else they specifically request such a format. Now that DNG is becoming so well known and used, if OpenRAW is ever to be taken seriously, it needs to adapt to DNG. It should be trying to raise the quality and documentation of DNG conversion, and encouraging companies, (camera manufacturers, software developers, etc), to use it.

OpenRAW doesn't exist in a world where DNG can be ignored for purposes of "Digital Image Preservation Through Open Documentation". The US Library of Congress identifies DNG as the preferred alternative to other raw formats. Librarians and archivists across the world are starting to watch DNG to make a judgement about when it becomes their recommendation. (I have read a number of papers on this topic that mentions DNG). Such people want identifiable formats that they can recommend, not just a proliferation of formats. They are starting to talking about DNG in the same sentences as TIFF and PDF, etc. They certainly aren't talking about NEFs, CR2s, etc, in that way! They never will.

We have TIFF (now owned by Adobe) largely as an alternative to each scanner manufacturer using their own scanned file format. Look how successful TIFF has been in opening up the industry. We need the equivalent for raw file formats, and there is now only one contender. Many photographers have voted with their feet. They weren't going to wait for some hypothetical future where camera manufacturers open up about older formats.

You are a funny fanatic, Barry.

First you try to push your own agenda by posting wrong statements ('DNG is lossless') than you back up and give camera makers the fault that DNG cannot have all data, because camera makers don't document it.

And thats exactly our point. DNG could be so much better IF camera makers would document openly their file formats.

We have absolutely nothing against DNG, but a file format which so far (at least the converted DNG) throws away data, is lossy and its too early to throw away data at this stage.

We made comments how to improve the DNG format for all the reasons you mention, with the sole intend to improve the situation and as you probably know, we are not in a competition with Adobe, far from it.

But your statement about 'anyone with an obsession about every last detail of metadata' really just shows how incredible clueless you are and how little you understand about this topic.

Juergen

OpenRAW's views on DNG -

You say: "DNG could be so much better IF camera makers would document openly their file formats."

And I have been saying that for a long time, here and elsewhere. Here is what I said nearly a year ago: "OpenRAW are doing the right thing, except for their antipathy towards DNG. And if they succeed with their lobbying, the world of "raw shooting" will transform faster into the world of "DNG shooting". That will be good for the future health of top-end digital photography."
http://www.barrypearson.co.uk/articles/dng/openraw.htm#conclusion

You say "We have absolutely nothing against DNG, but a file format which so far (at least the converted DNG) throws away data, is lossy and its too early to throw away data at this stage."

But OpenRAW has shown antipathy and even hostility towards DNG. For example, it published "DNG is not the answer" very prominently:
http://www.barrypearson.co.uk/articles/dng/commentary1.htm
http://www.barrypearson.co.uk/articles/dng/assets/prominent.jpg

And I repeat: "DNG files are capable of holding all the raw image data from cameras with CFA-based sensors, including those with offset sensors such as Fujifilm cameras. They are capable of holding all EXIF data, including Makernotes. They are capable of holding lots more, for example TIFF/EP metadata and XMP metadata". Do you accept that statement?

More recently, in response to "Is the Adobe DNG format the pragmatic solution or maybe the same old song?", you responded:

"I wish it would be a solution, but Adobe decided for marketing reasons that they allow a backdoor to camera makers to store information in so called private tags, which remain undocumented. So even if some cameras can save DNG formats natively, some of the information a camera decides to conceal can be saved inside the DNG format and it becomes another undocumented RAW format after all. Plus Adobe stopped documenting their PDF and PSD format after it reached a certain market share, so they can at any time release a DNG V2.0 format and decide not to document it. No, DNG is unfortunately not a solution".
http://www.barrypearson.co.uk/articles/dng/commentary2.htm#juergenspecht
Note how inaccurate some your response was.

At various times, OpenRAW has expressed "fear, uncertainty, and doubt" about DNG. OpenRAW could have stated a policy towards DNG that included helping to improve the specification, and/or helping to improve the quality of DNG conversions, and/or helping to promote the use of DNG. It has done none of these. (More precisely, I have done some of those things on the OpenRAW website, but the originators of OpenRAW didn't).

For example, OpenRAW disparages the idea of using a common, openly documented RAW format for solving this problem, in favour of a proposal for which there is no supporting evidence:

"Many have suggested a common, open file format for RAW image files for all camera makers to use as a solution to the RAW problem. A common, openly documented RAW format would fulfill many of the goals of OpenRAW, but is likely to face significant resistance from manufacturers who feel their "creativity" and ability to innovate would be constrained. Open documentation of all RAW file formats by manufacturers is the quickest and most satisfactory way for OpenRAW's goals to be reached".

Where is the evidence that "Open documentation of all RAW file formats by manufacturers" will achieve "Digital Image Preservation"? The latter requires software products in future which can extract and exploit the raw image data from image files. That requires knowledge of both RAW file formats and of the details of the cameras that created that image data so that it can be processed. And it requires the software company to believe there is a suitable business case. Where will those camera details come from in future? What will be the business case? It is just wishful thinking!

You say: "But your statement about 'anyone with an obsession about every last detail of metadata' really just shows how incredible clueless you are and how little you understand about this topic."

Chuckle! Some of the world's experts on digital imaging and asset management for image archiving agree with the views I express, rather than the views you express. More precisely, I agree with the views they express!

But I have made an open offer (to any readers) on my website: "If you know of errors in my pages, please let me know, with suitable supporting material, and I'll correct them". And another: "You may not agree with all of my opinions. If you publish an alternative position in some consolidated form, such as a web page, (rather than responses in forums), please let me know and I'll link to it". Given that I have published more than 20 pages on this topic there, where is your evidence that I am "clueless"? (I wonder if you have even read those pages? Or made any significant use of DNG? Or analysed DNG and other raw files using tools such as the DNG SDK and others?)

NEFs, CR2s, PEFs, etc, are NOT suitable for "Digital Image Preservation Through Open Documentation". They never will be (and Pentax at least accepts that). Mere documentation of camera manufacturers' raw file formats is not the solution to "The RAW Problem". The only raw file format specifically designed for that purpose is DNG. What is the alternative?

With all due respect, Mr. Barry 'DNG is lossless, except the little bits it throws away nobody needs anyway' Pearson, you are officially an Idiot.

And you can quote me on that :)

Juergen

OpenRAW rejection of DNG

OpenRAW once prominently displayed a document called "The RAW Flaw". The authors were Michael Reichmann (of Luminous Landscape) and Juergen Specht (of OpenRAW). It was published on both sites:
http://www.openraw.org/actnow/
http://www.luminous-landscape.com/essays/raw-flaw.shtml

There were illuminating differences in the OpenRAW version:

The Luminous Landscape version said:

"The Solution? There really is only one solution – the adoption by the camera industry of...
A. Public documentation of RAW formats; past, present and future
or, more likely...
B. Adoption of a universal RAW format"

The OpenRAW version didn't have "or, more likely..."! This increases the emphasis on public documentation, and reduces the emphasis on a universal RAW format.

Another difference in the OpenRAW version was lack of the paragraph:

"Finally, consider the problems of digital asset management and the cataloging of files. Many pros as well as amateurs have literally Gigabytes of RAW files. The various asset management programs can't hope to keep up with the ever increasing number of proprietary formats. And as time passes and these programs are enhanced, what are the chances that they will still be able to read your older RAW files?"

That paragraph made the point that it isn't sufficient to have public documentation, but it is also important to have a universal RAW format to reduce the proliferation. This was not the message that OpenRAW wanted to give.

Other parts of the page DO mention the use of a a universal RAW format, and mention DNG by name. That page is no longer prominently displayed at OpenRAW. People who didn't know it was there would probably not find it. The concepts of "a universal RAW file format" and "Adobe has put forward the DNG format as an open standard" have been hidden by OpenRAW. Whereas Michael Reichmann (of Luminous Landscape) is enthusiastic about DNG!

----------

OpenRAW said on the page "The RAW Problem", up to end March 2006, "Many have suggested (and Adobe has created) a common, open file format for RAW image files for all camera makers to use as a solution to the RAW problem".

From April 2006 it omitted "(and Adobe has created)". In effect, this retracted their 11-month position that DNG is indeed a common, open file format.

----------

Just after OpenRAW redesigned their web site at the start of April 2006, they prominently published a large article "DNG is not the answer" by Stuart Nixon:
http://www.barrypearson.co.uk/articles/dng/assets/prominent.jpg

This article was edited and renamed "Notes on the future of Open RAW formats, and a look at DNG" a few days later. The theme remained the same, including the words "DNG IS NOT THE ANSWER".
http://www.openraw.org/node/1482

That article has been much-quoted in forums, with no attempt at a retraction by OpenRAW or Stuart Nixon. It has promoted "fear, uncertainty, and doubt" about DNG, using the "OpenRAW" brand. In fact, it does not stand up to scrutiny:
http://www.barrypearson.co.uk/articles/dng/commentary1.htm

That article, and the way that OpenRAW CHOSE initially to display it in that manner, (Stuart Nixon couldn't have done that by himself), indicates the antipathy, perhaps hostility, of OpenRAW towards DNG.

----------

More recently, Juergen Specht has been quoted as responding to "Is the Adobe DNG format the pragmatic solution or maybe the same old song?":
http://www.ladinamo.org/english/raw-format-the-captive-photo.php

"I wish it would be a solution, but Adobe decided for marketing reasons that they allow a backdoor to camera makers to store information in so called private tags, which remain undocumented. So even if some cameras can save DNG formats natively, some of the information a camera decides to conceal can be saved inside the DNG format and it becomes another undocumented RAW format after all. Plus Adobe stopped documenting their PDF and PSD format after it reached a certain market share, so they can at any time release a DNG V2.0 format and decide not to document it. No, DNG is unfortunately not a solution".

That is simply "fear, uncertainty, and doubt":

The fully documented information is sufficient to render high quality images. (Typically, products that support DNG need only the fully documented parts of DNG files). If a camera manufacturer puts in private data, the DNG file becomes "a file with documented data capable of rendering a high quality imaage, plus some other stuff which may or may not be useful".

Adobe intends to release the full PDF 1.7 specification to AIIM for the purpose of publication by ISO. PDF for Archive (PDF/A) and PDF for Exchange (PDF/X) are already ISO standards, and PDF for Engineering (PDF/E) and PDF for Universal Access (PDF/UA) are proposed standards. Additionally, PDF for Healthcare (PDF/H) is an AIIM proposed Best Practice Guide.

TIFF has similar status to DNG - an openly specified file format owned by Adobe. Who worries about what Adobe may do to TIFF?

From the license: ""DNG Specification" means any version of the Adobe Digital Negative (DNG) Specification made publicly available by Adobe". If it isn't documented, it isn't a DNG specification!

----------

OpenRAW never had a balanced attitude towards DNG. Over time, its antipathy towards DNG has hardened, even while DNG has been making significant progress towards "Digital Image Preservation Through Open Documentation". Why?

OpenRAW's rejection of Barry Pearson

I like this headline better ;)

As you'll see from the ongoing argument between Juergen Specht and Barry Pearson you've just opened a can of worms...

DNG goes some way to being the open format you describe. It is proprietary (owned by Adobe) but documented, at least in the current version (and yes Barry, probably for future versions). Personally I agree with Juergen that there are questions over it's usefulness while we don't have full documentation of proprietary raw formats.

While I'm dreaming, let's throw in the ability to create a portable RAW file, something that's reduced in detail to get the file down to under 2mb that could be more easily distributed over the web..

You could always use PNG or TIFF if you want a portable image format. Wouldn't you rather distribute finished images rather than Raw images?

The whole topic is a can of worms, you are entirely correct about it.

Barry is unfortunately not consistent with his comments, and his advise is dangerous, because he tries to convince people that the DNG format is the solution to the 'undocumented RAW formats' and that deleting your original RAW's is save.

However, if you look into the current Adobe DNG specs, you find this warning there:

It should be noted that the Adobe DNG Converter will not necessarily maintain all of the private metadata in certain camera-specific raw formats because this informa- tion is not publicly documented and therefore not available to Adobe. However, the Adobe DNG Converter will maintain all of the original image data as well as all of the metadata needed for a high-quality final conversion. Arguably, the private metadata is not really archival, regardless of the format used, simply because it is undocumented. Nevertheless, Adobe recommends that, when photographers use the Adobe DNG Converter for archival purposes, they should maintain both the resulting DNG file and the original camera-specific file. The DNG file offers greater assurance of longevity, but the camera-specific file may contain more metadata. This distinction does not exist, however, for DNG files created by camera manufacturers because they can include all of the private metadata within the Digital Negative.

The source: Introducing the Digital Negative specification: Information for Manufacturers (PDF, Page 5)

So we have the same chicken/egg problem here also for DNG, it cannot convert all meta-data, because they are not documented, ergo there cannot be a lossless conversion from one file format to the other, if the data which should be converted is not understood.

To your other question, the RAW file is comparable to your analog negative. If you keep your negative for 10 years and then develop a print with the newest chemicals and papers, it will look way better than it was possible 10 years ago.

The same with digital RAW files. Since 7 years I store all my Nikon NEF files as 'straight from the camera' and as converted JPG files with the current RAW converter. If I today look back at the 7 year old JPG conversations and compare them with a todays conversion of an actual RAW file, I can squeeze way more detail out of the same RAW files than I was able 7 years ago.

So its good advise to keep your RAW files and NOT delete them, you will bite yourself if you value your photos and do so.

And note, even Adobe advises against it.

I agree with your comments Juergen. I don't use DNG as I don't think it answers the problem of Nikon not documenting NEF files. I don't see the point in converting to an imperfect format and then having to keep the original file as backup. I already fill up enough disk space with images without doubling the amount of storage that I need.

What prevents DNG or OpenRaw from having an embedded XML file that is the text-based conversion of all of the metadata from the original RAW/EXIF? That's one thing I've never understood about camera makers and their bizarre file formats. We already own the camera and have bought into the system. Why screw around with our workflow process and make things more difficult?

Winged Power Photography
OurMag.com :: My Soapbox
Polygon Comics

Good point. And I would like to say: ALL of the recent posts have made good points, Jeurgen should not call Barry naive or an idiot. Yes, DNG is not perfect. But I think everyone here believes that "lossy" means image detail is lost; I don't think the term "lossy" should be applied to loss of undocumented metadata **that you had no way to read anyway**. Having publicly documented RAW formats is the very problem OpenRAW wants to solve; DNG would benefit from that! Adobe would be one of the "third party tool" manufacturers this organization wants to promote!

So, converting NEF to DNG is not the perfect solution now. But if camera manufactures could be convinced to save raw as DNG natively, according to the public DNG spec, then DNG would in fact serve the goals of OpenRAW.

So one point Barry makes I completely agree with: OpenRAW should find as many opportunities as possible to have Adobe on their side, and maybe influence Adobe to help serve OpenRAW's agenda. Calling someone who, admittedly, is a bit over the top in support of DNG right now, an "idiot" is probably not going to help OpenRAW's ability to influence Adobe.

From my chair, and my chair alone -- the largest obfuscation to a universal workflow stems from the DNG format enabling proprietary extensions.

What do I mean by that?

The DNG format appears to enable different flavors of DNG much as if they were ingredients in an ice cream sundae.

1. Vanilla. At its simplest, DNG specs would contain the core raw data necessary to demosaic a picture image.
2. Flavonoid Rich. The quality of Vanilla can be enhanced by the camera maker through secret ingredients -- providing you access them from a camera maker sanctioned workflow or SDK.
--secret recipes may contain anything from content (e.g. additional tonal hints) to style (e.g. focus markers critical to forensic work).
--these flavonoid rich extensions are value added property and deemed proprietary to the camera makers. They can make it available for their own product (e.g. NX2) or license it to third party interests. Some of these flavonoids are used to create the in-camera JPEG -- which may take a lot of extra work to replicate outside the camera using a third party product.
3.Toppings. The DNG sundae is further flavored with the addition of sticky sauces, whipping cream, and nuts. Currently Lightroom or ACR keep those layer of toppings separate from the vanilla -- providing you access the DNGs through an Adobe workflow.
-- an example of some stylized toppings may be subject keywords and cooking hints.
-- while the toppings aren't that secret, access to the precise means to render them is only by way of an Adobe sanctioned SDK.

What this currently means to my workflow, is that because neither :

a) the camera makers
b) or the topping makers

SDK s will render a portable framework, the DNG format remains contaminated with proprietary data that can only be unlocked through very different workflows. Furthermore, and I disagree with Joe here, the failure to have a universal SDK which unlocks all the proprietary extensions capable within one file format makes them for all practical purposes lossy -- especially after one has invested in technology (cameras, software) and processing time (to customize the styling).

How to resolve this?

Resolution is dependent on where you sit in the chain of command, however from a production and archiving workflow, these would be my suggestions:

1. The DNG format is adopted by an ISO standards body who will be it's new custodian.
2. On the body of this board would be representation from the software industry, the camera industry, OS makers, the production and commercial printing industry, and photographers. An experts group.
3. A consensus for a standard vanilla flavor of DNG would be rendered.
4. Camera makers would start offering the option of producing in-camera vanilla flavored DNGs. Anything outside the ISO standard would need to be a proprietary file -- e.g. NEF, CRW, ORF, CR2, DNGn, DNGc, DNGo, etc.
5. Camera makers must license SDKs to third parties for their proprietary files, or for transferring non-critical flavanoids to the DNG container.
6. Adobe and other third party RAW processors must stop using the term DNG. If proprietary toppings are added, they must call them something else -- for example XNG (for XML enhanced negative file).
7. Adobe and other third party RAW processors must license SDKs to third parties for their proprietary XNG files, and for transferring these non-critical toppings back to the DNG container.
8. The DNG files must contain an audit trail describing when and what flavonoids or toppings were merged into the format. Everything published within a DNG file must be easily understood and acceptable to other partners both upstream and downstream from the original picture data.

The primary benefit to the workflow is that DNG becomes a truly portable framework allowing universal access to the proprietary data. Because it's an ISO standard, all software and OS vendors would be able to access critical data, and license non-critical value added components. Because all proprietary RAW files or XNG files can be merged through a values added process, there is a commercial incentive for camera makers and software makers to buy into a universal format. Photographers can rest easy that at a minimum their files will always remain renderable, and for a small premium also unlock all the flavonoids and toppings that have been added. All data can be broken out without further obfuscation.

Granted, there are many technical issues to work out within such a framework, and there are a million reasons why people speculate this can't be done, but IMO it would be infinitely superior to the alternatives we now have before us. There may be a lot of reason for failure to come to an agreement, but amongst them there is not one good excuse.