OpenRAW

I agree that neither Nikon nor Adobe is the villain here. And I agree that every manufacturer and software vendor is entitled to protect their products by every legal means, including the various intellectual properties laws. And I agree that neither Nikon nor Canon nor anyone else should be blackmailed into acceptance of any DNG standard.

My only demand is for full documentation for the files that are my original images. I own those "RAW" files -- not Nikon, not Canon, and certainly not Adobe.

Files that have been manipulated by older versions of any software are recoverable if and only if that software's support system and marketing group continue to offer access over time. That includes Adobe, Nikon, Canon, and many, many more. However, as long as I know and fully understand what is in the original RAW files, the fate of my images is in MY hands -- not theirs. They -- none of them -- have any right whatsoever to deprive me of full access to something that is rightfully mine.

Hi Rags

Question:

Are you in favor of openly-documented
RAW formats ?

Thanks ...

-- stan

Stanley,

Yes. Of course.

The question is what are the minimum metrics that other vendors need to access the data. Once that has been agreed to, the manufacturer should not be constrained from providing additional proprietary features. And standards compliance is a choice. The difference is that if there were an agreed upon OpenRAW standard, compliance could be certified.

Cheers, Rags :-)

Rags,

You make a good point in your analogy to film. The problem is, we are operating in a new paradigm. Digital is not film. Digital and film photography are different from the moment the light passes the shutter (and there are differences even before, as the light passes through the lens.) The biggest difference is that the the camera and the imaging medium are now one (until Leica releases their back) and cannot be seperated. Photographers are adjsuting. Camera makers are adjusting. Nikon has shown they get the new paradigm. Their 'advisory' about the white balance is proof. Nikon understands that it is no longer enough to build the best camera body. Their digital SLR's are judged on the final output of the camera (and by extension raw processing software.) Because of that they are trying to control the entire system camera, lens, processor and raw conversion. Using your analogy to film, would photographers purchase and use a camera that could only use one type of film? Probably not, but that is the situation with RAW and proprietary files if they are only able to be processed by one piece of software.

Digital is a new paradigm, but the shape and rules of it are still being set. Should the camera manufacturers decide it? Should a software company? No, neither should set it on their own. A standard file for all cameras is needed but I feel it is too early. We are still feeling our way around this new media. It will take time to shape this new type of photography and it should be done with the participation of all stakeholders. OpenRAW can help shape that new paradigm by involving the users of digital cameras and there output.

Rags: I'll comment on a few of your points:

"I do think that Adobe should not own the OpenRAW standard".

Adobe are not trying to own "a standard". That is not a good way of thinking about what they are doing. They own, and have published, a well-engineered specification, (a file format called DNG), and they have licensed anyone to use it if they choose. They have also given away a tool to convert other Raw formats into DNG.

No one has to notify Adobe if they decide to use (or stop using) the specification. So Adobe has no control over the uses to which the specification is put. No company or product using it is constrained by Adobe. Howewer, those companies are somewhat constrained by their own customers! There have been many endorsements of DNG by photographers and users of photographs. Companies will tend to use DNG because their own customers ask them to. And for good reasons. THAT is why DNG will assert more and more influence - via the people who matter.

"Adobe’s DNG format may be a good technical base for an open standard, but this technology is currently too dynamic and there are too many dependencies for a single vendor to own the standard if there really is to be one".

The technology is requiring changes to Raw formats rather slowly! Had DNG existed for a few years, the last technology change that would have needed a change to the specification would have been about a year and a half ago, for the Fujifilm SR sensor. No new camera in the last year and a half would have needed a change to DNG, had they all been using DNG for years.

Adobe has everything to gain, and nothing to lose, by working with any camera manufacturer who has a new technology that can't be handled by the current DNG specification. All the manufacturer would have to do is ask!

"I am unanimous in my belief that any useful OpenRAW standard has to address a lot more than white balance".

DNG obviously handles far more than white balance! Indeed, most serious users of Camera Raw don't worry about the white balance problem. ACR 3.1 supports the D2X. Nikon's encryption of the automatic white balance was an attack on its own customers, not on Adobe, trying to coerce those customers to buy Nikon Capture. Virtually all the debate over the last few weeks has involved criticism from Nikon's customers, with Adobe normally staying out of the discussions. Nikon's customers are worried that the white balance encryption is just the first step, and any more would be intolerable to many photographers.

"I am unanimous in my belief that we have to allow the camera manufacturers the freedom to explore new technologies, in both the hardware and the software".

Who is arguing otherwise? Adobe isn't. But, new technologies hardly ever need changes to a well-engineered Raw format. And the mistake Nikon made wasn't to explore new technologies, but to try to lock-in their customers to Nikon Capture, by encrypting part of the Raw format, with Nikon Capture being able to decrypt it.

This website says "Open documentation of all RAW file formats by manufacturers is the quickest and most satisfactory way for OpenRAW's goals to be reached". And if all camera manufacturers chose to use DNG from now on, that would immediately satisfy OpenRAW's goals for future cameras! What better way for a camera manufacturer to openly document its Raw file formats than by using a specification that is already published?

Ed: Nikon is undoubtedly the villain here!

You said: "I agree that neither Nikon nor Adobe is the villain here". But Nikon encrypted part of the Raw file from the D2X, and that encryption has no purpose other than to coerce D2X customers into buying Nikon Capture because it can decrypt that part of the Raw file.

If the OpenRAW website graded sins, they would probably judge lack of documentation of a Raw format to be a middling sin. But I guess they would judge encryption, as a deliberate attempt to hide information, to be one of the worst sorts of sin!

The only people who might try to blackmail anyone into using DNG are photographers and users of photographs. And for good reasons! When Adobe published DNG, they offered photographers and users of photographs what they wanted and needed. Anyone who resists is resisting photographers and users of photographs.

You say: "My only demand is for full documentation for the files that are my original images". What better way of doing that than using DNG, which ALREADY has full documentation?

In fact, documentation isn't sufficient. You need software tools, not just documentation! I guess you don't want to write your own tools, using such documentation.

Not only is DNG fully documented, but there are already many tools that support it.

Larry,

Yes, digital imaging is a newer paradigm. But you don’t have to throw out the baby with the bath water.

Another example I forgot to mention. Film data almost always included spectral response curves and similar technical data. This availability did not compromise trade secrets or patents. These spectral response curves are all but missing from any of today’s sensor specifications. They are the logical equivalent of the color space of the sensor itself. They are essential for the color accurate demosaicing and interpolation of the sensor data.

We already have enough collective experience with digital imaging to define a minimum set of metrics needed for image formation. Different vendors may decide to employ different algorithms with their own unique advantages or disadvantages. Vendors and manufacturers can still offer unique functions and features. We will still have choices and competition.

Time will surely call for revisions to the standard. For example, TIF today allows various compression techniques, layers and such. Some support some these, some don’t. But since TIF is only a de facto standard, it is hard to tell what applications can open a particular TIF file. It is difficult to determine compliance.

Barry,

Thanks for the comments. I just think that our time is better spent looking for a solution than looking for a villain.

All,

I think that the time is ripe to put the chatter into action. Hopefully OpenRAW can be that portal. It needs to transform itself into an organization that can attract industry respect, define an actual written specification, and define a process for certification. Simply asking all manufacturers to fully document their propriety raw formats is a lazy way out that will neither encourage compliance nor insure usability. Personally, I would like to see the issue taken up by the ISO. And again, DNG is an acceptable starting point for me at least. This should obvious without even saying it.

Cheers, Rags :-)

Larry, camera manufacturers are devising the new technologies, while Adobe are trying to rationalise them.

You say: "Digital is a new paradigm, but the shape and rules of it are still being set. Should the camera manufacturers decide it? Should a software company? No, neither should set it on their own".

Over the last few years, Adobe have been following the new technologies from the camera manufacturers, and trying to turn those technologies into something with much wider acceptance. The camera manufacturers are good at devising new digital technologies, but not very good at making those technologies work for photographers, and users of photographs. Adobe is very good at the latter.

Adobe didn't devise the Fujifilm SR sensor technology. Instead, after it had been devised, they catered for it in Camera Raw, and built its requirements into the DNG specification. It is now possible for a software package which has never even heard of Fujifilm to handle its Raw format and do useful and innovative things with it.

Adobe is in a position to examine the formats from all the manufacturers, and identify where the differences are totally unnecessary. (Which will be most of the time, of course! The camera manufacturers are not trying to eliminate unnecessary differences). Technology changes need rather few changes to a well-engineered Raw format. The last such change was a year and a half ago. All differences since then were unnecessary.

I'm sure that Adobe would like to be involved with the discussions about the demands on Raw formats. But, up till now, they have been trying to pick up the pieces afterwards. This is a pity, since Adobe understand the needs of photographers, and users of photographs, better than the camera manufacturers do.

Adobe was talking to the camera manufacturers about 8 months or more before launching DNG. And that was over 7 months ago. Nikon's attack on its customers, via encryption, occurred since then.

Adobe is not in the camera business. It is in the business of helping to produce the best possible results from photographs. Photographers know this. Isn't it about time that the camera manufacturers accepted this?

Rags: you say "I just think that our time is better spent looking for a solution than looking for a villain".

My position has always been "I am in the solution business, not the debating business". All of my posts here are focused on solutions. And, obviously, DNG is part of the solution. If you look at "The Problem with Proprietary RAW files" on this website, it identifies 4 problems:

1. "Limiting processing choices and creative freedom".

2. "Reducing choices for software that matches workflow needs".

3. "Increased probability that as time passes a RAW file will be unreadable or cannot be used to reproduce the photographer's original interpretation".

4. "Increased costs and slowed development of image processing software".

For many photographers, using DNG helps to solve "1" to "3". There must be many photographers who, like me, have been using DNG for several months, and who feel that we have "1" to "3" under control.

"4" is perhaps what this website is aimed at. It isn't ideal that the several software products helping with "1" to "3" do so with reverse-engineering, rather than via published specifications. Perhaps the conversion from proprietary Raw formats to DNG would be better if the manufacturers published their Raw formats, as desired by this website.

I'm an engineer rather than a theoretician. I am concerned with what actually works, now and in the future. DNG undoubtedly works, for any photographer with a matching workflow. (It won't work if you want to use a Raw processor that won't accept DNG, of course!)

The only point in talking about villains is so that we can see that Adobe are part of the solution, and Nikon are part of the problem. This will enable us to focus our attention on what the problems are, and who is causing them, so that we can try to stop these problems happening in future.

The only problem that camera manufacturers would face if they adopted DNG is that they could not then coerce their customers to use their own software. DNG requires all the basic image data to be available, unencrypted. As required by photographers, and users of photographs, of course.

Hi Rags

Thanks for an interesting discussion.

> Simply asking all manufacturers to fully document their propriety
> raw formats is a lazy way out

"lazy" is so judgmental. I prefer "minimal".

Additionally, "lazy" implies easy. It will not be
easy to get camera manufacturers to take this minimal step.

But it will be sufficient. By doccing their formats, all the
other dominoes we'd like to see will fall into place.

I'm into strategic minimal steps. They're realizable, and powerful. Non-minimal steps
are much less so. Stan's Fundamental Theorem
of Human Change Calculus.

> Personally, I would like to see
> the issue taken up by the ISO.

Hmmm. Slow, bureaucratic, and likely to be driven by
the manufacturers and their needs rather than by
the users (photographers). Are you sure ?

Best,

Stan

I very much agree with Stan here.

ISO would take years, but could deliver a good standard.

Rags: "Personally, I would like to see the issue taken up by the ISO".

Stanley: "Slow, bureaucratic, and likely to be driven by the manufacturers and their needs rather than by the users (photographers)".

It is possible that this matter would be administered by I3A, the "International Imaging Industry Association", (formed by the merger of DIG and PIMA). They don't only administer their own standardization program. They also act as secretariat and administrator for ISO/TC42 — the International Organization for Standardization Technical Committee 42 on Photography. And they serve as administrator of the USA Technical Advisory Group for the ISO/TC42 committee.

This is important, because their membership, (of various kinds), includes more than just the camera manufacturers. For example, Adobe is a Participating Member. (So is Canon, but I don't think Nikon is a member).

The problem is one stated by Stanley. It would take years, (unless there is a fast track that I am unaware of). What do we do in the meantime? If Canon asked OpenRAW today "what Raw format would you like to see in our next camera?", what would the answer be? If a photographer wants to alleviate the problems identified by this website, what can they do today? And - what would any ISO standard in this area be based upon?

The only sensible answer to all of these questions is DNG.

Mr Pearson,

I understand that *your* only sensible answer to the RAW problem is DNG; however, regardless of your faith in Adobe and regardless of Adobe making DNG a freely available & well documented 'open' standard, it is unlikely to happen. It is unlikely that any camera manufacturer would comply. (Were I the decision maker at Canon or Nikon, I certainly wouldn't.) However, if we convince the camera manufacturers to provide open and complete documentation for their RAW formats, most of the problems simply evaporate. We all get what we want: guaranteed access to the original image data and easy, accurate conversion to DNG should any of us so desire.

Full documentation of RAW image data is a demonstrably reasonable request in that we, the photographers, fully own the data. A demand that camera manufacturers use DNG -- another manufacturer's product, however universal it may become -- is not a reasonable request and is one that is very likely to be totally ignored.

Ed: here are the 3 questions I asked above. What do YOU think sensible answers are?

1. "If Canon asked OpenRAW today "what Raw format would you like to see in our next camera?", what would the answer be?"

My answer is "DNG". What is yours?

2. "If a photographer wants to alleviate the problems identified by this website, what can they do today?"

My answer is "convert to DNG and archive the DNG, if necessary embedding the original Raw file". What is your answer?

3. "And - what would any ISO standard in this area be based upon?"

My answer is "DNG". What do YOU think it would be based on?

This website states "Open documentation of all RAW file formats by manufacturers is the quickest and most satisfactory way for OpenRAW's goals to be reached". Since photographers can't use documents, but need software tools, I believe this translates into:

"Open documentation of a RAW file format"
causes
"software tools to be developed to support that format"
resulting in
"image preservation and creative choice of how images are processed".

And that, in turn, implies:

"Open documentation of DNG"
causes
"software tools to be developed to support DNG"
resulting in "image preservation and creative choice of how images are processed".

(And we certainly have those software tools!)

When you remember that DNG is a Raw format, you can see that anything that is said about "open and complete documentation for their RAW formats" automatically applies to DNG too. The only difference is that DNG has been published for more than 7 months, while it remains to be seen whether any other Raw formats will EVER be published. How long should photographers put their decisions on hold before going for the available solution?

The only disadvantage to a camera manufacturer in using DNG is that they can't then use encryption to lock-in their customers to their own software. If they don't intend to do that, use of DNG is good for them. They may, of course, be unable to see this.

Stan,

Good points. That was my lazy and minimal brain at work.

As Barry points out, the ISO does not operate in a vacuum.

They don’t invent things. They facilitate, document, and certify them. They have been known to respond in a timely fashion when properly motivated. Anyway, I do believe this should be the long-term goal at least. The I3A and ISO/TC42 sound like good places to start.

Cheers, Rags :-)

Mr Pearson,

I understand that DNG meets all the requirements of a universal RAW format except for one: Neither Canon nor Nikon (nor, most probably, any other camera manufacturer) is ever going to ask your question #1. DNG may be open, well-documented, and free, but it belongs to Adobe. Period. End of story.

With proper, full documentation of all RAW formats, the problem goes away (or, rather, it's manageable).

Ed, what evidence do you have to say "Neither Canon nor Nikon (nor, most probably, any other camera manufacturer) is ever going to ask your question #1"?

I have already seen a statement from Leica that they will support DNG. Ditto Hasselblad. I have not seen any statement from ANY manufacturer that they will ever publish their Raw formats. Have you?

It may be that the manufacturers won't do either, at least in the short term. What do we do in the meantime? All of my Raw files are in a format that is open and well-documented, and supported by perhaps 25 non-Adobe products. The fact that I have to convert them into that format is an irritant, but not a stopper. (I converted the latest set directly from the cards, which didn't add much to the copying time). I don't have, or need, documentation of the Raw format for my camera. What I rely on instead is that someone has been able to reverse-engineer the format, and make a free converter.

I would prefer that it wasn't necessary for someone to reverse-engineer formats. It would be better that they could work from proper documentation. But I can't see why the documentation MUST be openly published. I would have thought that availability under an NDA would be acceptable, as long as it was available to any developer, not just companies. I wonder if FULL documentation is too much to ask? Even DNG doesn't demand that! It allows the manufacturer to have its private data, ("secret sauce"), as long as the basic image data is openly available.

TIFF 6.0 belongs to Adobe. All the cameras we are talking about here support it. Obviously, ownership by Adobe is not a stopper!

Barry, a minor detail but I just found out that TIFF/EP is owned by ISO. It is sponsored by the TC 42 Secretariat that you referenced earlier. International Imaging Industry Association, Inc. (I3A) is actually listed under ANSI (USA). These are two well-recognized standards organizations with an impressive list of industry participants including the CIE, IEC, JTC, and 13 participating countries. The PDF document is available from either organization (ISO/ANSI) for about $125.

Yes, TIFF 6.0 is owned by Adobe and available as a free download. Yes, DNG basically supports either, but there are some noted exceptions.

I just thought that some folks might find this interesting information.

Cheers, Rags :-)

Rags, here is the relationship between DNG and the 2 types of TIFF.

TIFF 6.0 is owned by Adobe, and was owned by Aldus before that.

When you set your camera to TIFF, that really means TIFF 6.0. Raw processors normally have TIFF as an output option. Photo-editors normally read and/or write TIFF 6.0 as options. (You can actually use TIFF 6.0 instead of PSD for Photoshop - make sure you ask for layers to be preserved). If a magazine buys one of your photographs in TIFF form, they mean TIFF 6.0.

TIFF 6.0 images have the full colour informaion for each pixel, (except for monochrome images, of course). TIFF 6.0 is not suitable for Raw files, because it has no tags defined for holding sensor data, and there are many other metadata tags that it would need also.

TIFF/EP ("Electronic Photography") is owned by ISO. It was developed from TIFF 6.0, and much of it is copied directly from TIFF 6.0. (Canon, Kodak, Nikon, Fujifilm, and Olympus may own patents relating to aspects of TIFF/EP).

It has those extra tags for sensor data, (look for tags starting CFA - Color Filter Array). It also has a lot of extra metadata tags. Camera manufacturers typically (but not necessarily) start with TIFF/EP when designing their Raw formats, because it has many of the tags they need. Indeed, one manufacturer originally used the .TIF extension for its Raw files.

Like TIFF 6.0, TIFF/EP has very many options. And, of course, although it is a standard, there is no constraint on manufacturers to conform precisely to it. So resultant Raw formats proliferate, and there are lots of unnecessary differences.

(The ratified standard costs 150 euros. It may be possible to find a free download of the draft standard. Search for N4378.pdf but ignore the www.pima.net site because that domain changed hands when it merged to become I3A).

DNG, like TIFF/EP, is a development of TIFF 6.0. It has a few tags that TIFF/EP doesn't have, to cater for technology innovations since TIFF/EP was defined, and for control purposes, such as the 2 version tags that define the range of DNG versions that a particular file is compatible with. Those latter tags are why anyone is able to use DNG without fear of control by Adobe. Versions can't be changed once published, (changes need a new version). So a camera manufacturer (say) will write "1.0.0.0" and "1.1.0.0" as the range of versions into the file, for as long as those versions are suitable for their cameras, and Adobe isn't in the loop.

DNG cuts through all the unnecessary options in TIFF/EP, eliminates some of the more complicated structural options, and makes some tags mandatory. So it ensures that the image data can always be extracted, along with sufficient extra information that Raw processors can handle the image data without needing to know about the camera. (The file is self-contained). But it also defines how camera manufacturers can safely store any "secret sauce" in a DNGPrivateData tag defined for the purpose. "Safely" means that anyone else can re-write a DNG file while preserving the private data, because although its contents are private, its format is well-structured.

That latter ability to re-write DNG files without losing information, even if you don't know the content of the private data, is why Adobe is able to use DNG as a way of holding both the Raw data, and ACR settings and other additional metadata, in ACR 3.1. So only one file is needed, and you don't need sidecars any more.

Barry, could you please explain your relationship with Adobe in detail? Also your technical background?

Thank you very much,

Juergen

Juergen:

Relationship with Adobe:
"customer". (Nothing else). User of Photoshop for 4 years. User of DNG for 7 months.

Technical background:
http://www.barry.pearson.name/business/consultant.htm