I think that the presence of two...
I think that the presence of two solutions to the Raw Flaw diffuses the arguments. There is only one possible enduring solution: adoption of a single, self-defining, extensible industry standard for raw file encoding, built into every camera.
To put a point on it: if Adobe's proposed standard is as good as any, then that's the one for me. They have not displayed the pissy nature of other software giants in fiercely protecting their once proprietary formats. And they don't sell cameras. Perfect! As a reward, they should win the prize, if being credited without monetary benefit with creating a public domain format is a prize of any kind. By "pissy", I refer to Microsoft (of all damn companies) demanding restitution from everyone who uses their pathetic but popular FAT system on memory modules; and Compuserve being upset about others using their GIF file format. Adobe, where the lights are on and everyone is home, have eagerly accepted the common adoption of their Portable Document Format.
Why is a single format better than many that are publicly understood? The idea of having camera makers publish their formats yields the problem of everyone else having to write software to interpret them. For buyers of cameras from less-than-popular makers, this could be an unpleasant problem. Who would bother? Think of how cheap it will be. You never have to load up new software just to read someone's raw files again.
Consider where we would be if the data communications world could not settle on common protocols. We'd all be converting our brains out just to get an email in the door. And we'd be constantly buying nasty little gizmos from avaricious little trolls just to connect to a local internet provider. If, ten years from now, I can read the raw files from a camera that was released to the public 3 weeks earlier, I'd say we will be where we want to be.
So what do I mean by "self-defining" and "extensible"? "Self-defining" means that new information can be added without confusing old software because every file contains a description of the data as well as the data itself. This is important but not a biggy - EXIF already qualifies. "Extensible" means that new information can be added without changing the standard, because the standard accounts for it. It implies that, as the format matures along with photo technology, the extensions will not include re-definitions of how commonly understood "old" information is stored. Software that can read CRF 5.2.3 should have no trouble with a file encoded as CRF 3.1.1 (I made that up - CRF means Common Raw Format) and vice versa.
A bright future also requires that the format bullies restrain themselves, quite unlike the way Microsoft and Cisco have done in the past. Every manufacturer will create their own software and add their own extensions to their files. Big industry players putting vital information into self-indulgent extensions will mean that everyone else will have to follow suit to keep up. Microsoft tried to force their Java language extensions onto the world until the courts told them they were violating their agreement with Sun MicroSystems. Bill decided to create his own language (C#, and an entire framework to carry it - .net), in retaliation. Cisco is known to bully others on the standards committees into working with their flavours of emerging protocols, MPLS being an example: "This is how we're going to encode routing information. Our equipment is everywhere. Ignore us at your peril."
In summary, publicizing proprietary formats is only a stop-gap solution for existing formats. The most enduring solution will be an industry standard that anyone who wants to sell cameras will have to live with.
That should hold me for now.
In Response to Tim, OpenRAW is not a stop gap measure in...
In Response to Tim,
OpenRAW is not a stop gap measure in any way I can think of. To take you suggestion of a common format OpenRAW in now way impedes it's adoption. In fact it will make it easier to create and maintain an intermediate standard (DNG is an intermediate standard until cameras write DNG natively). I cannot argue that a common RAW format written by all digital cameras would be bad, clearly there are many advantages.
Adobe's DNG is (in it's current version) however a stop gap format. There are portions of the specification which, in my view, were included solely to entice the camera makers to adopt the format. DNGPrivateData and the makernotes provisions allow camera makers to continue to keep portions of their RAW files hidden. There are many other reasons that many others have cited (though none as persistently as Barry Pearson) both for and against DNG as the common RAW format but I won't enumerate either side here to make my point.
OpenRAW is really about a way of thinking. RAW files have become an important and powerful tool in photography. OpenRAW is an movement that is trying in several ways to convince camera makers that 'Open' is the right direction to take in RAW files, not closed.
Larry, Tim said "publicizing proprietary formats is only a...
Larry, Tim said "publicizing proprietary formats is only a stop-gap solution", not "OpenRAW is a stop gap measure".
That is why I agreed with Tim. There is no doubt that we cannot simply continue into the future with proliferating Raw formats, even if they were all to be openly published. We know how to solve this class of problem, and it always involves conformance to common specifications.
I have said elsewhere that I believe a possible future standard would be for an archival version of DNG or similar. Call it DNG-A, as an analogy with PDF-A. This would only have fully-specified, open, content. (Just as PDF-A would not include encryption).
But the current version of DNG resolves all 4 parts of "The Problem with Proprietary RAW files". After all, it would be better than the current state, and already large numbers of Raw formats CAN be handled by independent software. The private data wouldn't prevent access to the image, because that is in the open, mandatory part. It would just enable camera manufacturers to give an edge to their own software.
If OpenRAW objects to that, then it should take a 2-part approach: lobby for DNG, and lobby for minimal or no use of private data and/or publication of private data. We need an answer to the question "what should future cameras do?" It isn't "use whatever Raw format you choose, but publish it". It is "use DNG", at least until there is an alternative. Perhaps qualified by "without private data", if desired.
Barry, My main point here is in my last sentence. To me,...
Barry,
My main point here is in my last sentence. To me, OpenRAW is about open vs. closed.
As I see it, OpenRAW is seeking to cure the disease of proprietary digital camera output. DNG is a treatment for the symptoms.
Treating the symptoms is valuble but care needs to be taken so that the existence of the disease is not forgotten with the relief of symptoms.
To further illustrate, let's extend the two thoughts to their extremes. If all RAW formats were fully documented the disease of proprietary output would be gone. The symptoms would, in large part, be relieved. The problem of the proliferation of RAW formats would still exist, and it's only solution is a common RAW format. The creation of this common format will be much easier in with the disease gone.
If all digital cameras wrote a common RAW format that contained large sections of protected proprietary data the symptoms detailed in "The RAW Problem" would be resolved but the disease would still live inside the common RAW format.
The above is how I see OpenRAW as a concept and goal. I appreciate your comments and thoughts on DNG even though I disagree on the specifics. If the camera makers would openly discuss RAW formats as openly we would be a lot closer to curing the disease and limiting the damage done by the cure.
Thanks Barry, Tim and everyone else who has posted on OpenRAW.org, all the comments are changing the RAW format weather, and hopefully, a start of a favorable change in the climate.
Larry, this isn't OpenRAW versus DNG. (And, to be fair, I...
Larry, this isn't OpenRAW versus DNG. (And, to be fair, I think you have said that yourself elsewhere?)
A common Raw format is the strategic future, while publicizing proprietary formats is a mopping up operation because of the mess we are in.
These are all part of the jigsaw. I think that all of the bits of the jigsaw have a place under the OpenRAW "banner" in some sense, because it is such an excellent name. (That doesn't mean that OpenRAW should necessarily spend much effort pursuing all the parts, just that OpenRAW is perhaps a good place to present the vision. Indeed, "The Raw Flaw" goes much of the way). I noted, when following the search links in "press", that there was sometimes confusion about what OpenRAW actually was - lobby group, a standard, (disgruntled photographers!), or what.
I suspect that in future, but probably many years away, we may have a "common raw format / archival version", perhaps in ISO. It would not have any undocumented or encrypted bits. This is probably exactly in line with your vision.
A step on the way may be an initial "common raw format" that hasn't this level of purity, but still has all the sensor data and essential metadata in the published, mandatory bits. (Your "large sections of protected proprietary data" may be misleading. Since the open parts of this common raw format includes the sensor data and the metadata needed to make sense of it, there is a limit to what else would be in the private bits). That doesn't mean that it couldn't be used for archiving. But using it for archiving may mean one of the following:
1. Accompany it with documentation of those extra, currently private, bits. In line with OpenRAW's basic proposals. ("DNG + publicising" is probably better than "proprietary + publicising").
2. Have an option not to have those extra bits. You have given me some thoughts on this!
3. Live with the fact that processing the image without the extra bits may be harder, although still possible.
The creation of either version of common raw format has no need to wait fully documented Raw formats. In fact, of course, DNG is precisely an example of the second, "impure", version mentioned above. And the most trivial way of specifying the pure version would be simply to remove the private tags. So these common raw formats, albeit not yet used by camera manufacturers, could both exist next week, although only one of them actually will. That is why I don't agree with the sequence implied by "The creation of this common format will be much easier in with the disease gone".
To me, the main "disease of proprietary output" is a combination of:
A. The combinatorial problem; (X formats supported by Y products). Mario Westphal's superb post shows the problem for X formats and 1 product, and the world has better things to do than have that for every product. Much of his problem would exist even with full documenation. This has to stop, soon.
B. The dominant supplier problem; Canon and Nikon will always get first support, while my Pentax will be lower down the list!
Other parts of the jigsaw could include:
P. An open source version of the DNG Converter, able to react fast as new cameras are released, and as more information becomes available, for example breaking the encryption. Or new documents from the manufacturers, since these will not all arrive at once. This would break out of the Adobe release cycle, and also enable the sort of decryption that Adobe dare not do.
Q. Persuade camera manufacturers to accept DNG as input and/or output in their own Raw processing software. (In spite of its faults, Nikon Capture is better than the Pentax equivalent!)
The problem if OpenRAW doesn't position itself more strongly with DNG is that the many people who have been using DNG for up to 8 months may be puzzled about what OpenRAW is adding. I was puzzled at first, but I understand better now. Now, to me, "publicizing proprietary formats" is really about getting better DNG Converters, not living with the proliferation. But that's just me!
Response to Larry, I guess I presumed/assumed that if a...
Response to Larry,
I guess I presumed/assumed that if a camera manufacturer bought into a common raw format, they wouldn't be doing stuff like encrypting secrets into it.
For an open standard to work, it can't happen by implicit co-operation. Every meaningful open standard in computers and communications is guided by active committees composed of producers and consumers. Producers buy into the process and participate in maintaining the standard specification. Consumers buy into the process and make their requirements clear to the producers whose participation makes life on earth reasonable.
If, as you suggest, certain producers "... wrote a common RAW format that contained large sections of protected proprietary data ...", then you would be right. But I'm convinced that belligerent producers who pretend to cooperate while doing the opposite, will have some serious splaining to do. No manufacturer today has the kind of market dominance that Microsoft, for example, is famous for: even Canon and Nikon could find their market positions threatened by refusal to cooperate openly in a popular matter of this importance.
The original intention of OpenRaw may have been to convince producers to publish their proprietary formats, but sticking to that notion will become pointless the day the standards committee is struck. On that day, the age of proprietary formats will die like the age of dinosaurs died when "the meteor" struck. Some may live on, but they will become environmentally irrelevant and their progeny unviable. Published proprietary formats will become good legacy information for people using older cameras. Utilities will appear to convert people's old raw archives to the new open format, (like vinyl to CD). And that will be the end of it.
You see, it's not that a common format is a better approach, it's that it's certain to become the only viable approach.
Not sure if this point came up before, no time to read...
Not sure if this point came up before, no time to read through all of the posts.
But: a standard, common format is only viable if the technology has matured enough. With the current design cycles, there is simply no time for the manufacturers to go through the process of officially adding a tag to a common standard format to provide for a specific killer feature they might have. And they probably wouldn't want to do this anyway, since this would be a market giveaway. So what happens: it will be put in some manufactures specific part of the file.
This situation already exists: Nikons raw format e.g. is just a tif file with a lot of info in the makernote section. (by using tiff, Nikon is doing something right there BTW ;) The standard format will very soon be polluted with lots of non-standardized information, probably even not through ill-will of the manufactures, but out of practical reasons.
So I believe more in publishing the formats than in a common format.
However, camera makers can make things more easy for developers by keeping changes in their own format under control.
Also, given the scope of potential information in those formats, a common format that is efficient *and* provides full support is not much more than an idle wish. I can easily think of data that will make very much sense in the image data file, but that can be so maker specific that it will lose the efficiency of the common format if it would be included in there.
BTW: DNG, by being based on tiff, is a mistake. there are formats out there that are easier to work with, certainly if you want to make changes. Absolute offsets - aarrghhh!
cheers, jan
jan: You say: "a standard, common format is only viable...
jan:
You say: "a standard, common format is only viable if the technology has matured enough":
It HAS matured enough. If DNG had existed for a few years, the last technology change that would have needed a new version of DNG would have been the Fujifilm SR sensor, about a year and a half ago. No new digital camera since then would have needed a change to DNG. No new camera since DNG was actually launched 8 months ago has needed a change to DNG.
You say: "DNG, by being based on tiff, is a mistake ... Absolute offsets - aarrghhh":
There are no absolute offsets in the mandatory part of DNG. DNG is TIFF/EP brought up to date and made fit for purpose. It adds new tags needed because of technology since the TIFF/EP standard was developed. It eliminates unnecessary complications. It adds extra metadata to make the file self-contained, so that it can be processed by software that has no knowledge of the camera that took the picture. All mandataory tags are well-behaved, and can be written anywhere.
It introduces a new tag (MakerNoteSafety) that enables a camera manufacturer to state that its Makernote tag is relative rather than absolute. That enables processors to discard data they can't write back, while preserving Makernote data that is safe, even if not understood.
DNG is effective engineering, because it obeys the rule "if it isn't broken, don't fix it". Much of TIFF 6.0 and TIFF/EP isn't broken, so DNG just uses it. It fixes what it broken.
Comment: when talking about Raw files, it is very important to distinguish between TIFF 6.0 and TIFF/EP, because they have different implications. Just talking about "TIFF" is likely to cause confusion and ambiguity.
eeuh, AFAIK (just checked with the DNG spec), DNG is based...
eeuh, AFAIK (just checked with the DNG spec), DNG is based on TIFF 6.0 and TIFF/EP. These are full of absolute offsets (IFD and IFD entries). Changing something means recalulating all offsets. What is new in DNG is that there is provision for makernote tags that use relative offsets, but OTOH must be byte order independent; which means that makernote structure must or only use ascii, or has to use at least slightly different parsing code than the main tiff(/ep) part; which is annoying and inefficient.
cheers, jan
Ah, my misunderstanding! Do these offsets trade writing...
Ah, my misunderstanding!
Do these offsets trade writing speed for reading speed?
I don't think it makes a lot of difference for...
I don't think it makes a lot of difference for reading/writing. I'm not sure why absolute offsets were chosen in TIFF. Unfortunately the way TIFF has been extended with various schemes, this is becoming more and more of an annoyance. This type of data storage makes it for software packages difficult to treat files with tags they don't know - they can not safely copy them, and information gets lost. As things currently are, to safely read a tiff file (dng or other), change something and write it again *without* losing any of the data that is in the tiff is very difficult. Even if there is no 'unofficial' content, you have to implement all of the official extensions to basic tiff, so you can interpret the data correctly and write the correctly. Obviously this is a lot of work for tool developers, and the number of tools that will preserve various meta data are rather limited.
Forward compatibility is a problem as well - a tool that is older than the data it needs to treat can have problems with data types introduced after the tool was made.
The Canon crw format seems a lot better in respect to this - seems to look a lot like tiff, but with relative offsets.
DNG makes a lot of sense for Adobe, as it allows them to have a central format in their software, and their data formats are strongly tiff based anyhow - so this cuts on their development costs. But for the whole community of users and developers, a better format could be made that, in the long run, will offer easier development and more chance of data integrity.
cheers, jan
Tim, what a super article! That is the closest to my own...
Tim, what a super article!
That is the closest to my own view that I have seen posted here so far. I now use just DNG, and convert straight off the card, never even seeing my camera's Raw files. Your CRF (Common Raw Format) is certainly the only sensible end-point in all of this.
But ... I support OpenRAW, and link to this website from my own, despite having this reservation about their solution. There are probably two main reasons for this:
1. From your last paragraph: "publicizing proprietary formats is ... a stop-gap solution for existing formats". Until the cameras get things right, we still (unfortunately) have to play catch-up. I note that the 3.1 DNG Converter manages to copy and preserve the private data from Canon and Nikon cameras into the DNG, even though it may not decode it. But it doesn't for my Pentax (yet). It costs Adobe effort to work out how to do it. Had Pentax published its format, I suspect I would by now have that private data preserved.
2. During the recent fuss over the encryption of the WB in the D2X, what made so many people irate wasn't the proliferation of formats. It was the attempt to hide some of the data. "Lack of openness" is a better lobbying position than "lack of standards". This website has an excellent name, and should be a powerful lobbying force, gaining attention and hopefully creating a climate for change.
So, what I believe is that there is really only one solution, but it has more than one component, probably running in parallel. Sort out the current mess, and also head for a CRF. To some people, "published formats" is THE long term answer. No! To me, "published formats" is a way of getting the best quality DNG Conversion, and is just a short/medium term PART of the answer.
You hint at something that is lacking in most discussions. What is in it for the camera manufacturers? You say "If ... I can read the raw files from a camera that was released to the public 3 weeks earlier, I'd say we will be where we want to be". Make that "released today", not "released ... 3 weeks earlier" - there is no reason why not.
Any camera manufacturer that doesn't intend to lock-in its users to its own software wins by using DNG. The FIRST major camera manufacturer to realise this wins big-time! Who is promoting this message?