OpenRAW

You addressed almost the main reason why OpenRAW exists...because of the transient nature of file formats, only an open documentation of these
file formats help them to survive into the future.

But you also mix up 2 issues here, the data is not bound to a specific medium like in your Ampex example and the data is not stored analog, eg. can be recopied lossless to different media.

Any RAW format can survive - properly stored - into a far future and with an documentation, any programmer worth his/her salary can write a new file parser and open the images in the future and with then actual operating systems and hardware.

Hope this makes our mission more clear.
--
http://www.openraw.org/

I'm a new guy to this very interesting forum and haven't read all the no-doubt excellent comments yet.
My main area of concern is for the permanent preservation of digital records of any form. Hopefully I'm not repeating the obvious.
Part of my current work is looking at a "pre-ingest" strategy which someday (hopefully sooner rather than later) will lead to the establishment of a repository for maintaining long-term/permanent access to digital records.
A recurring theme in early investigations is file formats and the realisation that unless software manufacturers make their software open-source, quality and data may be lost never mind possible legal implications for proving integrity and authenticity of migrated versions.
So far, I am of the opinion that proprietary raw formats will not be accepted into a system designed to permanently preserve material as the overhead in managing such formats over time may be insurmountable.
Even Microsoft have relented over their Office formats.

Juergen, thanks for your reply.

You are right, my examples are more directed to the messenger rather than the language of the message.

However, my point is why would you want to write a new program in the future when we already have such programs today?

If you want to have access to all your old RAW files all you need to do is keep a copy of the software running.

Do not get mw wrong, I am for Open RAW . . . it would definitively make things easier, but I can not see it making things MUCH easier. :-(

As experience shows, in the future there will be new computers, new operating systems and old software becomes obsolete...camera makers will support legacy files only for that long, until becomes to much of a burden or expensive (or they go out of business like Konica-Minolta), so the future for current RAW files is very, very uncertain.

The documentation of current RAW files will make sure that new software can be written if the need arrises.

There is another advantage of new software, new algorithms get developed which probably get more details out of legacy files and since computer gets faster, these algorithms might not be able to run before on old hardware.

To give you an simple example, I converted all my RAW files since 1999 with then actual Version 1 RAW converters into JPG and stored them in my archive along with the original RAW files.

Converting these RAW files today with Version 5 RAW converter software and comparing them to the then state-of-the-art JPGs I created shows just how much all the algorithms got improved and how much better a converted image from an legacy RAW file can look like.

My RAW files originate only from 1999 upwards, which is just 7 years, but now think in longer terms like 10, 20 or 50 years.

After a certain time, every image becomes an document...you might lose the interest in your own images, but not your family or historicans.

Personally I manage our family archive of images, with some older than 100 years old and I don't want to see my current images destroyed by the greed of camera makers.

Juergen
--
http://www.openraw.org/

Open spec, not open source

You say "... unless software manufacturers make their software open-source ...". Manufacturers' source is not needed to solve the problems here. We shouldn't (and don't) need source code to be supplied by manufacturers, and it is unrealistic to expect it.

We need openly documented specifications that LOTS of people can write code to, without relying on any particular manufacturer. If any manufacturer supplies some source, that is a bonus. (For example, Adobe provide a freely-available SDK for DNG).

And we need "critical mass" so that there are economies of scale writing that software. We want our raw image data held in a format that has longevity, so that people will continue to be motivated to develop and supply software to handle it. If file formats proliferate, how would the "open source industry" continue to handle all the varieties? Variety must be reduced. We need a common raw format.

No need to rely on old s/w

You say "I know that probably one day I'll have to keep some old software running just to have access to all my RAW files, and I have no problem with that issue".

That only applies if you use file formats without longevity, or that cannot be converted to a more up to date format. So we need to plan for longevity by using formats that can support lots of cameras from lots of manufacturers. Then build in "evolveability" and "convertibility" so that as the formats evolve for new technologies, existing raw images files can either continue to be used, or be migrated to the up to date formats if necessary, carrying the raw image data forward.

DNG is designed to solved this problem. Why settle for less?

The software we need ...

You say "The documentation of current RAW files will make sure that new software can be written if the need arrises".

The software we need to solve these problems are better DNG Converters. Combined with better software to process DNG, what more do we need? The main aim of better documentation of current RAW files should be to improve DNG Converters.

We shouldn't want future software developers to have to wade through lots of published specifications for lots of camera models from lots of manufacturers. There will be limited motivation for them to do so, given the number of raw-capable camera models released every year.

If manufacturers document their raw file formats, we can achieve DNG Converters that not only include all the original data in a suitable form, but also have all of it documented. (The reason the data within DNGPrivateData isn't documented isn't a fault with DNG - it is because the manufacturers haven't documented it. If they do, that resolves this problem with DNGPrivateData, and people will stop complaining here about it).

Future s/w is also important

You ask "my point is why would you want to write a new program in the future when we already have such programs today".

First, YOU may have the program today, but those who want to process the images in future may not. Archivists and librarians may not be able to get hold or, or run, the necessary tools in future.

Second, what matters is whether the images can be processed using your future choice of workflow and tools? Otherwise, it may be too expensive or too much trouble to do so.

We need open documentation of image formats. I develop embedded software, mainly for the metal processing industry. I spend a lot of time having to reverse engineer communications links for legacy equipment. These links are proprietary and undocumented. It's an expensive and time consuming task.

I would hate to see the same thing happen to my digital images. We need camera manufacturers to document their formats so we have access to these images in the future. We can't rely on the existing software being available in the future. Future operating systems may not even run the software correctly or at all (ever tried running 8-bit DOS comms links on XP?). By giving access to the data format it will always be possible to create software to process these images. I don't particuraly care what format the data is in, DNG or proprietary as long as the documentation is available.

Why open documentation?

That is not a joke - here are 2 questions to ponder:

1. Today's software supported today's raw files without the aid of open documentation of those formats. Why can't future software do the same?

In fact, it will be an easier task, because today's formats are partially documented as a result of the reverse engineering that has already been done. (For example, the freely available source code for dcraw contains a lot of information about how to read today's raw files).

2. Why should we believe that future software will support today's raw files even if they are openly documented?

The evidence suggests otherwise. Major products such as Aperture & Bibble & Capture One & ACDSee don't support all camera models with openly documented raw file formats, even today while those cameras are current.

Software developers use cost-benefit analysis to decide which camera models to support, and undocumented raw files from Canon & Nikon are much more likely to be supported than documented raw files from Hasselblad-Imacon, Leica, Pentax, Samsung, Ricoh, etc.

Future software products are more likely to support raw file formats if the files are relatively common, rather than rarer ones that are openly documented. (It helps if they are both common AND openly documented!) So, we want camera manufacturers to use a common raw format in future, to increase the "economies of scale" and make it more attractive for future software to support it. And we need to be able to convert today's other raw formats to that common format, so that they too will be supported.

The primary need for open documentation of today's raw file formats is to enable high quality converters to be developed as soon as possible, rather than to expect future products to use the documentation in decades to come.

Longevity? Any digital file is by nature "inmortal". Your issue with converting one format into another is really an imaginary one, as any format can be converted into any other format, it is all a matter of developing the converter. Maybe developing a converter is beyond your capability, I know it is beyond mine, but that hardly makes it an impossibility.

You are right, but . . . it seems that you are more interested in getting more out of your old RAW files, than really making them available in the future. If you wanted to have access to such old files you just need to keep an old pc running Windows 3.11 (and the appropiate RAW software). I mean, are you going to try to keep enlargers in the market for the next 100 years just to have access to you collection of negatives? Or, are you going to digitized them (i.e. convert/scan them to a new format to keep them available)?

Thant was the whole point of the question in my original post: What is the real issue?

It seems that availability of the RAW files is just a smoke screen for the real intention behind most of Open Raw: extracting better images from old RAW files. There is nothing wrong with that, and I think it is a better, more upfront goal.

By "archivists and librarians" I guess you mean common-joe archivists and librarians, since I sincerely doubt that any serious and/or profesional archivist/librarian would not have the means to access old RAW files.

Your second popint is a valid one, but like a mentioned in a previous comment, it is more directed to getting more out of RAW file, than really making them available.

Professional archivists, etc

No, I don't mean "common-joe archivists and librarians". I mean the sort of archivists and librarians who work at national and international level, researching strategies for archiving and writing papers on the topic. (That is where I am getting some of my information from).

An example is the Library of Congress, who list DNG as their only recommended raw file format.

The real aim doesn't matter

It is unrealistic (and risky) to need to keep old tools available for this purpose. And, given that we have a simpler and better alternative for digital image preservation, what is the point of doing so?

It doesn't matter whether the aim is to get the same out of them or more out of them. We can solve the problem, so why not do so?

That is not a joke - here are 2 questions to ponder:

Who's joking?

1. Today's software supported today's raw files without the aid of open documentation of those formats. Why can't future software do the same?

In fact, it will be an easier task, because today's formats are partially documented as a result of the reverse engineering that has already been done. (For example, the freely available source code for dcraw contains a lot of information about how to read today's raw files).

The key phrase here is partially documented. While dcraw etc. do a good job, any software based on it is using Dave Coffin's assumptions. While I'm not saying that Dave is wrong, I'd rather know for sure that he is right or not by having the manufacturer's own documentation.

2. Why should we believe that future software will support today's raw files even if they are openly documented?

The evidence suggests otherwise. Major products such as Aperture & Bibble & Capture One & ACDSee don't support all camera models with openly documented raw file formats, even today while those cameras are current.

Software developers use cost-benefit analysis to decide which camera models to support, and undocumented raw files from Canon & Nikon are much more likely to be supported than documented raw files from Hasselblad-Imacon, Leica, Pentax, Samsung, Ricoh, etc.

1. If the software is not available then at least there is more of a fighting chance of developing something if the format is documented.

2. If a format is fully documented then the cost of supporting that format is considerably less than one that is partially documented or not documented at all.

In an ideal world all manufacturers would agree to a common format. Unfortunately we live in the real world. The cost to Nikon/Canon/et al. of retooling all their firmware and support software to use a common format would be prohibative. In other words there is a high cost for very little benefit for them. On the other hand the formats they currently use will already be documented in-house. It's a much lower cost to them to release this documentation. Even this is going to be an up-hill struggle as they are making money from selling their own processing software so again there is little real benefit to them.

The primary need for open documentation of today's raw file formats is to enable high quality converters to be developed as soon as possible, rather than to expect future products to use the documentation in decades to come.

I agree that there is a need for today's formats to be fully documented so that better conversion software can be created. However I also feel that the future-proofing of my images is important. Unless manufacturers are willing to release full details of their proprietary formats then I don't see what real benefit there is to converting current images to an open standard. Unless we are sure that we are converting all real information and that there is no loss involved in the conversion then I for one would be wary of any conversion.

Benefits of common formats

1. What evidence is there that "The cost to Nikon/Canon/et al. of retooling all their firmware and support software to use a common format would be prohibative"?

Several cameras and digital backs already use DNG in-camera. (3 more were announced in September). Some camera manufacturers provide software that supports DNG, for example their own raw converters may accept DNG conversions from for their own cameras, or they may provide their own converters to DNG. These are "proof of concept". Nikon and Canon probably have more resources available than the companies who have already done this. (They could ADD the common format, not necessarily replace the existing one, just as Pentax and Samsung have done with their new cameras).
http://www.barry.pearson.name/articles/dng/products.htm#manufacturers

2. In response to "... I don't see what real benefit there is to converting current images to an open standard" - there are more benefits than just the archival advantage.

If the format is well-designed and openly documented (as DNG is) it is safe for 3rd party products to write to the file without harming the raw image data. Therefore metadata can be accumulated within the file, making them self-documenting. This is known to make some workflows easier. (Other people get other benefits).
http://www.barry.pearson.name/articles/dng/benefits.htm
http://www.barry.pearson.name/articles/dng/xmp_dng.htm

3. In response to "... at least there is more of a fighting chance of developing something if the format is documented" - it needs "critical mass" / "economies of scale" for someone to be motivated to write suitable software. We've seen that even an OpenRAW supporter like Bibble needs that extra motivation.

Although OpenRAW originated with people responsible for Canon & Nikon mailing lists, many OpenRAW supporters use cameras from other manufacturers. In fact, that is probably why many of them are OpenRAW supporters! Canon & Nikon may be the only cases (if any) where there may be sufficient critical mass for distant future software to support today's raw files.

4. People often raise the possibiity that DNG may not have all the essential data for the raw image. But, in response to "... sure that we are converting all real information and that there is no loss involved in the conversion ..." - remember that DNG has ADDITIONAL data in the raw file!

First, Adobe products such as ACR & Lightroom use JUST the DNG data, so it certainly contains what these products, (and others such as Silkypix, Raw Developer, etc), need. It would be useful if, instead of just raising "fear, uncertainty, doubt", those people actually identified the data they are worried about! If it exists, something can be done about it. If it doesn't exist, it would be useful to cease the FUD.

Second, the task for those future software products isn't just to understand the raw file format sufficiently to read it and extract the data. It is ALSO to know enough about the camera model to make sense of that data. What is the colour response of its colour filter array? What is the strength of its anti-alias filter? (Etc). Such information is in a DNG file but typically not in original raw files.

In other words, if the question was instead "is it safe to convert from DNG to a camera manufacturer's private format", we might conclude that too much identified data would be lost in the process! We should be asking whether camera manufacturers' raw files contain sufficient data, instead of just asking that question about DNG.

I'm not arguing that there are no benefits to a common format. It would be great if Canon/Nikon were to announce tomorrow that they were adapting DNG. It's not going to happen though is it?

Let's look at the cost to Nikon (for example) of adapting DNG.

1. Assuming that only new models shoot in DNG (for the moment) then there is the cost of writing new firmware for that model to use a new format. There is a much higher cost there than adapting current firmware supporting current formats to a new model. In other words it is cheaper to make a Nikon D300 support existing Nikon formats than to adapt a completely new format. Why? Because Nikon will already have firmware libraries that can be adapted to a new camera. A new format means a rewrite of these libraries.

2. The current crop of support software (Nikon View, Capture NX etc.) then have to be rewritten to support the DNG format. Again not a cheap exercise although arguably cheaper than a firmware rewrite (based on my experience as both a low-level and high-level software developer).

3. Where does leave everyone who has an older Nikon? Are Nikon then expected to re-engineer all their existing firmware for their existing models?

The same applies to Canon et al. who have closed, proprietary formats.

Again I ask the question, where is the benefit to Nikon or Canon of doing all this work? Unless these manufacturers see a real benefit for the costs involved they are not going to adapt DNG.

Apart from the hit to Nikon and Canon's bottom lines there is also politics to consider. Why should the big two adapt a 3rd party format that they cannot control? If DNG was an ISO standard then there may be more chance of them adapting this format. There is less chance of them adapting a format that is completely under the control of another company.

I am not trying to raise FUD. I endorse the DNG concept but at the moment I don't see it as a practical proposition.

Others have done it!

Arguments that it would be a large cost to Canon & Nikon to support DNG fail because several other camera manufacturers (smaller than Canon or Nikon) already support DNG in some way:

- Some camera manufacturers use DNG in-camera. (Pentax & Samsung offer the user the option of DNG or their own raw format in recent cameras).

- Some camera manufacturers provide their own DNG Converters. Pentax offers this free for all their dSLRs so far.

- Some camera manufacturers supply raw conversion software that can read DNG files for their own cameras.

The argument that Canon & Nikon might not (yet) see the benefit of supporting DNG is true. And the same applies to doing what OpenRAW wants, which is to publish their raw file formats. It is important to keep up the pressure, until they get the message.

DNG is most certainly ALREADY a practical proposition. Lots of photographers use it and get benefit from it. (I've been using it for over 2 years). Just about all the people who advocate DNG in books, etc, use Canon & Nikon - the fact that those don't support DNG isn't a blocker, just an irritant. What matters far more is whether software products, especially 3rd party software, supports DNG, and that is becoming increasingly common. Bibble (an OpenRAW supporter) is an important case to be persuaded.