OpenRAW

Look at the DNG SDK

Although the DNG SDK doesn't do all you describe, for example it doesn't support embedded rendering code, it probably provides a lot of lessons.
http://www.adobe.com/support/downloads/dng/dng_sdk.html

While you are concerned with proprietary raw file formats, any scheme should also work for raw files originating in DNG format. After all, some cameras use DNG as their native raw format. And many photographers choose to use a DNG-based workflow even where their cameras don't output DNG.

The DNG SDK provides an API for reading, writing, rendering, and validating, DNG files. Perhaps the "reading" API could, instead of reading a DNG file, read another raw file format and convert it to DNG across that API? In other words, given that conversion from proprietary raw to DNG is possible at the file level, could it also be used in-store and offered back across the API?

I wonder how important your proposal to embed rendering code actually is? Raw converter suppliers prefer to use their own code, not the camera manufacturers' code, which is why few if any of them do conversion using the SDK supplied by some manufacturers. They therefore need access to the raw image data.

Hi Barry,

Developers can use their own algorithms on raw data, in the conventional manner - for which they just need to use the API's 'unwrapping' of the original raw file that is embedded within the API's file format. The API wouldn't remove functionality from an existing workflow, because the base-level API would support methods to extract the raw file as a whole, or segments of it (assuming a common mappable structure) as required.

Also don't forget that as an image processing developer, you could choose to write methods that read the raw file directly, so you can embed your own code to interpret the image as you wish. If there is already code within the wrapped file, then as a developer you can choose to use the original, or override it with your own improved or preferred algorithms, or add your own algorithms to the file.

To answer your point of "I wonder how important your proposal... [embedded rendering code]... is...", I suggest that without the embedded code, the file is very susceptible to being unsupported and unreadable in the medium and long term. At least with the code, there would be an industry standard for extracting the information without needing to know anything about the camera. Further, if the camera supported the API natively when writing files, then updates to rendering code would be released implicitly within the BIOS updates. Actually, the rendering code would be significantly smaller than the camera's alternative JPEG (portion of RAW+JPEG) image, and require less on-board processing to write the files.

It would be possible to support DNG too, though I expect the code to be quite 'fat', and I doubt that every aspect of the DNG format is openly documented. Supporting DNG from the API requires political will from all sides. All considered, I'm not yet sure of the value of wrapping DNG files, unless that's what the camera natively generates without losing any purity.

I realise that I've not directly answered some of your DNG points - which I accept - rather I have addressed the problems that you foresee with my API. I'm keeping an eye on DNG - If I believe DNG is right for OpenRaw, then I would gladly recommend that and drop my API. However, I believe that DNG, although a clever attempt to address current technical workflow needs, is not future-proof enough, is not open, and does not address the pertinent issues reported by the OpenRaw group. - JV.

'Coded Raw' project (API)

The project (for the API detailed in a previous post) now has a web page, at http://www.johnvalentine.co.uk/k00007.html

To progress, the project will need some volunteer help from java experts, and input from photographic experts and application programmers, particularly those involved in embedding information in files, and dynamically loading byte-code for execution. However, some more detailed concept work needs to happen first.

Stay tuned, and don't be scared to declare an interest or point out potential flaws. We should know soon if this project 'has legs'.

I'll keep any updates on my site (not here), as I don't want to flood this site with comments that look like adverts.

Cheers,
John Valentine.

DNG addresses OpenRAW -

John, I'll respond to a few of your points:

JV: "I suggest that without the embedded code, the file is very susceptible to being unsupported and unreadable in the medium and long term. At least with the code, there would be an industry standard for extracting the information without needing to know anything about the camera".

Remember that we live in a world where all raw files from important recent cameras either start as DNG or can be converted to DNG, and this state is likely to continue. (If a camera manufacturer decided to encrypt its files to prevent this, the situation might change. But such a manufacturer wouldn't cooperate with an alternative proposal such as yours either!)

Imagine a world where all raw files can be converted to DNG, and there is freely-available code that can extract all necessary image data from DNG files and produce images from it. In that world, how important would your proposal to embed rendering code be? What are the advantages and disadvantages of embedded versus external code to handle image data from a common raw format such as DNG?

I am comfortable with OOD as way of hiding irrelevant data details and differences. But if the data details themselves become a standard, there is no need to hide them. The OOD layer can be built on TOP of a documented data layer, and isn't there INSTEAD of it.

JV: "I doubt that every aspect of the DNG format is openly documented".

The DNG specification is openly published. The SDK to access DNG is freely-available, and comprises (115) CPP and H source rather than libraries. About 100 products from more than 50 companies support DNG in some way, including some cameras. Various tools are available to expose DNG files in all of their details.

It isn't simply a matter of whether every aspect of the DNG format is currently openly documented. What matters is whether any alternative to DNG could ever establish itself as even more open than DNG will be at some future date, and how long that would take. Could anything you propose be better documented than DNG at the end of 2006? At the end of 2007? When? Remember - DNG is also making progress. You need to compare where your proposal could be at the end of 2007 with where DNG will be at the end of 2007, not where DNG is now.

JV: "All considered, I'm not yet sure of the value of wrapping DNG files, unless that's what the camera natively generates without losing any purity".

According to a recent InfoTrends study with 1,754 professional photographers, more than 18 percent of photographers use DNG as part of their imaging workflow. That is probably more than professionals use the native raw formats of ANY single camera. That suggests that it has now become more important to support DNG than to support the raw file format of ANY single camera.

JV: "I believe that DNG, although a clever attempt to address current technical workflow needs, is not future-proof enough, is not open, and does not address the pertinent issues reported by the OpenRaw group".

There is no known alternative to DNG that is plausibly as future-proof as DNG. The problem that OpenRAW is tackling is caused by a proliferation of undocumented raw file formats. DNG is designed to solve both the "proliferation" problem and the "undocumented" problem. DNG addresses the future. There is a tendency for OpenRAW to concentrate on current needs, such as getting formats documented, rather than addressing the future, for example by getting proliferation under control and proposing self-contained raw file formats.

DNG is as open as any plausible alternative. (Er ... IS there a plausible alternative?) What aspect of DNG is NOT open? It is obvious that if manufacturers publish their raw file formats, they will also be publishing the formats of DNGPrivateData which is where Adobe put the cameras' Makernotes. If OpenRAW succeeds in getting manufacturers to publish their raw file formats, that will make DNG fully open. (Any lack of openness is caused by the camera manufacturers, not by DNG).

DNG addresses EVERY single pertinent issue reported by the OpenRaw group! Describe one that it doesn't.

Re: DNG addresses OpenRAW:

Barry, Thanks for your response! It will take some time to do justice to your well-researched post. For now, just know that I do not take an anti-Adobe, nor anti-DNG stance; I'm quite happy to see it be successful.

One new point that springs to mind on reading your post is that DNG is the 'most acceptable' format, rather than the 'perfect' format, a situation I am sure will improve with time.

My personal belief is that we need to strive for accessibility of raw data. Documentation is a means to that end, as is DNG, as is my Coded Raw API proposal (it's even possible that my proposal for 'embedded VM code with standard interfaces' could become part of the DNG specification). After accessibility, future-proofing becomes an objective for me.

After a file format stops being the current favourite, and when applications stop supporting it, that's when documentation is vital - to help decode the data, or re-engineer a parser that can interpret it. I see abstration and embedding the 'difficult part' of understanding a raw file as useful solution to the problems with the longest term. On a basic level, this means extracting the original file (only partly useful), streaming the sensor data (useful), rendering images as raw data in standard or easily-understood format (vital), extracting meta-data (useful) and rendering alternate images (useful).

I am not pretending to understand eveyone's needs (that's why OpenRaw and other bodies take surveys), but some aspects of file design and the software life cycle are inevitable, no matter what the needs or typical working habits of the users.

BP: "...all raw files from important recent cameras either start as DNG or can be converted to DNG, and this state is likely to continue. (If a camera manufacturer decided to encrypt its files to prevent this, the situation might change. But such a manufacturer wouldn't cooperate with an alternative proposal such as yours either!)"

I'm nervous about the encryption situation too. My Coded Raw API addresses that, by giving the camera developer the chance to write the embedded code that generates the renders and supplies the raw data and meta-data to the application. The camera developer would own the IP rights over the embedded code, and there's no need then for application developers to reverse-engineer the file format (possibly in a manner that does not meet with the approval of the camera maker), therefore no need to encrypt the data (and if it is encrypted, then the embedded methods would decrypt it).

BP: "...What matters is whether any alternative to DNG could ever establish itself..."

Agree. Sorry, I'd have got away with that point if I had made it four days earlier :o) However, nothing happens until it happens.

BP: "DNG is designed to solve both the "proliferation" problem and the "undocumented" problem. DNG addresses the future."

Good. Like any other format, it must win majority approval to be successful, otherwise, it becomes 'just another format' to confuse the industry - DNG seems to be gaining ground in that battle. I have no doubt that DNG is doing the best job of storing data that is common to all instances of use.

The concern I have for the future of DNG is that MakerNotes / PrivateData might further obfuscate the data (in the manner of encryption, or just a tangled proprietary mess), moving the current problems into a universal container. At least there is hope for standard data.

The other concern I have is that DNG data is still proprietary in most cases (both in terms of non-standard data, and cross-application re-usability), and seems to take a lot of work to implement from the developer's viewpoint (I might revise that view in future, as I investigate the SDK further). There have also been concerns expressed here about restoring the original raw file from the DNG format.

I think the Coded Raw API design addresses these issues, but like you say, it would take time to develop, and significant investments of time and cash have already been made in DNG, which is gaining more of the territory. Rather than just rolling over and accept that DNG will get there eventually, I see no harm in encouraging alternative ideas (whether competitive or co-operative) to tackle problems in different ways - it will happen eventually!

Relplies to a few points -

JV: "One new point that springs to mind on reading your post is that DNG is the 'most acceptable' format, rather than the 'perfect' format".

That sort of comment doesn't provide a basis for identifying what problems exist, and for solving them. For example, if the problem is that DNGPrivateData is not documented, then the answer is to ask camera manufacturers to document it, or document their Makernotes. (I believe the main impact of OpenRAW, if it succeeds, will be to hasten the progress of DNG, by improving the quality of DNG conversion).

JV: "After a file format stops being the current favourite, and when applications stop supporting it, that's when documentation is vital".

Documentation is important long before that. Look at the OpenRAW survey, and the proportion of photographers who, (some or all of the time), use products other than those supplied by the camera manufacturers. (Only 18% use the manufacturers' products ALL the time). That other software should be based on documented formats, not reverse-engineering. This is one of the reasons I support OpenRAW, of course - and perhaps I am misunderstanding the emphasis you appear to place on "After ...".

JV: "by giving the camera developer the chance to write the embedded code that generates the renders and supplies the raw data and meta-data to the application.... there's no need then for application developers to reverse-engineer the file format ... therefore no need to encrypt the data".

This appears to miss the point about why a camera manufacturer would encrypt anything, if it isn't to prevent access to "the raw data and meta-data" by applications! What else would they be hiding? The danger is that, if manufacturers are offered a get-out called "you can provide embedded code instead of having applications access the raw file directly", this would thwart OpenRAW's objectives of full documentation of all raw image data.

Would that "raw data and meta-data" be supplied in a standard format, so that one set of documentation for it would work for all cameras, or would per-camera-model documentation still be needed? If the former, why not store it in the raw file in that format to start with, or have a separate converter that will turn the camera's raw files into a standard format that can then be archived?

JV: "The concern I have for the future of DNG is that MakerNotes / PrivateData might further obfuscate the data ... moving the current problems into a universal container".

This is a problem that has often been raised by OpenRAW, and I don't believe it is a valid objection to use of DNG. If OpenRAW succeeds in its objective to get Makernotes documented, this documentation will serve for DNGPrivateData as well. For example, if, through OpenRAW's efforts or otherwise, Canon publish the Makernote format for the 30D on 28 August 2006, then on 28 August 2006 the DNGs for the 30D will be fully and openly documented, including the DNGPrivateData, and it will cease to be an issue with OpenRAW. OpenRAW can't succeed without also eliminating its objections to DNG!

JV: "The other concern I have is that DNG data is still proprietary in most cases".

If that is about the DNG format itself, that is typically how formal standards get started. Standards bodies tend to adapt from (working) specifications provided by companies and other bodies. When ISO wanted a standard format for raw files, it started with TIFF, owned by Adobe. When they wanted to standardise a means of publishing documents, they started with PDF and evolved it into PDF/A and PDF/X.

If it is a comment about Makernotes, (hence DNGPrivateData), then it is a minor issue, both for reasons identified above, and for the simple fact that you don't actually need to use the DNGPrivateData to render the image. I believe that most raw converters don't need to use Makernotes at the moment.

JV: "DNG data ... seems to take a lot of work to implement from the developer's viewpoint".

True, if implemented fully, although less than implementing code for each individual camera comcerned. But it can be a one-off activity - once someone has written code to access DNG data and render images from it, others can use that code if they want. And such C code is published - dcraw by Dave Coffin, which also renders from DNG. When a raw converter supports DNG properly, whether using its own code or dcraw, it supports DNG for lots of cameras, including future cameras.

As an exercise, it might be worth putting some real numbers into a calculation of how much embedded code will be needed over (say) 10 years, if raw-capable cameras continue to appear at a rate of 50 or more per year, and how much DNG-access code will be needed if raw-handling software products appear at whatever their current rate it, perhaps 10 or 20 per year.

It is the reality of numbers that can sink some attractive-sounding proposals. It has always been a problem with OpenRAW's main proposal that it isn't plausible for 50 or more cameras per year over many years. That is why you are right to think in terms of a standard format - proliferation has to be controlled too.

I am not against the idea of a raw file format with embedded code. But it needs to take into account the real world, which includes DNG in it. That is why "co-operative" rather than "competitive" is the stance to take.

I am glued to the unfolding OpenRAW survey report! I am pondering the implications of domination by "ACR" and "camera manufacturer software". I would love to know the trends here - what will it be in (say) 5 years time? That will surely influence the take-up of any solution to "the raw problem".

Parallel efforts.

"The concern I have for the future of DNG is that MakerNotes / PrivateData might further obfuscate the data"

The concern I have is that the camera companies are under no obligation, or pressure, to publish their private data or maker notes. There's just no leverage. I believe that even the weight of the OpenRaw survey is insufficient to achieve total free and unfettered access. Why would it? Approx 20,000 respondents vs. Millions of consumers? While statistically significant, camera makers would have to have their oligopolies impacted before they felt any pressure to comply.

So where does that leave us? Competition!

I endorse a goal of free and unfettered access, but I feel that goal is best obtained through leverage--not pleas. There is no reason why concurrent efforts can't exist. It doesn't need to be an all or nothing scenario. If DNG, or some other competing format is the first to achieve a critical consensus of inter-vendor compatability, then camera makers would have little choice but to embrace an entry strategy that favours their wares.

An entry strategy with a lower threshold that doesn't give away the secret sauce would be much more palatable than one that does. DSLR makers have gone to extreme measures to distinguish themselves rather than just become another commodity. At the very least, a mechanism to protect makernotes and private data in an open format would consolidate that data into a specifically mapped field, as opposed to stringing it and/or encrypting it through a number of fields. With time, and sufficient market demand, the sauce could be unlocked from the vault.

The difference I see in the proposals currently offered, are on the one hand, an all or nothing approach for full compliance vs. an incremental approach to comply with maturing demand. I will gladly embrace the latter than wait indefinately for the former.

Developer consensus will accelerate the development of tools needed to apply inter-vendor compatability. However community support for compatible tools will catalyze efforts to achieve camera vendors to jump on board.

the born 2 design
design guy

A sensible approach -

nunatak: "Developer consensus will accelerate the development of tools needed to apply inter-vendor compatability. However community support for compatible tools will catalyze efforts to achieve camera vendors to jump on board".

This is about the evolution of what you call the "ecosystem". Lots of parties, all pursuing their own agendas, all influencing one-another because they change the environment in which the others make decisions. Therefore, month by month, things change almost imperceptibly, but over longer periods, such as a year, things change out of all recognition.

My analysis is that the ecosystem can be broken down at the "top level" into the following, (although other decompositions are possible, of course):

1. Photographers themselves. (Obviously many sub-categories!)

2. Users of photographs, one of your areas of expertise. I would include "providers of services to photographers" here, although that may be subtly different.

3. Independent software developers, including suppliers of raw converters, viewers, managers, tools & utilities, etc.

4. Camera manufacturers, with the important sub-categories within their product ranges of "cameras & backs" and "supporting software".

5. Librarians and other archivists. Some might put these into "2", but I think they have a usefully different perspective.

It has been said "technologies take much longer to penetrate the marketplace than their advocates expected; but once they do, they have a bigger impact than their advocates predicted". (Think of the laser!) I think it is because advocates often don't consider the need to carry the whole ecosystem with them, and then fail to predict the more global impact of the resultant revolution in the ecosystem.

I consciously track the progress of DNG according to the above 5 categories. They can also be used for predicting and tracking other ideas in this arena. I think they are close to what I have seen you write here and elsewhere. The categories can also be used to devise programmes to change the ecosystem - they may identify separate projects needed within the programme. They also show the need, which you identify, to take a step by step approach, not expect everything to happen at once, and not to worry too much if the early stages are not up to the target quality.

A deeper insight than most -

Barry, I think you have a deeper insight into the ecosystem than many of the other players. Of course I have deduced that from my observations that you like to explore perspective from outside the fixed niche. :-)

It's inevitable explorers are the first to expose themselves to competitive perspectives. This is a theme as common to Marco Polo as it is to Witgenstein's ladders. One needs to first climb out from where they are to see where they have come from.

If I were to map "my understanding"of the various niches onto the ecosystem--which, incidentally, may be just as fallible as any other-- it would be as follows:

1. Product-centric. This group engages in the trade of photographs, as a commodity to complement the product they are promoting. Their goal is NOT usually to promote the photographer, unless the photographer is the product itself. They are at the top of the food chain as they design the product that pays for the tools, and services beneath them.

2. Photographer-centric. These are the artists, the narrators, the worker bees, the content producers that manufacture the raw product -- sometimes even the final product.

3. Service-centric. These are the value added group. It includes subgroups such as
a) managers--i.e archivists, librarians, indexers, navigators of knowledge and assets.
b) producers--i.e technicians, production workers-- pre-press and printers.

4. Software-centric developers, including suppliers of raw converters, viewers, managers, tools & utilities, etc. This group is really service-centric -- or will evolve into a service subgroup once it's pioneering role is over. Yet in the early stages of a new ecosystem it plays a crucial role in establishing frameworks. Any delays or obfuscation plays to the benefit of maintaining it's significance to the cost of the whole community.

5. Camera-centric. Once again I'd classify as a pioneer group and any delay plays to the benefit of maintaining it's significance. Eventually though, as was the case with film, it will find a new home in the service-centric group as the camera returns to being a commodity item.

Because of their pioneering expertise, we currently have 5 and 4 dominating the ecosystem. As long as they make the ecosystem more efficient, one may say they are the tail wagging the dog. However, in a world where commerce and economy is the great in-equalizer, the significance of this dynamic will not be allowed to continue indefinately.

Whichever way one looks at them, all catagories are ego-centric. Each feel their role is more significant than the other. Which group at any one time is more significant in a dynamic ecosystem is debateable, but the one thing that isn't are the resources which subsidize the economy of the whole ecosystem. :-)

the born 2 design
design guy