Tag Archives: recovery

Recovering footage from formatted cards.

Following a series of recent discussions about whether or not it was possible to recover files from XQD cards that have been formatted by mistake I have obtained some clarification from Sony of what can or can’t be done.

This information is specifically for XQD cards and the PXW-FS7 but probably applies to most Sony cameras and also SxS media. I’m not sure about SD cards.

Formatted In-Camera:

The bottom line is that if you format the card in the camera you will not be able to recover any previously shot material. An in-camera format completely erases everything on the card. This is done to ensure that material shot on the cards cannot be recovered by another production company in the case of card or camera rentals. So there is no point in attempting any form of data recovery on a card formatted in the camera as there is nothing recoverable left on the card.

Formatted by a computer:

When you format a card with a computer it is possible that the material will still be on the card. However different operating systems handle the formatting of the cards differently, so there is no guarantee that the data will be recoverable and often it won’t be recoverable. For very important material it may be worth attempting to recover the card. Sony may be able to assist with this in some cases.

Deleted Clips:

Clips deleted from a card can typically be recovered provided they have not be recorded over by a later recording. Again Sony may be able to assist with this.

Delete or Format?

Based on this new information from Sony I may be adjusting my workflow. My own workflow has always been to off-load material from a card. Then to do a parity check to compare the original files on the card and what is now on the hard drives. This checks not just the file size but also the general structure of the files so should pick up most problems with any copies. My last check is then to skim through the files with Catalyst Browse or my edit application to make sure the clips are there and playable. Only then do I format a card. In light of this new information I may use my computer to delete the clips from a card rather than format it. Of course this will only ever offer some benefit if the card is not recorded on again causing the previous files to be over written, but it might add an extra chance of data recovery should the backups get lost or some other disaster occur. From time to time I would format the cards in camera as this helps keep the cards in the best possible condition.

ProRes Raw Over Exposure Magic Tricks – It’s all smoke and mirrors!

There are a lot of videos circulating on the web right now showing what appears to be some kind of magic trick where someone has shot over exposed, recorded the over exposed images using ProRes Raw and then as if by magic made some adjustments to the footage and it goes from being almost nothing but a white out of over exposure to a perfectly exposed image.

This isn’t magic, this isn’t raw suddenly giving you more over exposure range than you have with log, this is nothing more than a quirk of the way FCP-X handles ProRes Raw material.

Before going any further – this isn’t a put-down of raw or ProRes raw. It’s really great to be able to take raw sensor data and record that with only minimal processing. There are a lot of benefits to shooting with raw (see my earlier post showing all the extra data that 12 bit raw can give). But a magic ability to let you over expose by seemingly crazy amounts isn’t something raw does any better than log.

Currently to work with ProRes Raw you have to go through FCP-X. FCP-X applies a default sequence of transforms to the Raw footage to get it from raw data to a viewable image. These all expect the footage to be exposed exactly as per the camera manufacturers recommendations, with no leeway. Inside FCP-X it’s either exposed exactly right, or it isn’t.

The default decode settings include a heavy highlight roll-off. Apple call it “Tone Mapping”. Fancy words used to make it sound special but it’s really no different to a LUT or the transforms and processes that take place in other raw decoders. Like a LUT it maps very specific values in the raw data  to very specific output brightness values. So if you shoot just a bit bright – as you would often do with log to improve the signal to noise ratio – The ProRes raw appears to be heavily over exposed. This is because anything bright ends up crushed into nothing but flat white by the default highlight roll off that is applied by default.

In reality the material is probably only marginally over exposed, maybe just one to 2 stops which is something we have become used to doing with log. When you view brightly exposed log, the log itself doesn’t look over exposed, but if you apply a narrow high contrast 709 LUT to it, it then the footage looks over exposed until you grade it or add an exposure compensated LUT.  This is what is happening by default inside FCP-X, a transform is being applied that makes brightly exposed footage look very bright and possibly over exposed – because thats the way it was shot!

This is why in FCP-X  it is typical to change the color library to WCG (Wide Color Gamut) as this changes the way FCP-X processes the raw, changing the Tone Mapping and most importantly getting rid of the highlight roll off. With no roll-off, highlights and any even slight over exposure will still blow out as you can’t show 14 stops on a conventional 6 stop TV or monitor. Anything beyond the first 6 stops will be lost, the image will look over exposed until you grade or adjust the material to control the brighter parts of the image and bring them back into a viewable range. When you are in WCG mode in FCP-X the there is no longer a highlight roll off crushing the highlights and now because they are not crushed they can be recovered, but there isn’t any more highlight range than you would have if you shot with log on the same camera!

None of this is some kind of Raw over exposure magic trick as is often portrayed. It’s simply not really understanding how the workflow works and appreciating that if you shoot bright – well it’s going to look bright – until you normalise it in post. We do this all the time with log via LUT’s and grading too! It can be a little more straight forward to recover highlights from Linear Raw footage as comes form an FS5 or FS7 compared to log. That’s because of the way log maintains a constant data level in each highlight stop and often normal grading and colour correction tools don’t deal with this correctly. The highlight range is there, but it can be tricky to normalise the log without log grading tools such as the log controls in DaVinci Resolve.

Another problem is the common use of LUT’s on log footage. The vast majority of LUT’s add a highlight roll off, if you try to grade the highlights after adding a LUT with a highlight roll off it’s going to be next to impossible to recover the highlights. You must do the highlight recovery before the LUT is added or use a LUT that has compensation for any over exposure. All of these things can give the impression that log has less highlight range than the raw from the same camera. This is not normally the case, both will be the same as it’s the sensor that limits the range.

The difference in the highlight behaviour is in the workflows and very often both log and raw workflows are miss-understood. This can lead to owners and users of these cameras thinking that one process has more than the other, when in reality there is no difference, it’s appears to be different because the workflow works in a different way.