Following a series of recent discussions about whether or not it was possible to recover files from XQD cards that have been formatted by mistake I have obtained some clarification from Sony of what can or can’t be done.
This information is specifically for XQD cards and the PXW-FS7 but probably applies to most Sony cameras and also SxS media. I’m not sure about SD cards.
Formatted In-Camera:
The bottom line is that if you format the card in the camera you will not be able to recover any previously shot material. An in-camera format completely erases everything on the card. This is done to ensure that material shot on the cards cannot be recovered by another production company in the case of card or camera rentals. So there is no point in attempting any form of data recovery on a card formatted in the camera as there is nothing recoverable left on the card.
Formatted by a computer:
When you format a card with a computer it is possible that the material will still be on the card. However different operating systems handle the formatting of the cards differently, so there is no guarantee that the data will be recoverable and often it won’t be recoverable. For very important material it may be worth attempting to recover the card. Sony may be able to assist with this in some cases.
Deleted Clips:
Clips deleted from a card can typically be recovered provided they have not be recorded over by a later recording. Again Sony may be able to assist with this.
Delete or Format?
Based on this new information from Sony I may be adjusting my workflow. My own workflow has always been to off-load material from a card. Then to do a parity check to compare the original files on the card and what is now on the hard drives. This checks not just the file size but also the general structure of the files so should pick up most problems with any copies. My last check is then to skim through the files with Catalyst Browse or my edit application to make sure the clips are there and playable. Only then do I format a card. In light of this new information I may use my computer to delete the clips from a card rather than format it. Of course this will only ever offer some benefit if the card is not recorded on again causing the previous files to be over written, but it might add an extra chance of data recovery should the backups get lost or some other disaster occur. From time to time I would format the cards in camera as this helps keep the cards in the best possible condition.
Any of the Sony cameras that use SxS or XQD cards include a media check and media restore function that is designed to detect any problems with your recording media or the files stored on that media.
However the media check is only normally performed when you insert a card into the camera, it is not done when you eject a card as the camera never knows when you are about to do that.
So my advice is: When you want to remove the card to offload your footage ensure you have a green light next to the card, this means it should be safe to remove. Pop the card out as you would do normally but then re-insert the card and wait for the light to go from red, back to green. Check the LCD/VF for any messages, if there are no messages, take the card out and do your offload as normal.
Why? Every time you put an XQD or SxS card into the camera the card and files stored on it are checked for any signs of any issues. If there is a problem the camera will give you a “Restore Media” warning. If you see this warning always select OK and allow the camera to repair whatever the problem is. If you don’t restore the media and you then make a copy from the card, any copy you make will also be corrupt and the files may be inaccessible.
Once the files have been copied from the card it is no longer possible to restore the media. If there is a problem with the files on the card, the restore can only be done by the camera, before offload. So this simple check that takes just a few seconds can save a whole world of hurt. I wish there was a media check button you could press to force the check, but there isn’t. However this method works.
It’s also worth knowing that Catalyst Browse and the old Media Browser software performs a data integrity check if you directly attach an SxS card or XQD card to the computer and access the card from the software. If a problem is found you will get a message telling you to return the media to the camera and perform a media restore. But if this is some time after the shoot and you don’t have the camera to hand, this can be impossible. Which is why I like to check my media in the camera by re-inserting it back into the camera so that it gets checked for problems before the end of the shoot.
I promised I would re-visit some of my Picture Profile stuff. I thought I would start with this one as it is one of the least well understood settings. It’s effects are quite subtle, but it can mean the difference between a noisy picture and a clean image, but also between a sharp image and a soft image, in particular in areas of subtle detail or low contrast detail such as foliage, grass and textures.
Crispening is a part of the detail correction circuit. It does not in itself, as it’s name suggests (at least on an EX of F3) make the image “crisper”. What it does is control the contrast range over which the detail circuit operates. Basically it sets the threshold at which detail correction is applied to the image, which in turn can make the image look a little sharper or less sharp. The apparent sharpness itself is controlled by the Detail Level and Frequency controls.
Why is this useful? Well it allows the user to choose whether to opt for a cleaner looking image or a sharper looking image. An important consideration is that this adjustment does not change the actual resolution of the image or the noise level of the camera, but it does make subtle details in the image more or less enhanced and as noise is also a subtle, even if unwanted detail within the image it will also make noise more or less enhanced, thus more or less visible.
Imaginary waveform showing real picture information plus noise.
In the first illustration I have drawn an imaginary video waveform signal coming from the camera that contains a mixture of noise and both subtle and more obvious picture information. The bigger the up/down change in the waveform the more obvious the change in brightness (and thus contrast) on the monitor or TV would be. Throughout the image there is some noise. I have indicated the noise level for the camera with a pair of red lines. The EX1 and EX3 is a moderately noisy camera, not the worst, nor the best for an HD camera, but pretty good in it’s price range. So if we can do something to make the noise less obvious that would be desirable in many cases. Crispening can help us do that. Crispening ONLY has an effect when you are applying detail correction to the image. It sets the threshold at which detail correction is applied. The default setting on an EX is zero.
If we reduce the crispening setting, lets say to -60, it REDUCES the threshold at which detail is applied which generally makes the pictures look sharper. Looking at the second and third illustrations you can see how if you reduce the threshold too much then detail correction will be applied to even the most subtle changes in the image, including the image noise. The little black spikes I have added to the diagram illustrate the way the detail “enhancement” will be added to both noise and subtle contrast changes as well as larger contrast changes.
Black spikes represent detail correction being added to real picture information and noise when crisping set to -60.
This will make the pictures look more noisy, but… and this is important… it will also help bring out subtle low contrast textures in foliage, skin, fabrics etc. A area where perhaps the EX1 and EX3 don’t do terribly well.
If you want a clean image however where noise is less visible, then raising the crispening level to a high positive value, lets say +60 will increase the threshold at which detail correction is added, so signal changes will need to be bigger before detail correction is applied.
Much higher crisping threshold when set to +60
With a high positive number the image will look cleaner and less noisy, but you will loose some enhancement in textures and low contrast areas as these will no longer have detail correction applied to them. This can lead to a slightly muddy or textureless look to tress, grass, skin and fabric.
The real problem areas are the subtle textures and low contrast areas (circled in orange) where the true image detail is barely above the noise level. It’s very difficult to bring these out without increasing the appearance of noise. Unfortunately there is no clear answer to how to set the crispening level as it will depend on what you are shooting and how much noise you can tolerate. I tend to have crisping set between +10 and +30 for most things as I do tend to do a fair amount of grading work on my footage. When you grade noise is often the limiting factor as to how far you can push the image, so I like to keep noise under control as much as possible. For green screen and chroma key work I push crispening up to +40 to +60 as this helps me get a cleaner key, especially around subtle edges and hair.
Problem areas circled, subtle textures get lost if detail level set too high, although image looks much cleaner.
If I am shooting exteriors and scenics with lots of foliage, grass etc then I will sometimes go down to -30 as this helps bring out the subtle textures in the leaves and plants, but this can make noise a little more pronounced, so it’s a trade off. And that’s what Crispening is all about, trading off subtle textures and detail against more visible noise. Ultimately only you can make the choice as to which is more important, but the Crispening level control gives you that choice.
Click on the link above to download a set of my latest scenefiles. Un-zip and copy to the root of an SxS card, the in the file menu load the files.
These are mainly matrix tweeks. neut2 is one I like that gives rich primary colours while still reasonably true to life. Cine1 is a sudo filmic look Film1 is meant to emulate well saturated film stock DSC-1 is based on Chroma-Du-Monde chart for accurate daylight color Neut is my first matrix tweak for a less green look and warmer skin tones.
What is “Crispening” and how does it effect the picture?
Crispening is one of the adjustments you can make in many of Sony’s video cameras that adjusts the way the image is sharpened via the detail correction circuit. On an EX1 or EX3 it is in the Picture Profiles section. If use wisely Crispening can be used to help deal with camera noise by making it less visible, thus giving a cleaner image. Crispening works across the entire luma (brightness) range. It’s really difficult to explain how the level adjustment works, it is a threshold adjustment for the detail circuit, but I’ll have a go anyway.
First off lets consider how the detail circuit works. The camera uses delay circuits to compare how the brightness (luma) levels of adjacent pixels are changing, both from left to right and line by line. If the circuit sees a rapid change from light to dark or dark to light (or light to lighter, dark to darker etc) the circuit regards this as an edge and detail correction is applied by brightening or darkening the transition, exaggerating the edge. This is seen in extreme cases as a black or white halo around edges.
On the EX cameras crispening works by adjusting the threshold at which the light to dark transition between pixels triggers the application of detail correction. So when you set a negative number, say -99 even the slightest luma difference between pixels will have detail correction applied. Set it to +99 and it takes a much greater luma change to trigger the detail circuit.
What you need to understand is that if you set crispening such that the threshold before detail is applied is 100mV (for example) then between 0v (black) and 99mV little to no detail correction will be applied, keeping blacks clean by not applying detail correction to any noise with an amplitude less than 100mV. But if there are subtle textures in the image, going say from 500mV to 599mV (mid tones) then no detail correction will be applied here either, so the image will appear a little softer, only larger luma changes of more than 100mV will have detail correction applied. These small luma changes can be anywhere within the full luma range and it is not confined just to the darker parts of the image.
Raising the crispening level setting to a positive number raises the threshold at which detail is applied to the image, so a high number prevents detail correction from being added to small luma changes. A negative number means that detail correction will be applied to smaller luma changes, this increases the appearance of noise but also makes textures appear sharper.
One thing to consider is that the noise the camera produces is not only in the blacks. If the noise amplitude (level) is for example 5mV, then if you have a subject at 500mV (mid tones) it will still have random 5mV noise added to it. It just tends to be that noise is most visible in the blacks as 5mV of noise on a 5mV (very dark) signal is modulating (varying) the signal by 100% so it’s quite obvious, however 5mV on top of 500mV is only 1% so less obvious, but still there and still visible.
You should remember that the cleaner you can make the recorded image the less stress there is on the codec. This in turn means less mosquito noise and macro blocking giving an image that looks cleaner still and grades better. I struggle to see the difference between crispening at 0 and at +20 in most normally exposed shots, but if I look closely I do see less noise in shadow and low contrast areas. Low contrast areas tend to have little detail anyway, so being able to clean these up a little helps in post production.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
