Part 2: Full HD ‘digital zoom’ using crop – a viable option? HD video with DSLRs – The good, the bad and the ugly.
UPDATED at bottom!
Much in this post is speculation, especially the theories about how the Canon CMOS sensors record video and what is theoretically feasible to achieve with a firmware upgrade. These speculations are of course not unfounded and rely on information given by other users and technology experts. I am not a technology expert or engineer, and I speculate purely from my – maybe naïve – standpoint as a DoP and filmmaker that is keen on the things to come.
Thanks for checking back! Last time, in Part 1 of this two-part series, I looked at the disadvantages of shooting video with these shiny new DSLRs we adore so much. I focused mainly on the problems that are caused by the CMOS sensors that are optimized for still image shooting.
This time, I will focus on the good bits. To be more precise, I will focus on what THEORETICALLY would be possible with DSLRs if the manufacturers or some software crack like Trammell Hudson (with his Magic Lantern alternative firmware for the 5D Mark II) would come up with a new firmware. So let’s call it a little vision. It would be a way to put some of those still image pixels to good use for video shooting.
Video acquisition on a DSLR in practice:
I scribbled up this little graphic to show you how the acquisition of 1080p video works on the example of the Canon EOS 5D Mark II (it works the same way in the 7D or 550D/T2i, with the only difference being that their CMOS sensors have a slightly smaller resolution and are smaller, APS-C sized).
Canon doesn’t have official spec sheets that explain what exactly they are doing to acquire video with these cameras. However, many people who are much smarter than me have come up that this must be the way how they do it. I’ve summed up all the facts I could find on various web forums and posts and summed it up in these graphical interpretations. Please be aware of the fact that much of this is still speculation, but this is the most likely way video seems to be recorded on DSLRs.
When shooting 1080p video on the 5D Mark II, only every third line of the 3,159 lines of the 16:9 area of the sensor is recorded, which results in 1,053 lines (in total, the sensor consists of 3,744 lines, but the additional lines are only used for the different aspect ratio of still images).
Every line consist of 5,616 pixels each and is downsampled to 1,920 pixels, while the 1,053 lines are interpolated to 1,080 pixels. The final 1,920*1,080 raw video is immediately compressed into H.264 in a QT-MOV container and then written onto the CompactFlash memory card.
This constant twisting and wrenching of the images results in awful aliasing artifacts and increase the effect of the rolling shutter, both of which I explained in my last posting.
So let’s look at what this high-resolution CMOS sensors might actually be good for.
The 640*480 pixels crop mode on the Canon EOS Rebel T2i / 550D:
Let’s start with the obvious things. As all you T2i/550D owners out there will already know, our new little DSLR camera sports a new video mode that is lacking from its bigger siblings: the 640 x 480 crop mode.
It just takes the 1.920 x 1.080 resolution and crops everything around it to leave us with 640 x 480 pixels. (At half the data rate, about 23 MBit/s.)
Now you might say this is not an option that is viable for a filmmaker, and you are right. But it could also be seen as a step in the right direction.
I recorded a test clip in the 640 crop mode and found it uses half the data rate of the 1080p mode (23 MBit instead of 44-45 MBit). It might have been useful if it used 44 MBit as well as this would have increased the quality of such a pseudo-telephoto mode quite a bit.
Unfortunately, it also uses line skipping to acquire these images. In fact, it seems to use the same line skipping mode as the 1080p mode, with the only difference being that the majority of lines are disposed during the downsampling process.
The theory: 1080p crop mode – no line skipping, no aliasing, massively increased “focal lengths”
Think about the alternative: Why not make an OPTIONAL crop mode without using line skipping in the process? Line skipping wouldn’t be necessary if you used only part of the sensor anyway. It would even be possible to use the exact same pixels without involving any downsampling or interpolation whatsoever. Of course that could not replace the line-skipping as we still need the whole sensor real estate, but it could be employed as an additional mode and provide us with a “free” additional fixed “focal length” / decreased field of view.
NOTE (this has kept coming up in the comment section): I see this as an OPTIONAL additional feature that would not at all replace the way video images are currently acquired with these cameras. But it could be a feature for sports and other situations where such a “focal length” / small field of view could be extremely helpful.
The recorded image would quite simply be only a 1,920*1,080 pixels area in the middle of the much larger 5,616*3,744 pixels CMOS sensor of the 5D Mark II.
As a result, we wouldn’t suffer from many of the issues deriving from the twisted way HD video is usually recorded on a hybrid DSLR camera. No line-skipping means less aliasing and also less ‘skew’, aka ‘rolling shutter’.
Using a 200mm telephoto lens would instantly give you access to effective 600mm on a 5D Mark II, and getting rid of all the disadvantages of HDSLR video as we know it now.
It would even be thinkable to introduce an “in-between-mode” with line-skipping only every second line. This could give DSLR filmmakers access to two additional telephoto stages with every lens. Effectively, you would cover a much longer area of focal lengths with a single lens.
So the only question remains: Is this technologically feasible? Maybe, maybe not. Geniuses like Trammell Hudson, who developed the alternative 5D Mark II firmware which allows audio monitoring and other goodies usually not available in the 5D Mark II, have said that it might be an option. It might simply not possible with the current generation of cameras because of lacking CPU horsepower, but it might be feasible with future generations – even if the manufacturers themselves won’t allow it, because they’d probably fear to sell less lenses.
Just imagine: One 70-200mm telephoto lens to rule them all, going up to effective 600mm on a 5D Mark II (and of course even further on crop-sensor cameras like the 7D or 550D/T2i).
I would really love to have a discussion about these ideas, I’d love to hear your thoughts!
Also please check out this Cinema5D forum entry that I just discovered – they came to similar conclusions like me, maybe some of these guys want to talk about it right here.
Somebody in a forum brought up that the DOF would change and that it would look like the image from a 1/3″ camcorder. That’s just plain wrong.
The DOF will stay EXACTLY the same. Think about it: The projected image is still the same, therefore the DOF is the same. You only take the middle part of this image and acquire it in 1080p, which results in nothing more than a 3x decrease in field of view > a small portion of the entire sensor area acquired in Full HD resolution without line skipping (1:1 pixel ratio)!
DOF would only be different if you would somehow change the projected image onto a smaller area, but that’s not what we do here.
It’s true that it wouldn’t be as shallow as on a real 600mm lens if you use a 200mm lens with 3 x magnification factor, but it wouldn’t be less shallow either. It would just stay the same of course. As an additional advantage, our “fake 600mm” would be much more light sensitive than your average “real” 600mm lens that would likely cost thousands of Euros (or dollars).
Thanks to fellow filmmaker and photographer Peter Hainzl (I know him from Twitter) who really gave me the idea about a “digital zoom for DSLRs” in the first place.
On a completely unrelated note: I was shooting with him at a local skate park in Vienna recently, check out the video that he cut from the footage he shot – I’m in it, too:
EDIT 5th May 2010:
Some people in the comments keep claiming that we would lose the beloved shallow depth of field if we were to employ an optional window mode. That is just plain wrong though, because the projected image of the lens wouldn’t change a bit. We would only take the middle part of it and that’s about it.
Commenter “Blendin” explains it best:
“The rule is, same lens, same DOF. Nothing more, Nothing less.. nothing to do with the sensor (size, or size of the pixel). so, with 200mm you got the DOF of 200mm. the crop fator only make the image bigger or smaller.
With the same glass, how come people think the DOF of the image can be different ?”
He might not have phrased it entirely correct, but what he means is a simple fact.
Think about it:
YES, we take a smaller sensor area, but that does not at all mean it would look the same as on a small sensor.
WHY? Simply because we also only take SMALL SECTION OF THE IMAGE that exact same lens is projecting. IT IS THE SAME IMAGE, we just take a high resolution “window” right out of the middle of it. Nothing changes about the physics of the sensor OR the lens – which would be necessary to constitute a change in depth of field.
I hope now everybody gets it 🙂
This “crop mode” was included in our List of features requested to Canon in 2009 (and even before):
Indeed, the crop mode is a “windowed” sensor function. It doesn’t skip lines, at least it was proven on the 5D Mark II with the “Zoom level x10” function, which is implemented as a manual focusing aid.
We have done very “fast tests” and it looks like the “rolling shutter” phenomenon is decreased in that mode. But even more: the “aliasing” effect is completely gone!
So even if the rolling shutter/jello effect is present as in normal Movie recording, the benefit of not having any aliasing artifacts makes this VERY important for filmmaking purposes.
Especially on a camera like EOS 5D Mark II, which is a Full Frame sensor DSLR, so the “crop” factor when using a “crop video mode” would be smaller than in an APS-C sized sensor camera.
Of course it would be really hard to get wide angle focal lengths anyway, but much easier than with an APS-C camera.
But sometimes the crop factor is an advantage. It will depend on your needs.
It comes with already known benefits: the “extra” effective focal length, and the increased image quality, since the sensor is using only the central part of the sensor/frame (in case you select that area, if the camera function allows to select more than the center. The 5D Mark II Zoom level function does allow the user to “move” the selected viewing area across the entire frame!)
A disadvantage of this feature is that the noise is much more visible, as you can easily see using the Zoom Level function. That’s probably because the camera has much less data to work with for noise reduction process (due it’s only reading a small area instead of the entire sensor, or the full frame area with skipped/binned lines)
But this mode was suggested and requested to Canon long ago, and they implemented it on the entry level T2i/550D.
The 5D Mark II seems to be really capable of doing the same task, and maybe even at least at HD (1280×720).
The remaining question is: Will Canon include it on the 5D Mark II?
As always, the more customers requesting, the more chances.
Been looking forward to this article.
Being quite new to DSLR film making, and the cameras themselves, your articles prove to explain how everything works which is great.
I think your point about the 1080 crop mode is great. The rolling shutter is a pain which needs to be fixed.
Thanks for a great read.
Look forward to the other parts!
Thank you for this article! I am postponing purchase of DSLR camera for about 5 years now (using digital 5.1Mpixel point and shoot Nikon, after abandoning 35mm SLR Nikon). Decision time on what to buy is near but… Lately, I got to the point of moving away from Nikon and leaning more towards buying Canon because of full HD video. Your article explained few points well (comments bellow also) since I was not quite sure, will I be happy with full HD video made by DSLR and should I go with camera that has it or not.
I just don’t get it.
Why does Canon just record in 5616×3240 and then use plain old simple downscaling with factor 3 (always picking the average of every 9 pixels) to get a 1872×1080 image? Put two 24 pixel borders to the side of the image and voila, you got 1920×1080.
The 550D may be too slow to process all that data, but the 5D has two of those processors…
The 5D has one DigicIV not two.
You’re right – only the 7D and the 1Dmk4 do feature a Dual Dicic4 Processor.
hey nino, which monopod were u using in this clip?
Intriguing possibility about the crop feature. Makes perfect sense to just use the pixel size needed rather than enter into math processor tasks. Really if this is too eliminate the rolling shutter and aliasing one could enter into crop mode when these aspects are more apparent. Like heavy duty motion. I guess less is more in this case. Go 5DmkII Team. I am going to be on the T2i team in about 25 days!
What about the depth of field using only 1920*1080 area of the chip? I would think we`ll loose the look of the Canon 5D we all adore so much.
The rule is, same lens, same DOF. Nothing more, Nothing less.. nothing to do with the sensor (size, or size of the pixel). so, with 200mm you got the DOF of 200mm. the crop fator only make the image bigger or smaller.
With the same glass, how come people think the DOF of the image can be different ?
As you say the optical properties do not change and we are talking about looking at sections of the image projection that would normally be on a full frame.
The thing to remember is that to define depth of field you have to set an acceptable level of focus and this is arbritrary and relative to the output magnification. A lot of the calculators that come from the photography world are based on ratios designed for making physical prints from negatives. In the video world we have a vast range of potential magnifications and a “negative size” of 1080p which is shared by a big array of different sensor sizes
Of course it is completely wrong to say that engaging a video crop mode would not change the depth of the focus field. If you have the same field of view, same lens, same focal length, same aperture setting, and same focal distance, you’ll have the same image projected onto the same sensor. What changes that relates to depth of field? The effective sensor resolution, of course. And that is directly related to your circle of confusion criterion. In crop mode, the effective resolution in the image plane goes up – you have 1080 lines, but in a much smaller area. Higher resolution means a smaller circle of confusion criterion, which means a shallower focus field.
It’s super easy to test this theory. Take a full-frame full-resolution photograph. Downsample it to 1920 x 1080, and compare that to a 1920 x 1080 1:1-pixel crop. Cropping does not change the depth of the focus field. Downsampling does.
But as others have pointed out in the comments, in practice you won’t necessarily want the same lens, focal length, or perspective if you are using crop mode. It depends on what you are trying to achieve. Smaller sensors generally mean smaller aperture lenses, which mean deeper focus field.
Nino, I think you’re off base.
Yes, depth of field wil be affected.
You said it yourself- “DOF would only be different if you would somehow change the projected image onto a smaller area, but that’s not what we do here.” This is precisely what you propose.
That C5D thread died because it’s one of those ideas that look good on the surface but doesn’t withstand scutiny.
Great work. I am blendin, from cinema5d. Let’s hope some one can make this idea work.
I think there is some misinformation in this article. To say “massively increased focal lengths” is a little bit misleading. The focal length of a lens always stays the same, no matter what camera you use it with. A 50mm is always a 50mm lens, be it on a 7D, a 5D, a 16mm camera, etc. What changes is the field of view. The smaller the sensor, the narrower the field of view with any given lens. Also, to say that the DOF stays the same isn’t completely right. If you want to keep the same field of view when switching from line skipping mode to windowed mode, you will have much deeper DOF because you need to use a wider lens. If the size of the sensor that is used in the windowed mode ends up to be about 1/3 inch, you get the same DOF that you would get with any other 1/3 inch camera (HVX200 ect.). So why not use a 1/3 inch camera to begin with that offers all the advantages of a proper video camera…
I really like the idea of the crop mode, with a single but:
As you wrote there would be a 3x magnification, means my prime lens EF 50mm/1.4f (80mm on my t2i) would become a 150mm (240mm on crop bodies).
If i’d use a wider lens to virtually get close to my 50mm, the lens for my t2i would probably be the EF 14mm f/2.8L II USM. And this lens is 1. expensive and 2. wouldn’t give me the same low light capabilities and shallow depth of field as the 50mm/1.4f…
So one could say that if you mainly shoot video with your 50mm prime lens you’ll be lost…
the best Telephoto lens that i have used on an SLR is the Canon EF 70-200 F/2.8 lens. Best image quality ever.-“,
Waiting for part 3 patiently…..
But what if you want to shoot a wide angle? (e.g. 20mm)
That would be a limitation, right?
Try these words:
Yes, using the same lens results in the same DoF regardless of the area of the sensor that is being used, AND the images are different in that the smaller area results in a narrower field of view (with the same resolution in this case) – it’s a more telephoto image.
If the lens is changed so that the field of view remains the same then the DoF changes.
I’ve often used a Super35 camera to match 16mm by blowing up the image in telecine. I’ve found that the DoF characteristics do indeed match the greater DoF I would get if I were shooting with 16mm camera.
A smaller imaging area forces the photographer to recompose to achieve the same framing as he would with a larger sensor. To be perfectly precise, he must move further away from the subject. As anyone who has spent any amount of the time behind a camera knows, depth of field increases with distance from the subject of focus. Ergo, when you compose for a smaller imaging area, the effect is a deeper depth of field, even though the relationship is not a direct one.
So yes, for the same reasons that physically smaller sensors yield differences in DOF, so will your 1080 crop. Why you thought it was immune defies logic.
Is the diagram for the crop right? I have never seen anything technical about this mode but it doesn’t seem to explain the very high level of magnification you see in this mode if it was line skipping. The maths say 7x is the max possible and that looks about right. This is still based on the crop image circle of the 550D so you need to go up another 1.6x to FF equivalent.
I wouldn’t be too concerned about the drop in bit rate. 480p is only 15% of 1080p in terms of bits all other factors being equal, so halving the bit rate is very conservative. Is it useful? Well stretched up to 720p it holds up pretty well due to the better definition. If you try to achieve the same mag by zooming a 1080p clip in a 720p sequence it looks like mush. DoF is only that of the lens but it does turn your $100 nifty fifty into a 350mm f1.8 so you will be able to shooter at much smaller apertures which will compensate. The crop doesnt use the corners so wide open should be fine
I presume you have seen that Panasonic have beaten Canon to the punch with a 1080p crop on the GH2. They have been just as bad at bringing this to anyones attention though.
Canon seems alone in chasing pixels these days and you have to wonder why given the issues with low light performance, line skipping and rolling shutter. I like to hope that’s because they have thought of something really clever to do with them. Things like a 4K crop within the S35 image circle or using neighbouring pixels at different sensitivities to extend dynamic range. One can dream!
Such a feature would unfortunately not produce the kind of image quality that has led us all into falling in love with the look of dslr footage. It is all about sensor size, and also about DOF.
You’re technically right that DOF doesn’t change when a sensor is cropped. But what would happen in practice is this:
Imagine I wanted to get a nice, medium shot of an actor who is a few feet in front of brick wall. On a 5D, I could currently shoot him with a 50mm f1.4 at say f4, and render the wall a little less sharp than the actor.
If I shot with the same camera without line skipping, that’d crop to the center of the shot, turning it into a close-up (which yes, still has the same depth of focus I’d planned).
But I want a medium shot, head to waist, not a close-up. So I’d have to do one of two things: Either I’d have to switch to a 17mm wide-angle lens to cover the same angle of view, which would have significantly different DOF characteristics, or I’d have to move much further away from the actor. Both of these choices would make it much more difficult to throw the background out of focus. The second option would also significantly alter the perspective of the shot, giving it a more distant feel.
Basically, the smaller the area of sensor that is used to capture an image (regardless the size of the total sensor), the more difficult it becomes to achieve any shallow depth of field, and the more all of the video starts to take on the cluttered, deep-focus aesthetic we all associate with consumer camcorders. It’s not that every shot needs shallow DOF, but that small sensors make control of DOF much more difficult.
Until the DSLR makers give us cameras that capture the whole sensor and then simply scale the image down to 1080, I’ll happily suffer with moire and rolling shutter. Better that than to effectively turn my 550D/T2i into a Canon HV20 with interchangeable but impractically long lenses.
Wow Nino, I can’t believe how many people are confused about this. I feel bad that you had to explain it so much. Good post.
A few hours ago you wrote: “YES, we take a smaller sensor area, but that does not at all mean it would look the same as on a small sensor.”
This is the point of contention. I’m not sure how to interpret this statement in a way that makes sense to me. A small sensor and an identically sized area cropped from a larger sensor are simply functionally identical. I don’t see a way around this.
Think about it this way. If you have a 50mm lens on a generic box of a camera. The lens is throwing a big circular image on the film plane. You can put a 35mm-sized sensor on the film plane and capture a 35mm-sized section of the center of this image. You can put a tiny 1/6″ camcorder sensor at the film plane and capture a 1/6″ portion of the projected image. You can put the 35mm-sized sensor on the film plane and capture a cropped, 1/6″-sized portion of the projected image. The image projected on both the 35mm sensor cropped to 1/6″ and the actual 1/6″ sensor is exactly the same. A 1/6″ cropped section of a 20cm x 20cm sensor would also be hit by the same photons that would hit the 1/6″ sensor in the same spot. –If the sensors are of equal quality, Capturing a 1/6″ cropped area of a larger sensor offers no advantage to capturing that same section of the projected image with an actual 1/6″ sensor.
As to whether or not the DOF of a nice 85mm portrait lens at f4 is the same, regardless the size of sensor, yes it is. You are correct on that point.
But what I was trying to say earlier was not that DOF of the lens changes, but that the changes to one’s filmmaking required by such a crop factor reduce the ability to control DOF. With a 3x crop, that 85mm is no longer a portrait lens, it’s now a better choice for sports photography. So in order to get the same shot, one has to either move a distance away, which results in a shot with much deeper DOF, or switch to a 3x wider-angle lens, which results in much deeper DOF. With a large sensor, one can shoot a wide shot with shallow DOF. With a small sensor that becomes very very difficult, regardless whether it’s an actual small sensor or a small area cropped from a larger sensor.
Ah, I see part of the problem.
“Nothing changes about the physics of the sensor OR the lens – which would be necessary to constitute a change in depth of field.”
The physics of the sensor are actually not a variable that should be part of this discussion (unless you want to get crazy technical and discuss whether some wayward photons don’t make it into the receptors of a CMOS chip that would register on a piece of film –I can see the possibility of there being a small pinhole-camera effect at every cell, but this is a red herring as far as goes this discussion). The depth-of-field visible in footage shot on DSLRs is not generally more shallow than that of smaller sensor cameras because there is something special about the design of the sensors, but strictly because the sensors are larger, which affects the focal lengths one can use, which affects DOF.
Just to put the DOF debate to bed, here’s a little image I threw together by means od an example:
In short: DOF is impacted by the ratio between focal length and sensor size. In the example of the digital zoom, the lens and sensor size remain unchanged since the lens is still projecting its image over the whole sensor.
The digital zoom operates by only reading data from the centre 1920×1080 pixels, so the DOF is exactly the same because the lens:sensor:pixel ratio is unchanged, therefore neither is the circle of confusion.
All that changes is the apparent focal length (hence the ‘fake 3x zoom’ Nino discusses.
Now I’m still not convinced this is really useful to anyone except dedicated sports/wildlife filmmakers since the zoom factor is so high, but this remains to be seen.
Hmm, no image.
Here’s a link: http://rollingshutter-films.co.uk/images/digizoom.jpg
Ok, I give, explain to me how using a 12x8mm section of a
full frame sensor is any different to simply using a 12x8mm
Funny to read this after Canon released the EOS 600D which has exactly the video mode described here. Maybe Canon is reading over your shoulder 🙂
Actually the thing about Dof is quite easy: same Lens, same apeture (!) = same dof. Problem is, when you want to get the same image in matters of cadrage, you could do 2 things: either replace your object or camera (if possible) and increase the distance between object and image, or use a wider lens. Both will increase your dof. And we all love our DSLRs for their Shallow dof, don’t we? So if you want to take a close up with a 85mm e.g. you would usually place the person, you film like 1-2m ca. crop 3x so you have to place it like 4-6m or so in front of your lens. So with the same cadrage you will get with the ff – cam 2-9cm dof (round about) and with the 3x crop round 13-30cm dof. Get a dof tabel and calculate yourself 😉
If you could choose, when to use the crop, it would be quite cool. If not… well actually we are just experiencing those problems with the scarlet ^^