Image Image Image Image Image Image Image Image Image Image

Nino Film Blog | April 20, 2014

Scroll to top

Top

53 Comments

Save terabytes of storage with your DSLR footage / tips for storage options

Save terabytes of storage with your DSLR footage / tips for storage options
Nino Leitner

I usually don’t blog about other people’s blog posts, but this one struck me as so important to the DSLR workflow that I just couldn’t resist sharing it with you.

Twitter is an invaluable source of useful information. It is social media at its best, and good news travel fast (you can follow me on Twitter here). Mario Feil, a fellow filmmaker from Germany, is one of the people I am in frequent contact with via Twitter, and he had this incredible idea of how to reduce the immensity of data we have to wrangle when we shoot DSLR video.

Big thanks to Mario for this simple but efficient method!

It isn’t only due to the size of the native footage that we are stuck with lots of data – it’s because of the fact that we can’t edit the native footage in its native H.264 form and have to convert it. Most editing platforms just can’t handle it because it is just a finishing format essentially.
So we follow tutorials like this one to convert the footage into an editable format like ProResLT and therefore create huge amounts of additional files of the exact same footage. You got everything twice as a result, and oftentimes the converted files are larger than the originals, depending on the codec you use.

So you will only edit the converted files but always keep the original files as they are your master files – and every converted file will have to lose a little bit of quality of course. Never deleting the native files is essential to be able to always fall back on the maximum quality that you have available.

BUT there are good news: you can delete something else and thereby save hundreds of gigabytes or even terabytes of data.

Free up terabytes of space on your hard drives

Mario Feil’s approach is as simple as it is brilliant, here’s the short version of how he does it:

When you are done editing your project (using the converted H.264 as ProResLT or other formats), you can delete or move your converted ProResLT files (or whatever other format you are using for editing) and re-open the Final Cut Pro project. Of course, the application won’t find these files as a consequence and ask you to reconnect the media.

When you reconnect, you simply refer to the NATIVE files instead of the converted files – both should have the same names and lengths, their only difference should be the codec within the .mov-wrapper.

Et voilá!

Final Cut Pro will automatically connect everything with the native files and you can re-render all effects right onto them, guaranteeing you the best possible quality from your DSLR footage while saving you possibly hundreds of gigabytes of converted data.

Should you ever need to change something about the edit later on, it should be quite easy to re-convert all the native footage into an editable format again and reconnect the other way round.

Please do check out Mario Feil’s new blog for details on how to do it, it will save you several harddrives if done right. I have just saved 400 GB from a recent project – it’s just incredible.

Storage options for DSLR video footage

By the way: I get often asked how I store my footage. Let me explain you briefly.

The rise of solid state media recording has given us many blessings, cheaper cameras without flimsy mechanics and things like higher framerates for slow-motion recordings – all of which was just impossible with tape-recording camcorders only a few years back.

The biggest downside to that way of recording, however, is that you have to think hard about how to store your footage properly. Hard disks are mechanical mediums as well and if they fail – and boy, they do fail from time to time – a lot more footage is lost than it would be with a broken tape recording. Therefore, backing up is essential.

Different people use different approaches to save their footage, but in essence, you can’t get around HDDs as affordable ways of saving your data. Many people use RAIDs, which combine several hard drives into arrays that offer increased storage reliability through redundancy. The problem with RAIDs is that they are usually expensive and often not very portable – so when on the road with your DSLR and laptop, you don’t really want to bother with a device of that size.

There are very nice exceptions though:

I also had my share of bad experiences with external hard drive enclosures. Some manufacturers build their enclosures in such a way that you can’t swap hard drives – if you do, you can’t read out the data anymore, believe it or not. Happened to me with an external Maxtor enclosure. My only option would have been to go to expensive data recovery services, and they charge per megabyte.

Another problem with external hard drives is that they change the shape and form of these things every few months, and if you produce around 1 TB of video data per month like I do, this can get annoying VERY quickly. Also, different drives have different connectors, you end up with USB2.0, Firewire 400, Firewire 800 and eSATA at the same time. Extremely tiring, with miles of wires and leads you have to carry along.

So what is the other option?

It’s a very simple and affordable alternative: an external hard drive dock!

You take any industry-standard 3.5″ internal SATA hard drive and just stick it into the dock, and it will mount. These docks are available with USB or eSATA connectors, and I prefer the latter for the most speed you can get (same speed as internal drives, actually). I just bought an eSATA expansion card for my MacPro to make it work, but USB is fine too – and some of the newer ones already support the super-fast USB3.0.


QuickPort Duo on Amazon.de, on Amazon.co.uk

The great thing about this solution is that it is simple and very affordable. You can shop around and get any internal SATA drive you want for the dock, stick it in, format it, done. All in all the dock including the hard drive will cost you less than one external hard drive, but after you purchased the dock, you only need to purchase new hard drives as you produce more and more data. I have a closet full of hard drives now, every single one neatly labeled and numbered, with a database on my computer telling me on which drive I can find which recording.

I use the QuickPort Duo that gives me the chance to connect two drives at the same time via eSATA. One of them is my backup, so I mirror the contents of one drive onto the other. When the mirroring is done, I edit from one of the drives. If one drive should ever fail, I still got the other drive with the exact same content – I could continue working right away.

Of course some people think it is not a sturdy solution like external hard drives are – that might be partly true as the hard drives aren’t protected when they are sticking out of that dock. However, there are protectors available for this, and I have never ever had any problems with this solutions in the almost two years I have been using it now. It’s also quite portable and the docks are so cheap that you could buy one for every location you frequently commute to.

I currently use countless Samsung 1TB and Western Digital 1.5TB drives with my dock, and not a single one of them has failed. My personal opinion is to stay away from Seagate drives for now as they had massive failings some years ago with some firmware glitch (they have resolved that issue, but they haven’t regained my trust yet).


Western Digital HDD on Amazon.de, Samsung HDD on Amazon.de
Western Digital HDD on Amazon.co.uk, Samsung HDD on Amazon.co.uk

As I said, different people use different solutions. I know many people (like Mario Feil, for instance) who love the Drobo, a NAS device that offers automatic backup and data migration and a lot of security for your data. The downside is that it is slow and bulkier than my solution, but probably even more secure.


On Amazon.de
On Amazon.co.uk

What do you use for storing and backing up your data? I am very eager to hear what other people use as people should be aware of all the good options. There is hardly anything more important than data security.
We should never forget that what we produce will not be here forever, but we can at least try to make it last as long as we can. So what’s your solution?

Comments

  1. You really should check out Adobe Premiere Pro CS5 – native, online editing of H.264, R3D and a host of other formats. No conversion to intermediates needed. Give the demo version a drive – you might be pleasantly surprised!

    • Yes, I’ve heard of that – will definitely give it a try! Will need some time getting used to working with another editing application though. Thanks.

  2. Yeah, he is right… Premiere Pro is great for DSLR footage…

    But if you are using After Effects or Premiere Pro (PRE-CS5) You can use the same kind of workaround for relinking the proxy footage.

    If you want to try it out, load up your project make your edits with the ProRes footage, or whatever you use… and then close the project, move the footage, and then open it back up. Then you can use the relink, or reload footage after you right click on the file, and you should be able to reload the old DSLR footage.

    Although, I have had some issues with AE crashing with the native EOS codec format Canon uses… (only AE CS4)

  3. Nino, thanks for a great post.
    I currently look forward to finding my solution to storage problems and it helped me a lot. Well, I am not sure whether it really makes sense to get back to H264 and getting rid of ProRes files. What about throwing away your source footage after conversion? Storage is cheaper than time and I think in a long time scale you would benefit having slightly more data (keeping ProRes files vs keeping H264), but not being forced to reconvert.

    Can you give me more details about this mirroring solutions using a dock? I understand that you simply connect 2 docks at time right?

    Thanks,
    Paco

    • Well, getting rid of the ProRes files simply makes more sense because of the fact that their quality is worse than the originals of course, not because they are larger. You ALWAYS lose quality with standards conversion. Compare a H.264 file from your DSLR to its converted ProResLT equivalent at 100% and you will see what I mean. Look especially close at gradients like the sky etc., where compression is often most apparent. It’s even easier to spot if you convert H.264 files to XDCAM EX.

      The dock I use has two separate eSATA connectors for each drive that can be plugged into the dock, yes. If you use it with USB, you need only one lead, but of course it will have to share the bandwidth. eSATA is always the best option because it can deal with your drives at the same speed as your internal connectors – there is nothing speedier (except for a fibre channel RAID of course).

      Unfortunately, Apple doesn’t offer eSATA “out of the box” with their systems, but you can buy a cheap accessory card for you MacPro or an ExpressCard eSATA extension card for your MacBook Pro (if you are lucky enough to still have one that offers that essential expansion slot).

      • Hi Nino,

        I have just bought a new MacBook Pro 17″. Can you recommend any eSATA-Extension cards? And how do you store your used hds? Just in the bags they came in? Great article by the way :)

        Thanks,

        Josch

  4. ...m...

    Presently using the Sony files to PremierePro and FCP. Backing up all on iOmega, which would present some of the difficulties that you cite if things went terribly wrong. Have been wondering about the HDSLR materials and specifically how you people handle two channels of 48Hz sound????

    In this age, data management is vital. Like the best vault for film negative.

    Film. Remember film??

  5. Avid Media Composer also edits dslr footage natively along with most other formats. I know its expensive (as much as adobe master collection) but its worth it.

  6. Great post! Thanks Nino! I was leaning toward something like this, so your post came at the perfect time.

    Also, the source switching technique also works for editing hi res video on a slow computer. You can edit in 720, then switch it out with the highest quality you have for final export. Works great.

  7. Matt

    I am surprised at how SURPRISED everyone is by this… has HDSLRs really brought that many noobie/inexperienced filmmakers to the industry?

    What the blog post refers to is a simple offline/online edit which any professional editor will do ALL the TIME when dealing with footage not suitable to cut natively (essentially in every scenario/format save for DV).

    ProResLT is SPECIFICALLY designed for offlining and should not be used as a mastering format WHATSOEVER. Let alone should anybody be transcoding from the raw H.264 to ProResLT and then into (what they think) is a higher ProRes or QT codec. Once you degrade with lower bitrate compression you CANNOT get it back.

    I don’t mean to come off like a troll but it worries me sometimes how many people who call themselves “filmmakers” are out there just following tutorials or figuring things out as they go. The web is degrading the quality of work out there – especially since people think that just because their footage looks good in a 853×480 box on the web that they are following a proper workflow and know how to edit.

    If you want to learn and educate yourself about post-production, there are plenty of books written by very talented editors at any local bookstore.

    • Matt, while it is true that an offline edit is nothing new for many professional editors, it is indeed news for the many filmmakers just getting into the business and maybe even starting out with DSLRs.

      It is however absolutely not true that ProResLT is “specifically designed for offlining. You are mixing things up here. It was introduced by Apple for other reasons, ProRes 422 (Proxy) is intended for offlining.

      Check for yourself on Apple’s site for that:
      http://www.apple.com/finalcutstudio/finalcutpro/apple-prores.html

      “• ProRes 422 (Proxy). For craft editing or offline editing on a MacBook or MacBook Pro.
      • ProRes 422 (LT). For projects such as news, sports, and multicam events that require reduced file sizes at broadcast quality.”

      Also, it isn’t true that the described way of doing things is a standard procedure for offline editing. This is just not true. Final Cut Pro has a dedicated workflow for that, and it does not involve the tricks described here.

  8. You need to watch the DSLR shootout on Zacuto’s website. In the latest episode they compared the original H2.64 files to the converted Pro Res files. The Pro Res files looked way better than the original files. Something about converting to 4.2.2 makes them stand up to grading and colour correction too.

    So… don’t go deleting your pro res files. In fact, don’t delete anything.

    Your Original H2.64 files are your rushes… your footage. Convert to Pro Res and then back up somewhere safe. Edit using Pro Res files or even Proxies (made using Magic Bullet Grinder).

    I wouldn’t delete a single file… ever. Drives are cheap.

    • Thanks for your comment. Good point, I’ve already seen the Zacuto shootout but can’t recall the ProRes conversions looking better, but that could well be true AFTER color correction. BUT I have also seen banding in the sky of many converted H.264 files that wasn’t there when they were still in their original format. So I guess the truth might lie somewhere in between.

      It’s true that drives are cheap but it is nevertheless cumbersome to keep twice as much as you absolutely have to for projects that are done and over. Another option might be to let Final Cut Pro generate independent clips (including file handles for minor edit adjustments later on) of the final edit – something which is possible from within the application without any tricks. You could still trash all the unused converted files and then you are left with your ProRes edit.

  9. ScottieB

    As some others have mentioned – they do talk about the benefits of converting to ProRes in the Zacuto shootout (toward the end of part 3 I think) but it is easilt the LEAST clear part of the entire project, and is quite brief compared to everything else they touch on. Philip Bloom himself mentions in the video that it doesn’t make a whole lot of sense technically, but the stuff just looks better when converted to ProRes first. BUT, on his own website he has a post about how excited he is to try Premiere with native files – and no follow-up.

    So I’m STILL looking for info on this. On the one hand it doens’t seem to make sense that converting will help – I mean any conversion is a generation loss, right? But on the other hand, I can also understand how converting to 4:2:2 could give some “overhead” for better processing/color correction.

    I still don’t have CS5 so I can’t test this myself, but I’ve heard lots of speculation, and very little solid info… anyone out there have any experience with these two different solutions?

    Thanks for the article!

    • ScottieB

      So I had a little time to run a test with my T2i. Posted a video, can provide stills. Video is 1080p can be downloaded.

      http://vimeo.com/12813991

      There is a very slight difference but I can really only see it when I layer the clips and then switch on and off. I didn’t push the color correction all that far.

      • Thanks Scottie for the test … though I can’t see much difference, and I guess it only gets more difficult with Vimeo’s compression and also your compression before uploading it to Vimeo.

      • ScottieB

        Yeah – I couldn’t see much difference without compression either, though to be honest. I’ve only done a couple of projects so far and all of those were done with ProRes. This test was enough to convince me to try a “native” project next. So far Premiere CS5 on my MacPro with GTX 285 handles the footage pretty well, although things like scrubbing or playing in fast reverse/forward aren’t nearly as responsive as with ProRes. We’ll see…

      • Great – keep me posted on how Premiere handles native footage in comparison to ProRes in Final Cut Pro.

      • ScottieB

        Not sure if anyone is still checking this but I had some time to play with Premiere CS5 this weekend.

        Honestly, I ended up a bit disappointed. I’ve never had a problem with Premiere per se, just been using Final Cut since Verson 1 so I’m used to it (before that it was Premiere 5 on mac, but in the days of DV, FCP was clearly boss).

        Anyway, at first CS5 seemed just fine working with the h264 footage from my t2i. I have a mac pro 8-core 2.8ghz with a gtx 285 gpu.

        But once I got into a groove with editing, I found the native footage to not be responsive enough for how I edit. Once I get going I move pretty quick – I like to able to play, stop, reverse, fast forward, pause, frame advance etc really quickly with no lag – like in the days of tape editing!!! But with the native footage there tends to be a slight pause between when you press ‘play’ and when the clip starts playing. It seems it needs to buffer a bit before it plays. (This is at 1/2 quality playback).

        Sadly this is a dealbreaker for how I work – I just got totally taken out of my groove because of these delays. I ended up going back to the ‘old’ way and converting to ProRes first.

        It’s certainly the best I’ve ever seen a program work with h264 natively, and the fact that it can edit it without much trouble is impressive. CUDA is exciting stuff, and it isn’t really a lie to say that CS5 edits natively. However, it isn’t the same as using something like ProRes… it works, but it isn’t as responsive.

        For certain things that need quick turnaround native editing will be awesome.

        But for the way I work with my SLR, which is less run-and-gun and more slow and steady, I’m gonna stick with ProRes – even if I DO work in Premiere (which I may continue to do, because a trip to After Effects is inevitable).

      • Thanks for the valuable insight, Scottie! Will try to get my hands on CS5 soon.

  10. Why not get the new Avid5 straight away? No conversion necessary any more… :)

    • Yes, but it’s a different story to invest that much and learn the whole thing :-) Last time I used Avid I was still in film school …

  11. Nino, thanks for the great post.
    Do we know why a prores looks better then the h.264? It sounds like the photoprint looks better then the actual negative. That is also my experience. h.264 looks often grainier and sometimes aliasing disappears when converted to prores. Any clue?
    Keep up the posts here.

    • That is a good question. Is it really looking better? Haven’t made that experience, but I can be wrong. I can’t think of a plausible explanation for that … have you looked at the original H.264 vs. the converted files or at a H.264 that has been recompressed?

      • Yes, Colors look a bit richer but not only that the h.264 image has something grobpoorig to it, something corse. The converted one has great details and seems a bit sharper.
        No idea why that is.
        Regarding your blog can you integrate a function when we write comments and there is an update that we get notified via email? I think the function works on my wordpress blog. I assume you use here wordpress.

      • Thanks for the info regarding the notification – I didn’t know that commenters aren’t notified when somebody replies to their posts. I will look into it and send you a “test reply” once I get it to work (it doesn’t seem to be a standard function).

  12. One more thing, do you have edited with such a HD station in USB? Are there firewire 800 versions out there?

  13. Great blog.
    We’re filming a feature length film right now with the Canon T2I and I love your blog for info.
    I picked up a DockMaster after trying to figure out what to do with all my old Hard drives after upgrading all my computers.

    Fantastic!

    • Sweet! What is the feature about?

      • the feature is a girl and her roommate try to make their first movie to win a contest and bail the girl’s brother out of jail.

        You can see a rough trailer here http://www.vimeo.com/12689370

        The Dop and I have brought up your blog in several conversations … he mentioned “Pimp your dslr” articles and I was talking about your battery tests the other day after we picked up some non-canon battery’s.

  14. RY

    CS5 don’t need convertion to proressLT anymore. you can edit native H.264 files.

  15. Thanks for the info. You are doing a wonderful job communicating your message. Keep up the good writing.

  16. Stu

    Nino, are you editing with drives in these docks, or just copying material over for archiving?

    • I’m editing from one of drives directly in the dock, but the other one is backup and isn’t touched and remains switched off once the footage is offloaded and cloned onto the two drives.

    • Stu

      Nino, very interesting solution, thanks much for posting.

  17. Barry Best

    Brilliant solution to a problem I guess many are having. I just spotted your tweet in time, as I was about to spend about £200 on an external WD esata 2tb drive, should be able to get the port and about 3tb for that price. I love the plug in and use, unplug and store aspect. brilliant.

    • I’m happy to help! Any purchases made on Amazon.co.uk through my links are much appreciated (the Amazon.co.uk links are below the .com product links above, just text links!). Thanks!

  18. Nino,

    Thank you for posting this. I have been looking for a better backup solution then the one I have (which is two usb lacie drives). This is faster and cheaper, really appreciate it. If you find a good way to store them please let me know, I could see how that could be a problem.

    Michael

  19. Haha, I feel like such a fool for not having thought of
    this! So simple. Anyways right now I’m editing with Premiere CS5
    with native support for H264 files so no need to convert. Only
    works for short projects though.

    • It’s great that Premiere CS5 can handle these files natively, though I’m not sure if working and editing in an 8-bit space is a good idea. H.264 compresses so efficiently because it works like a JPEG and drops information you can’t see. If you save an image over and over again as a JPEG is degrades massively. Likewise, H.264 was meant to be a finishing codec, that’s why I’m not sure it’s necessarily good to work in that codec.

      • True, a lossy codec will be degraded if it’s saved over and over again, but the workflow doesn’t call for that. I belive the quality is the same as if you convert to ProRes. Also you’r not bound to work in 8-bit space. The color correction tools etc. work with up to 32-bit even if the original footage is only 8-bit. It’s a little CPU heavy though, but with mercury GPU acceleration it’s pretty smooth.

      • That sounds great Markus. Will give it a try soon!

      • I havn’t done any scientific comparisons, but I don’t see why the quality should be different. However I havn’t tried it for anything longer than 10 mins. Wouldn’t be suprised if Premiere will crash on you if you load a 30 min project =P

  20. Here is my solution. First of all, I only save the card data and (sometimes) document the settings in a textfile, so I can always re-produce the ProRes or any other edit codec files.

    I have two RAID-1 FW 800 LaCie’s. One is for projects I’m working on (speed and space), the other for storage (safety). The great feature about these LaCie discs is that you can replace both discs. If it is full I can pull both discs out and replace them with new disks. Both old discs are stored at different locations. A HDD dock is great to read the old discs.

    • Good idea, very similar to how my workflow is working.

  21. Markus Haken

    Hey Nino,
    can you recommend any esata cards for the MBP? I have the old 2007 modell.
    Thanks,
    Markus

    • Sorry, I don’t have one … but I heard there’s one from Delkin that is quite good!

Submit a Comment


− 2 = 6