Bright lights, big data

By Curtis Collicutt, Cloud Developer, Edmonton

The Black Magic Cinema Camera in EdmontonIn my spare time I make short films, and my goal is to eventually shoot a full-length feature film in Edmonton. I like everything about filmmaking: writing scripts, doing the sound design, and, of course, working with the cameras.

I have two Canon DSLRs. These are inexpensive cameras that take beautiful motion pictures. However, while there have been several feature films made with DSLRs (such as Upstream Color, a 2013 Sundance selected film by Shane Carruth, which was shot on an $800 Panasonic GH2) they have several limitations, one of which is that they take the raw image from the sensor and apply a codec to it. This greatly reduces the amount of information stored in the resulting data files. While this creates much smaller and more convenient file sizes, it also means there is not a lot that can be done to the image in post-production.

Many people don'€™t realize how much post-production work is done to films. It'€™s beyond the scope of this blog post to discuss colour correction in post-production, but suffice it to say '€” in general '€” the more data that is stored in the files produced by a camera, the more a colour correctionist has to work with to completely change the look of the film. Being able to alter the look of a film in post-production is an important part of creating a beautiful movie, or at the very least making it teal and orange.

This is where the term '€œdigital cinema camera'€ comes in. While DSLRs are amazing for filmmakers like myself, they are not what many filmmakers would consider a digital cinema camera. In order to meet that definition, a camera needs to produce '€œraw'€ image data.

What do I mean by '€œraw'€?

Like a photographic negative, a raw digital image may have a wider dynamic range or color gamut than the eventual final image format, and it preserves most of the information of the captured image. '€” Wikipedia

While there are many motion picture (or video) cameras, few of them are capable of producing raw motion picture files, in part because there is so much data to handle. There is no way that the $20 SD card in my DSLR can keep up with the raw image data a digital cinema camera captures.

Here are a few cameras that produce raw motion picture image files:

  • Arri Alexa
  • Red Epic
  • Red One
  • Red Scarlet
  • Digital Bolex
  • Black Magic Cinema Camera (BMCC)

To me, as a no-budget filmmaker, the most interesting camera on that list is the BMCC. This is because it is a $3,000 camera that is capable of '€œshooting raw.'€ But in raw mode it produces a tremendous amount of data '€” for every 30 minutes of recorded video, the camera requires ~256GB of storage. So, for example, if I am shooting a 10 minute short film at 24fps with an expected 10:1 shooting ratio, I will produce almost one terabyte of files to store. It'€™s recommended to have two backup copies of the data, so that means three terabytes of data for a ten minute short film! The BMCC camera is capable of using lower data rate codecs, ie. ProRes 422, but if shooting in raw mode, they create a lot of data very quickly '€” around 8.5GB/min!

This much data creates all kinds of problems. As a no-budget filmmaker…how do I store it? How do I archive it? How do I share the footage with an editor or colour correctionist who might be in a different city? Once you get over 3TB, it'€™s hard to even do sneakernet file exchanges. Money can solve most of these problems, but money is difficult to come by for emerging filmmakers like myself.

Even the biggest film productions, such as The Hobbit, have storage issues. Production on movie generated between 8-12 terabytes of data per day. Once completed, the film required petabytes of storage in multiple datacentres!

In the 21st century, big data isn'€™t just relegated to industry, science, or government. As a filmmaker, for $3,000 plus a couple solid state drives, I too can have all the problems of big data.