The Black and Blue

Filmmaking Tips for Camera Assistants

Media Management 101: Basics of Data Wrangling

With digital cinema showing no signs of slowing down – and data rates accelerating – having the skill in your repertoire to properly manage data is crucial. Even if you never do data loading yourself, knowing how the process works helps you manage your department.

by Miscellaneous Tips

In this video I give a brief overview of my basic media management toolkit, how to set up an organized folder system on the hard drives, and a process to make sure you never accidentally erase footage (hint: green tape = good).

Also, this 2nd video below explains in more detail a quick and simple way to check that the footage has been downloaded and backed up safely.

Best viewed in full-screen mode so you can read the on-screen text. You can also watch this on Vimeo.

Data Wrangler Basic Toolkit

In the video I give a brief overview of the tookit I use and mention some items specifically. Here is a list of equipment you see in the video:

In no way is this a super-professional high-speed kit, but it’s good enough to perform basic data wrangling tasks. If you’re a camera assistant consistently working on digital shoots, it’s a good idea to invest in some of these items.

One note about hard drives: it is standard for the production to buy them. Don’t ever use your own money or money they are paying you to buy the drives. Once the footage is on them, the drives will be filtered throughout the post-production pipeline. You are more than encouraged, however, to recommend a few hard drive options.

Video Transcript

Hi everybody this is Evan Luzi from The Black and Blue and today I’m sitting in a hotel room at a Washington, DC hotel doing a little data loading.

Now this isn’t something that I normally do, but every once in awhile a job comes up that, you know, the price is right, the hours are right, so I go ahead and take it.

But I thought it would be useful for you to know sort of the basic data wrangling kit that I have as opposed to somebody who might build a cart worth thousands of dollars. This is sort of the bare bones thing and it’s not the fastest kit in the world, but you might be able to build it by spending a couple hundred extra dollars and it could give you a few jobs every now and then.

So let me just sort of go over what I have going on here. I actually have a dump going on right now of a 32 GB card and it’s been going on for about 40 minutes already — it’s got about 17 minutes left.

We have it dumping to two different hard drives with 2 Terabytes on it each. I’m not sure how much footage they’ve gone through the day yet — I’ve got three 32 gig cards from A Camera and about four from B Camera. And then I had the sound guy’s data, too, to load.

So were dumping it on two backups. That’s pretty standard. The more backups you have, the better, if you can afford three, but two is essential. And one thing an AC taught me that I’ve always remember is always make sure you have two copies of everything before you willingly erase anything.

So don’t format a card until you have two copies, don’t put anything on a new hard drive until you have two copies. Basically you want to have a redundant copy before you move, transfer, modify — whatever you’re gonna do with it — always just have a backup.

So anyway, the main part of my kit besides these hard drives, which production provides — I would never buy my own hard drives, especially because they go to editorial or through the post pipeline when you’re done — but I have an old Macbook Pro. This isn’t one I bought for data loading, it’s just my personal computer. It’s one that I bought in college so it’s kind of old, so I’ve kind of had to beef it up a little bit with peripherals to make it acceptable. Especially with data rates getting so heavy these days.

So just recently I bought a Lexar Expresscard CF Reader for Compact Flash cards and this goes in the little express slot on the side of your Macbook Pro. Newer models, only the 17″ has that expresscard slot, but the newer models also have Thunderbolt which is insanely fast so peripherals will start coming out for that so you can take advantage of that if you aren’t blessed with the expresscard slot.

Also in here I have a similar expresscard adapter for SD cards for some Nikon cameras, most sound guys use SD cards, some of the lower DSLR’s — like the one I’m shooting on right now, the T3i, uses SD cards.

So I have that. Got the cards I’m dumping. I’ve got two rolls of camera tape.

Now this is kind of a different data loading job for me because I’m doing it after they’ve shot everything at the end of the day. Normally I’d be doing it throughout the day.

So what I like to have is green tape that way when I double check the card, when it’s OK to format, I can wrap it in green tape to the point where an AC would have to unwrap it to get to it. They can pop it in the camera and they can format it knowing that the green tape means it has been checked.

I usually go up to the AC or the camera operator or the DP and tell him if you get it without the green tape don’t format it! You know, bring it back to me, it could’ve been my fault but I just want to double check it.

The worst thing that could happen when you’re in this job is having something get accidentally erased because it all comes down to you: there’s nobody else checking footage on set.

So that’s sort of the basic tools I’m using — it’s pretty simple. Most hard drives will come with cables.

Oh — the thing I do have that you can’t see is a battery backup system in case the power goes out and as I’m shooting it’s kind of stormy outside so I’m really glad I have it. It will buy me about 40 minutes worth of time — which, you know, I couldn’t finish the job on that, but I could at least make sure nothing gets damaged. Make sure that the hard drives don’t get damaged if the power goes out.

So the next thing I want to talk about is the layout of the folders that I make on the hard drives to organize the footage.

If you’re doing a multi day shoot, at the most basic level, I just do “Day 1,” “Day 2,” “Day 3,” “Day 4,” etc.

Within that, I’ll make multiple folders. I’ll do an “A Camera” folder, a “B Camera” folder (so on and so forth) then I have a sound folder.

And within those folders, say for the “A Camera” folder, I’ll dump the first — whatever the first card I get — that’s roll “A001,” so I make a folder in that called “A001.”

Then I just transfer everything from the card into that folder — even the thumbnail previews. I don’t even mess with the structure of the card itself, I want to save absolutely everything.

This is integral especially on RED footage where messing with the tiniest little detail could corrupt every little R3D.

So you don’t ever want to just pick and choose which clips you’re downloading, just dump it all into one folder.

So as the day goes on when I get another card from A Camera it will be “A002” and that goes on. And that carries over over the days. So if at the end of Day 1, I’m on “A008,” at the beginning of Day 2 I’ll be on “A009.”

And so you do that for every camera. There are some days where you only shoot 4 rolls on B Camera, but 10 rolls on A Camera.

Now within the “Sound” folder, most sound guys will have you dump all their stuff at the end of the day or maybe do a dump at lunch and a dump at the end of the day as well. And within those folders, if you want to, you can just do “1” or “2.” Or you could do like “SND001,” just something to denote that it’s sound.

Every now and then I get on a job where they want me to keep log notes and those I do a Text-Edit document on my Mac. I’ll just save that to whatever day it relates to.

Or if I want to leave note for the editor or whoever is in the post-pipeline — say if a clip is corrupted or if I’m using the labels feature on my Mac — then I’ll leave a little key for them and I’ll type in “Please Read Me” as the title, and I’ll type in whatever notes I have for them.

I tried the route where I’d go up to a producer and be like, “Oh hey make sure you tell him blah blah blah” and it would just get lost in the shuffle. They would either not remember or they’d mess it up with the message and the editor would end up calling me anyway.

It’s a lot easier if you just write everything down into a little text document and save it to where they need to see it. And I think most editors and post people are pretty good. If they see a file they know it’s been put there intentionally and they’re gonna look at it even if they aren’t sure what it is.

So that’s kind of how I do the folder structure. The key for this is you want it to be super organized so when it gets into post-production people aren’t bugging out at how confused they are over which footage is which.

So once I’ve dumped it, I’ll go through and check each folder. And how I do that on a Mac is I hit the apple and the “I” and get the info. I can get the exact number of Gigabytes that a card has on it — you know, how much space is taken up.

Then I check those folders on each drive and make sure that that number is consistent.

And then what I’ll do is I’ll go through and I like to check the first clip on a card, a clip in the middle, and then a clip at the end. And if there’s a bunch of clips — like on this one we have over 200 — I’m probably going to do about 5 – 8 spot checks.

What I do for that is I open it up in Quicktime, I watch the first few seconds, I scrub through the whole thing quickly, watch the end, then it’s good.

Sometimes if you’re on set you may be pressured to be turning cards over very fast, in which case you do have to do the spot checks fast, but please never sacrifice doing your job right for the time.

If you’re getting really pressured, just go up to them and be like, “Look I just need to double check that these cards are safe. I don’t want any footage lost.” People may be frustrated, they may be annoyed, but they’re also going to understand.

Just one thing I want to add to the green tape thing I went over earlier. If you can add a verbal cue, where you say, “This is OK to format,” then I think that is more than enough. People really seem to appreciate when I go up and say, “This is OK to format.”

Because I think whenever somebody formats a card they have that moment of hesitation where it’s “Oh, I hope he checked it,” so I tell them upfront that I did check it. And that little worrisome moment is gone.

So make sure you do say, “it’s OK to format,” though. Don’t be ambiguous — don’t just say “this is good,” “this is OK.”

That could be taken in any way, so just say “it’s OK to format,” make sure it has green tape on it or whatever system you want to do and that’ll be good to go!

So that’s how I do media management. It’s not super complicated — it’s more like copy and paste — it’s to the point where I had the time while this is going on to make this video.

If you can get a job here and there doing this, I would recommend it. It’s simple work, but it usually pays pretty decently and it’s not so bad.

I mean, it could be worse: I could be out hauling gear in the storm or working through the night. Right now I’m in a cushy hotel room that’s 69 degrees just sitting at a computer.

Alright guys well thanks for tuning in. I’m sorry if the sound was a little bit. I’m right off of Dupont Circle in DC, which is a really busy intersection.

Also I’m sorry this video turned out a little bit longer than I hoped. But I hope you enjoyed it nonetheless. See you later!