Thinkbox Software is a key partner for us, with several pieces of software critical to our pipeline.
Deadline was the obvious choice for our render farm manager. When we were just starting up, both of us as founders – me ( Hristo Velev ) and Martin Naydenski, had previous experience with it. There it had supplanted the previous solutions at places we’ve worked at and we had great impressions with it.
The first impression you get from Deadline is how vast a range of software it supports and the general openness and flexibility it embraces. It has become the centerpiece of our pipeline, hosting a whole Python and Maxscript library that deals with all the minutia of running the daily process. From our custom simulation submission tools to send Thinking Particles, FumeFX, Naiad etc jobs to the farm, through rendering, to generating mov videos from EXRs for upload to Shotgun and our customers for review.
As in any production environment, our base needs are reliability and performance. So here’s where Deadline shines for us. We have scaled from 5 to 50 machines without even needing to plan for it on the farm management side – Deadline’s decentralized architecture, where there is no single point of failure like a manager app that could crash. He makes dealing with an expanding, heterogeneous, even location distributed collection of machines, a breeze. Looking at the much bigger size of the other studios running deadline, we’re comfortable that even with a 10x growth, we’ll be sailing smoothly.
The ‘central’ component of Deadline – the Repository : is actually just two things – a folder with stuff, where your jobs and scripts go and the database. Both can be easily backed up, restored on other machines or migrated. The slaves can easily be retargeted to use a different repository location. There is another Deadline service – Pulse. It does things like power management to wake up or shut down machines, clean up the database, automatically upgrade slaves and etc. It is easy to protect by redundancy. You could set them up on any machine and it will just ping slaves and get the job done.
But the great core is just a foundation – on top of it lies a really mature implementation. We’re often surprised by the Ali Baba cave of hidden treasures. One summer it got really hot and we realized that we have a cooling problem and we can’t immediately resolve, but it turned out Deadline could help. We bought a small network thermometer, after that we instructed Deadline to follow it’s readings and shut down the hottest servers when the ambient temperature got too hot. He keeps them down till it got cool enough, usually in the evenings, before allowing them to rejoin. Then, we got Shotgun and it’s great review workflow. We used Deadline as the connection between Shotgun and the rest of production – submitting versions from the Deadline monitor and using the Draft plugin to automatically create web-ready videos from the EXR render sequences, which ones we shared with supervisors and clients with just a couple clicks.
The granularity of control you get over how jobs and servers behave is also a great sign of how mature the software is. Several types of server grouping with different types of behavior, smart weight based job scheduling, license based limit pools – everything we’ve needed! Even that it includes some, let’s put it mildly – exotic ideas are already there.
The custom maintenance jobs allow us to even move software deployment on the farm to Deadline. Create a job that installs a piece of software using a simple command line script and it’s being taken care of, when servers get turned on for rendering. You can write your heart out with customizations there, just beware control freak pipeline engineers. Going too far is the main risk 🙂
And though we haven’t jumped on the cloud bandwagon yet – simulation data for our shots is usually way too huge to easily get across networks. We’re keeping a close eye on it, sooner or later it will happen. And it’s great to see Deadline has that next step covered already – it can do some crazy stuff like automatically deploy on the cloud and create instances on new machines when you need them. Sci-fi excitement level material, you should check it out!
Hristo Velev, co-founder and FX lead, Bottleship VFX