Chihuahuas can be scary. But the real reason it served as reference for the Warg was: reality.
Despite it being based on a fantasy set of novels, the driving force behind visual effects supervisor Jason Smith’s approach to just about every VFX shot or sequence in Prime Video’s The Lord of the Rings: The Rings of Power was to find something real to ground the effect in.
Númenor, as an environment, came from real locales. The Sundering Seas started as real ocean plates. The orc Warg featured chihuahua, and other, reference. And of course the eruption/formation of Mount Doom was a conglomerate of real-world phenomena.
Here, Smith, who hails from Industrial Light & Magic, tells befores & afters about his philosophy and methodology behind acquiring earthly references for these key Middle-earth moments. Indeed, Smith’s determination to capture practical reference from a boat at one point made him incredibly sea sick (but it was worth it, he says).
b&a: In this show, the matte paintings and 3D worlds were incredible, especially for Númenor. I wonder if you could talk a bit about your philosophy for environment work here.
Jason Smith: Well, thank you so much. It’s great that you felt that way and great that you noticed because it was one of the things that I really endeavoured to push hard on this project. Because a lot of the time, I’m more of a creature guy. A lot of the shows that I take on, I do so because there’s some creature that I’ve always wanted to do.
Now, this project was definitely a creature project too. But growing up, being into Tolkien, part of what caught my attention about Tolkien were the settings and how real -and big- every place felt. I remember watching Rankin/Bass cartoons, and those caves felt so real. And there was never an end. You never felt, ‘Oh, now I’ve seen the whole city.’ You always felt like, oh man, that cave was creepy, but there was a dark spot down there that might be another passage. And we never even looked down there. Who knows what’s down there? And your imagination wasn’t boxed in.
All of this is to say that there was a lot of mystery to me, in Tolkien’s environments, and a lot of the magic was there. And so when we started this, I just didn’t want to produce things that felt like a CG cartoon or a theme park. This was something Ron Ames and I started talking about very early on. He’s a producer on the show and the visual effects producer but also a creative partner. He is a big proponent of this whole approach. I didn’t want Númenor, in particular, to feel like a theme park. I didn’t want Khazad-dûm to feel like a theme park. And so the philosophy was that every single thing that we created, we wanted to be based on something real.
When I was tackling Khazad-dûm, for example, he supported us abseiling down into the Lost World Cavern with a team of people who could do scans and photography and who could guide us through the caves. We were alone in the caves, climbing and swimming through and doing scans. We would turn out all the lights and sit in the pitch black while a LIDAR scan captured the rocks and stalactites and stalagmites.
There are many examples across the show where I would look at these fantasy things–take a frozen waterfall that’s thousands of feet high and above an icy valley. And I would think, okay, this is a fantasy thing, for sure. We’re already starting off on the wrong foot. But I wanted to make something that people look at and say, ‘Oh, I think that’s real.’
So we would always just try to find, what is the real thing that we can use, that we can leverage? The frozen waterfall, for example: sure, I can’t find a frozen waterfall that high, that’s above a frozen valley, that has this enormous waterfall associated with it. But what we could do is we could find the pieces. Our locations team, at our request, found a frozen waterfall called Wye Creek in the South Island. And it only had two more weeks until it was going to melt! So we went straightaway and shot that with our helicopter team.
And then we found Bowen Falls, which is a beautiful, big waterfall in the South Island of New Zealand. We cast that to be the waterfall. And then we cast this beautiful valley with winding rivers below Castle Mount to be the valley below.
Every single thing we did was based on something real. With Númenor, the production design was a this beautiful deep kind of collaboration with production designer Ramsey Avery, but when it came to making it real as a place in our series, we also did the same thing: We would say, ‘What are the photographs of real cities that feel like we want Númenor to feel?’ And nothing was a direct reference, nothing we could steal directly. But I remember we found different cities in Europe, particularly Eastern Europe, and we were able to look at aerial footage or photographs. They weren’t always the most beautiful kind of postcard moments, but I picked images that made me feel like, ‘Yeah, that’s a real place.’
We’d find a photo, and we’d say, ‘That has the feel of a real place.’ And I’d put it in a folder, and I’d send it to visual effects supervisor Nigel Sumner at ILM and say, ‘Here’s another one.’ And he’d say, ‘Well, what’s interesting to you about this one? I find it striking that it’s overcast. It doesn’t have any direct sun.’ And I would say, ‘Yes! Exactly! What if we did a shot where there was no direct sun, and that’s just the way it is in Númenor that day? And that’s one small step away from the default look that all CG seems to have. ’
b&a: I even felt like that approach worked for some of the Sundering Seas shots, where maybe in some other film or show, it would be so much more over the top or dramatic, but here it felt chaotic and real where it needed to.
Jason Smith: Yeah. I think there are a couple of reasons for that. One is, ILM already does amazing oceans, right? But we made the choice to go out on a helicopter and shoot ocean plates. And I had to admit to our production, ‘I don’t know how many of these are going to make it into the show. And I have to be honest with you, I’m not going out with certain shots that I’m planning. So the chances that none of them are going to be used directly is real. But what I can tell you is that every single one of these plates is going to go to Industrial Light & Magic. And they’re going to develop their ocean system, matching these.’
I deliberately shot varied plates. Your distance to the ocean is varied. Your time of day is varied. Your direction to the sun is varied. And yes, we shot front-lit plates. We turned around from the sun, let the sun be behind us. And we shot all of those examples, breaking all the rules like that, and came back with tons of reference.
One thing, though, is that Ron recognized that we were too far away from the ocean for many of our shots, because the shots would be on the decks of ships. So then I made the mistake of going out on a boat for almost 24 hours–I found out that I’m fine zooming around in a helicopter upside down around Chicago buildings –but in a boat, I’m totally NOT fine. I’m the opposite of fine. And so I spent 20 hours very, very ill.
But it was worth it because with a crane arm, we were able to get over the edge of the boat, and get right down by the water, and we shot from there. And again, we shot every angle, from pre-dawn until after sunrise. Because on a boat, you can go across that night-day boundary in a way that’s harder with a helicopter. We shot night plates, we shot coming into the very beginning of dawn. We shot towards the sun, and with the sun to our left, and to our right, or even the sun behind us.
Then we took all that footage back, and looked at it, asking, “What are the weird surprising things about these plates? What are the surprising variations?” One of the things that I’ve realized is that we are all smoothers. In dailies, we sit there and we say, ‘I noticed an edge. Let’s smooth that out. Oh, I noticed this weird pop in the guy’s eye socket.’ Because of that, everything starts looking smooth. It starts looking like we’ve done that. It starts looking like we’ve gone in and taken everything interesting.
I remember when I was working as a creature artist on Pirates of the Caribbean I was doing the flesh simulation set-up on the face of Davy Jones. We were looking at all the surprising things the actor Bill Nighy was doing. There was a shot where he turns to Jack and says something like, ‘Price?’ And he does a weird pop, and his upper lip blows up like a balloon and then pops. And all of us were looking at that, going, ‘No animator, I don’t care how good they are, would ever just throw that in and take it all the way to final and show it just saying, Here’s an idea I’ve got.’ It’s too risky. You’re going to spend days animating something that people, all of us, are going to question. Like, ‘I don’t know if that’s really the character of Davy Jones to do that.’ And we’d smooth it out before it had a chance. But the fact that this actor had the guts to make that choice for the character and just do it meant that this surprising thing could be embraced.
With our oceans, it was the same thing. We let some of our oceans be a little bit bland. We let some of our oceans be a little more front-lit. We let some of them have a flatter look, while some of them also had those weird stripes of more or less oxygen. You get these unexplained stripes through the ocean. I’ve never been able to determine exactly what the cause is. But whatever it is, we tried to get as much realism as we could, and we tried to follow that realism without smoothing out what was good about it.
b&a: Jumping to something completely different, creature work, I really like the Warg encounter with Arondir. Tell me about, again, maybe the philosophy for a creature interaction like that. Because you have done lots of creatures.
Jason Smith: Yes, and this is a creature that’s close to my heart. I’m hoping to play around more with this creature in the future if the universe allows me to do that. Because it’s one of those creatures that, going in, well, I knew that we could make a werewolf. Alternatively, I knew we could copy what had been done in the previous incarnations. But neither of those things were all that exciting. Doing the werewolf would certainly be fun. But what was really exciting to me was the opportunity to say, ‘What would an orc’s pet giant dog monster be?’
We tried a bunch of designs. You might have guessed this, by the way, but we looked at some videos of chihuahuas. Now, I think these dogs loved their owners, but when their owner comes in to pet them, they’ll start this face. It is the scariest thing you’ve ever seen. And then they start fighting their owner. And they’re really going for it. They’re lightning speed, and they’re going in, biting, biting, biting, biting. They’re not hurting the owner at all. The owner’s laughing, which seems to make the dog more furious and they’re sitting there biting over and over.
What I realized was, if you took the bravado of that eight-pound animal, and the pure hate that it’s showing to this person who feeds it, and you scaled that eight pound thing up into a creature that’s almost as tall as I am and weighs about 800 pounds of mostly muscle, well, if you had that, it would be the scariest thing ever. We took the things about these small dogs that make them so terrifying in their own way, and we funnelled that into this character.
The ice troll we created for the Durnost sequence may have been terrifying but, the ‘insane chihuahua inside a 600lb pug’ as the team describes the Warg in Episode three, is definitely next level.
.#LOTR #LOTRTheRingsofPower #TheRingsofPower #amazonstudios #VFX #WētāFX pic.twitter.com/ci8vH234Cr— Wētā FX (@WetaFXofficial) January 10, 2023
We wanted to make a creature that was just horrific for reasons that you couldn’t quite understand. Part of it is the familiarity, that it might remind you a little bit of that pup that’s sitting right next to you. And if that pup scaled up, would he still put up with you? If that tiny dog scaled up to this 800-pound nasty thing, would you survive an encounter with your own chihuahua?
All the interaction was planned based on experience from recent projects. Just knowing that you have to get the interaction right, or the whole thing dies. And you can’t go to a digital double any sooner than you absolutely must. That has to be a last resort. Luckily, we were with Wētā FX when we needed one, so it was done extremely well.
Of course there are amazing stunts, too. Wire work and even other stunt people pulling the victim around manually. And I think that that helps to keep the feeling that this creature is interacting with them.
And then one other thing I’ll say is the messiness as well. Every moment of that creature had multiple clips of four-legged carnivorous animals that we were referencing. And we had them picture in picture with the animation. Every single moment of that creature had a real moment of a creature from earth that we were referencing. And guess what? They do weird things. They do unexpected things. That tongue lick that’s so unexpected and looks almost cute. That’s a wolf doing it, that we referenced. And it’s not cute. It’s a very scary thing.
Even trying to inject some of that messiness into the interaction with the elves where it comes around the guy and picks up his thigh and then goes down to eat him. It’s not just a direct attack. He comes around and picks up and then goes for it. So trying to make things as surprising and weird as possible, I think, is part of the way that we approached it.
b&a: One thing that is so memorable from the series and also surprised so many people was the flooding and then volcanic eruption of Mount Doom. I’m always interested in what was on the page, what you had to do in planning and prep, and then in executing this.
Jason Smith: Well, that’s such a great question. It’s fun to actually think back because I had a uniquely privileged position on this project in that I walked in the door when they were still breaking the episodes. This is an idea that they had, back to their very first meeting about the show. Their very first pitch to get the show, they had already thought of this idea, and it was on the page. When I first read about it, it was only a couple of sentences, as you can imagine. It was like, ‘Then this object gets used to open this gate, which actually causes the water to flow this way. And it causes this event, based on a real phenomenon, which is a water induced volcanic explosion.’
Back then–we’re now talking three and a half years ago–that definitely stuck with me. I started thinking about it for a while. I knew I had plenty of time to even decide who the VFX vendor was. I started working with a little editorial team there. I said, ‘Anything that you can find on volcano eruptions. I don’t care if it’s real world or not. Even the cheesy volcano movies from the seventies. I want to see them. I want to see how this has been looked at before, and I want to see the real world reference. I want to see basically everything.’
We came up with hours of stuff to look at. I’d look at the things about that that made it horrific and scary and interesting. As I was looking at all that, some of it did fold into conversations with the showrunners and directors as well. I remember actually talking to them about some of the ways that I thought we could make it scarier by drawing it out, and having them see it happen before it’s immediately upon them.
As we started discussing it, this idea that they see it before they hear it came into being. And after they hear it and then feel it with the shockwave it finally hits them. That three-beat progression is like that video of a tsunami that comes in, and it’s so slow. It’s that dread– you’re feeling the tsunami rising, but you don’t get how dangerous it is yet, and so you’re just watching it. But then it just keeps getting worse and worse and worse. That’s scarier than an immediate impact.
In visual effects, we were uniquely connected with editorial as well. That meant that we could go in and, even as we were even just coming up with storyboards, we didn’t have any footage shot yet, we were able to start cutting boards together and talking with the showrunners and the director of that episode, Charlotte Brändström, and saying, ‘What about this? What about this turn of events?’
They were so welcoming to us. I’ve got to say, it was really, really satisfying to be able to be a part of the unfolding of that event. I’ve seen many times that things may not be explicitly on the page in visual detail, but the writers will use enough words to give us the FEELING they want and then we know what to do visually. So in a way it’s on the page because the feeling is there.
The vendors really brought some amazing work on that sequence, too. We had three vendors on it: Wētā FX, ILM, and Rising Sun Pictures Australia. We had two vendors who were doing the same pyroclastic ash clouds, incredibly detailed fluid simulations, and they couldn’t NOT match. So we treated everybody like one company. Every week, we updated the edit, and we sent everybody’s work out to everybody else as a sequence cut. They all welcomed that exchange. It really took a team effort, but it was like these three companies, for that moment, became one company, and they all pulled together to get the sequence done beautifully.
b&a: I thought there was some really nice use of scale differences in the show that felt different to Lord of the Rings, because of course it had its own approach to scale. Tell me about the approaches you took.
Jason Smith: I think what was unique about our process was that we had a lot of structure around it. Before we showed up for any day to work on those kinds of shots that need scale, we had a process in place that allowed us to rehearse it and know it and be ready to do it ahead of time. We’d know exactly what the platform heights needed to be for forced perspective or exactly where the camera needed to be or for shots that were motion control, we had a to-do list for the entire production.
We also had a little ‘Ten Commandments’ list for scale. The first one was ‘sell hard and sell early.’ All that means is, when you come into a scene with multiple scales, show it off. Be big. Move the camera, have them hand something off, really see the faces. Because what you earn, right there, is a little trust. And the audience settles back into their chair, and they relax a bit.
And now you can do easier shots over doubles all day long. And nobody’s even thinking about it anymore. And you’re getting through those shots on the filming day so quickly. It’s great. But you’ve earned it. You’ve earned that reprieve.
Another one of the commandments that I like was ‘change techniques.’ As soon as the audience figures out what you’ve done, you’re already using a different technique. So as soon as people are saying, ‘I bet that’s motion control…oh, wait a minute, maybe not.’ And that’s what we want. We want that moment where we’re having them look over here, like a magician. And then right when they’re looking more closely here, whoop, we’ve done something over there. Now we want them to look over there. And then, whoop. It’s that back and forth. It’s really a type of sleight of hand.
