How Respeecher’s Voice Cloning Brought Young Luke Skywalker to Life in The Mandalorian

The Mandalorian is a Star Wars television series created by Jon Favreau for the streaming service Disney+.

The Mandalorian became the most successful Disney+ original series to date, topping the charts for television series in 2019 and 2020. Critics widely acclaimed the first two seasons, which received ratings of 93 and 94 on Rotten Tomatoes. 

The Mandalorian’s success is not only due to the outstanding work of the cast, the showrunner, and the screenwriters. The technology used to create the environments of the Mandalorian plays a vital role in its production quality.

The Mandalorian was the first production to be filmed using real-time rendering for realistic, parallax environments. It's also one of the first TV shows that introduced Respeecher's voice cloning to first-tier movie production.

The Challenge

Luke Skywalker was set to make an appearance in the final episode of the second season of The Mandalorian.

With Mark Hamill being 68 years young, and Skywalker barely twenty years old in the time period of The Mandalorian, Respeecher had its work cut out.

In the Making of Season 2 Finale chapter that aired in August 2021, production teams revealed some details about applying voice cloning in the series.

"When we realized that we wanted to show a younger Luke Skywalker, we looked at all the techniques available," said The Mandalorian's showrunner, Jon Favreau.

The producers settled on using deepfake technologies for visuals and voice to make a younger-looking version of Luke Skywalker.

Jon Favreau's team partnered with Lola VFX for video production and Respeecher for synthesizing the young Skywalker's speech.

The Project in Numbers

The best-kept secret of “The Mandalorian” Season 2 was Mark Hamill’s surprise cameo in the finale as a young Luke Skywalker, who offers to complete baby Grogu’s training in The Force. Yet how Hamill was de-aged has been kept under wraps until the launch of the episode of “Disney Gallery: Star Wars: The Mandalorian” on Disney+.

How Does Voice Cloning Technology Work

In order to clone a voice, Respeecher’s AI-powered speech synthesis software needs examples of that voice (which we call the target voice). Usually, 30 minutes of quality recordings are enough. The software then analyzes the voice until it can produce a clone. Further testing is then conducted to ensure that the synthesized speech cannot be distinguished from the original voice.

Discover more about how we ensure (and even improve) the quality of the recording in this whitepaper about Respeecher’s audio super-resolution algorithm.

Hamill's target voice was captured using many early years' worth of radio broadcasts, interviews, ADRs, and dubs.

After analyzing the younger Luke Skywalker's voice, Respeecher could recreate it in a way that it is almost identical to the real version.

"Something people didn't realize is that his voice isn't real. His voice, the young Luke Skywalker’s voice, is completely synthesized using an application called Respeecher," says Jon Favreau.

Interestingly, this went unnoticed for the whole 9 months between the release dates of the Season 2 and the documentary.

Key Highlights

Aside from using Respeecher’s unique technology, there are several factors that The Mandalorian team makes note of.

One of the crucial goals during The Mandalorian’s production was keeping Luke Skywalker's cameo a secret.

"In the first season, Jon was able to successfully keep the name Grogu a secret before it was revealed. For the second season, we had to make it through the entire season to the airing of the last episode to not have this spoiled. In hindsight, it's a bit of a miracle that it remained a secret, and I'm thrilled because for people to discover that in real-time was magical," says Peyton Reed, Mandalorian's Director.

None of the hundreds of people working on the show leaked the details. This delivered the surprise to all the fans of the Star Wars universe on the day the final episode was released.

“The fact that we were able to keep my involvement a secret for over a year with no leaks is nothing less than a miracle. A real triumph for spoiler-haters everywhere!” - Mark Hamill tweeted.

The second factor that the creators of The Mandalorian paid great attention to was the ethics of using Deepfake technologies.

Considering that we have reached a point where the original video and voice are almost indistinguishable from the fake, the ethical use of this technology is more relevant now than ever.

Jon Favreau shared his thoughts on what anti-fake protection might look like:

"We were alluding to the fact that there's a type of blockchain identification that everybody has. I wonder if certain images or videos that are released in an official capacity could have some kind of a stamp on them. Because it is becoming harder to tell the real from the fake."

While the industry has not yet developed uniform standards for combating the malicious use of deepfake technology, Respeecher has taken some steps in this direction. 

We spend a lot of time and energy to educate the public on the threats of synthetic media technologies and promote ethical standards among deepfake creators and content creators, by showcasing amazing uses of voice cloning technologies in real projects and participating in a number of initiatives in order to guard against the misuse of such technologies.

Conclusion

The entire Respeecher team was honored to work on such a huge project.

The result of this cooperation is sure to attract even more high-profile projects to voice cloning technology. This is important for both the entertainment industry and audiences worldwide. The former gets unprecedented opportunities to create unique content with iconic stars, while the latter gets to see their long-loved heroes on the big screen again.

Voice cloning brings unprecedented opportunities for new talent and content creators, allowing them to better balance their workload between high-demand actors and those who are not facing that intense demand. Respeecher transformed the way productions handle ADR and dubbing, saving time for actors and money for production teams.

The widespread attention and adoption of deepfake technologies will accelerate the need to regulate the use of the technology. Respeecher always prioritizes eliminating the possibility of others being harmed by deepfake technologies, which cannot be done without uniting various media industry players and stakeholders around the goal of adopting synthetic media technologies in the most ethical way.

We deeply appreciate Jon Favreau and his team bringing attention to how synthetic media can reshape media creation in the future, how it could be harmful in the wrong hands, and how we can control it. We also greatly appreciate his showcasing of voice cloning technology in the best possible light on one of the biggest TV shows of 2020.