Picture this: A virtual avatar that looks and sounds exactly like the person it represents but can be driven by the movements and voice of anyone. In this case study, we'll examine Respeecher’s partnership with Pinscreen and the Motus Lab team at the University of Sydney to create a virtual avatar for Mike Seymour, a digital humans researcher and leader of Motus Lab.
The Challenge
Our emotional connection with people we interact with is largely determined by two factors, their face and their voice. Pinscreen, a tech startup founded by Hao Li, and Motus Lab are leaders in the development of AI-based technology to allow one actor to drive the face of another, fully inhabiting another person. One use of this technology is to dub content without creating a disconnect between face and voice. But the visual part of a virtual avatar is only half the story. Equally important is allowing one person to drive the voice of another.
The Solution: Respeecher's Voice Cloning Technology
By converting the voice of Grant Reaber, Respeecher’s Chief Research Officer, to sound like Mike Seymour, the team managed to create a stunningly lifelike virtual avatar, with both Mike’s face and voice being driven by Grant’s performance.
How was this achieved? The team shot the demo with multiple cameras tracking Grant's facial expressions to ensure the visual model had an accurate representation of them. No special capture sessions or training on Grant’s appearance was needed. Meanwhile, Respeecher trained a model on Mike's voice capable of converting Grant’s (or anyone’s) speech to Mike’s. Though Grant and Mike have very different speaking styles and accents (Grant is American and Mike is Australian) the effect is compelling.
"The research teams are rapidly advancing the art of re-enactment to deliver a faster and more faithful rendition of the original actor’s actions in a way that is most true to their performance," said Mike Seymour.
The Result
The collaboration between Respeecher, Pinscreen, and the Motus Lab team was nothing short of magical.
"Today, while AI is rapidly advancing, it is great to see two ML companies that started years ago still expanding and servicing the M&E industries today," noted Seymour. Despite challenges from the war in Ukraine and working through extreme conditions, Respeecher's team continues to service partners and productions worldwide.
Conclusion
This case study is an example of how AI and machine learning are transforming the entertainment industry, particularly in voice and speech technology. Respeecher's collaboration with Pinscreen and Motus Lab demonstrates the incredible power of combining visual and audio re-enactment to create more authentic and engaging experiences for audiences.
