Earlier this year, Respeecher contributed to the 2023 Accelerator project "Synthetic Humans for Entertainment & Accessibility." An objective of the project was to blur the lines between reality and synthetic media using advanced technologies to create lifelike synthetic humans.
The Challenge
To meet the demands of the challenge, Respeecher was going to need a two-pronged attack. The first requirement was to create a synthetic rendition of Maria Callas, the renowned Greek opera singer. We haven’t been focused on Maria’s signing performance, but on her speech, as we were responsible for a part where Maria handles press conference questions. This meant not only replicating her voice but also capturing the nuances and emotive qualities inherent in her speech.
The second requirement revolved around developing a photorealistic sign language interpreter capable of satisfying the demands of accessibility in broadcasting. To deliver on these objectives, every partner involved in the project was going to have to up their game within the voice and motion capture technology space, as well as in creating photorealistic 3D models. If the final result delivered anything less than enhancing the inclusivity and diversity of the media content, the project would be deemed a failure. So, while Respeecher went to work on voice cloning, the other companies spun up their video production departments.
The Solution
Respeecher, Signly, Pluxbox, and 4DR Studios worked on making this vision a reality. Throughout the process, we put multiple motion capture technologies and synthetic human creation toolkits through their paces. Respeecher specifically contributed by cloning voices, notably recreating Maria Callas' voice for interactive interview segments.
We meticulously analyzed and processed audio recordings from her archive before using our AI algorithms to create the most accurate voice model of Callas. The other teams' expertise in lip-sync technology and real-time facial expression capture played a crucial role in making the synthetic Callas both sound and appear more realistic.
The Result
When all was said and done, the project’s final result was due to the collaborative success of every team involved. The synthetic version of Maria Callas not only represented the technical advancement of the synthetic media space but also created an entirely new emotional and nostalgic experience for audiences.
Likewise, the sign language interpreter model showed off its potential for expanded use in broadcasting. Overall, the project demonstrated the potential for synthetic humans to create inclusive, emotionally resonant, and visually appealing content. The published technical paper, as part of the Final POC Results, details the key points of success and lists policy guidelines that emphasize the project's contribution to AI and sustainable production in synthetic media.
