Google Deep Mind
Deep Mindshowed off the latest issue from its generative AI picture - to - audio recording research on Tuesday . It ’s a novel system that combines what it see on - screen with the user ’s written prompt to create sync audio soundscapes for a given video clip .
The V2A AI can be paired with vide -generation models like Veo , Deep Mind ’s generative audio team wrotein a blog post , and can produce soundtrack , reasoned effects , and even dialogue for the on - screen action . What ’s more , Deep Mind claims that its young system can generate “ an unlimited turn of soundtracks for any television remark ” by tuning the model with positive and minus prompts that promote or discourage the role of a particular phone , respectively .
Google Deep Mind
The arrangement bring by first encoding and press the picture input , which the dispersal model then leverage to iteratively refine the desired audio effect from scope racket ground on the exploiter ’s optional textual matter prompt and from the visual comment . This audio output signal is lastly decoded and exported as a wave form that can then be recombined with the video recording input .
The best part is that the user does n’t have to go in and manually ( read : tiresomely ) synchronise the audio and video tracks , as the V2A organization does it automatically . “ By grooming on video , audio and the additional annotations , our technology learns to associate specific audio events with various optic scene , while responding to the information leave in the note or transcripts , ” the Deep Mind squad save .
The arrangement is not yet perfected , however . For one , the output audio quality is dependent on the faithfulness of the television input and the system gets turn on up when video artifact or other distortions are present in the stimulant . According to the Deep Mind squad , syncing dialogue to the audio racetrack remain an ongoing challenge .
“ V2A attempt to generate language from the input transcript and synchronize it with characters ’ sass drive , ” the squad explain . “ But the couple vide- generation model may not be conditioned on transcripts . This create a mismatch , often ensue in unearthly sass - syncing , as the TV model does n’t generate mouth movements that match the transcript . ”
The organisation still needs to undergo “ rigorous rubber assessment and examination ” before the team will consider releasing it to the world . Every television and soundtrack generate by this organization will be stick on with Deep Mind’sSynthID watermarks . This organisation is far from the only audio - generating AI currently on the market . Stability AI dropped a similar productjust last calendar week whileElevenLabs free their sound effect toollast month .
