Home » Software development

MediaPipe: Enhancing Virtual Human Beings to be more practical– Google for Developers Blog Site


A visitor post by the XR Advancement group at KDDI & & Alpha-U

Please keep in mind that the details, utilizes, and applications revealed in the listed below post are entirely those of our visitor author, KDDI.

AI generated rendering of virtual human ‘Metako’
KDDI is incorporating text-to-speech & & Cloud Making to virtual human ‘Metako’

VTubers, or virtual YouTubers, are online performers who utilize a virtual avatar produced utilizing computer system graphics. This digital pattern came from Japan in the mid-2010s, and has actually ended up being a global online phenomenon. A bulk of VTubers are English and Japanese-speaking YouTubers or live banners who utilize avatar styles.

KDDI, a telecom operator in Japan with over 40 million consumers, wished to try out different innovations constructed on its 5G network however discovered that getting precise motions and human-like facial expressions in real-time was challenging.

Producing virtual people in real-time

Revealed at Google I/O 2023 in Might, the MediaPipe Face Landmarker option finds facial landmarks and outputs blendshape ratings to render a 3D face design that matches the user. With the MediaPipe Face Landmarker option, KDDI and the Google Partner Development group effectively brought realism to their avatars.

Technical Execution

Utilizing Mediapipe’s effective and effective Python bundle, KDDI designers had the ability to find the entertainer’s facial functions and extract 52 blendshapes in real-time.

import mediapipe as mp
from mediapipe.tasks import python as mp_python

MP_TASK_FILE = " face_landmarker_with_blendshapes. job"

class FaceMeshDetector:.

def __ init __( self):.
with open( MP_TASK_FILE, mode =" rb") as f:.
f_buffer = f.read().
base_options = mp_python. BaseOptions( model_asset_buffer= f_buffer).
alternatives = mp_python. vision.FaceLandmarkerOptions(.
base_options= base_options,.
output_face_blendshapes = Real,.
output_facial_transformation_matrixes = Real,.
running_mode= mp.tasks.vision.RunningMode.LIVE _ STREAM,.
num_faces = 1,.
result_callback= self.mp _ callback).
self.model = mp_python. vision.FaceLandmarker.create _ from_options(.
alternatives).

self.landmarks = None
self.blendshapes = None
self.latest _ time_ms = 0

def mp_callback( self, mp_result, output_image, timestamp_ms: int):.
if len( mp_result. face_landmarks) >>= 1 and len(.
mp_result. face_blendshapes) >>= 1:.

self.landmarks = mp_result. face_landmarks[0]
self.blendshapes = [b.score for b in mp_result.face_blendshapes[0]]
def upgrade( self, frame):.
t_ms = int( time.time() * 1000).
if t_ms <

Like this post? Please share to your friends:
Leave a Reply

;-) :| :x :twisted: :smile: :shock: :sad: :roll: :razz: :oops: :o :mrgreen: :lol: :idea: :grin: :evil: :cry: :cool: :arrow: :???: :?: :!:

© 2024 magadanstat.ru