Animations & Retargeting

VRM Blendshape Group Name	Triggered by	Asymmetry support	Observations
NEUTRAL	No other expression active	Not applicable
A	Open mouth neutral	Not applicable	not triggered if JOY, SORROW or O expressions are active
JOY	Open mouth while smiling or eyebrows up	Not applicable	JOY has priority over SORROW
SORROW	Open mouth while frowning or eyebrows down	Not applicable	JOY has priority over SORROW
O	Open mouth while lips are pursed (lips funnel)	Not applicable	JOY and SORROW have priority over O
SURPRISED	Open mouth while eyes wide open	Not applicable	not triggered if JOY, SORROW or O expressions are active
FUN	Smile with mouth closed	No	FUN has priority over ANGRY
ANGRY	Frown with mouth closed	No	FUN has priority over ANGRY
U	Purse lips with mouth closed	No	FUN and ANGRY have priority over U
BLINK_L	Blink left eye neutral	Yes
BLINK_R	Blink right eye neutral	Yes
Visemes (Audio Based Lipsync)
A	Visemes AA and AH	Not applicable
I	Visemes IH, Y and FV	Not applicable
U	Visemes OY	Not applicable
E	Visemes EY and EH	Not applicable
O	Visemes AO and AW	Not applicable

Perfect Sync 

The Perfect Sync Animation Standard aims to enhance the expressivity of the avatar with more atomic movements of various parts of the face, giving the user more control over the emotions and expressions one wants to exhibit.

The standard uses 52 blendshapes from the iOS ARKit Face Tracking module, so it is best to follow the ARKit docs on how to build them (see: https://developer.apple.com/documentation/arkit/arblendshapelocation - each Blendshape is demonstrated through images). List of the 52 shapes and expected names down below.

Observations:

• Perfect Sync Animation Standard uses Blendshape Groups rather than raw blendshapes names, same as the VRoid Standard.

• jawForward shape isn’t mapped at all as we feel there’s no suitable tracker (not even ARKit) that can provide the consistent tracking data for this tracking input.

• Viseme mapping is the same as VRoid Standard.

Rigid Head Motions Retargeting

The standard VRM model exported from VRoid Studio has no skeletal animations, so the tracked head motions are directly retargeted on the model’s skeleton joints. The system assumes the model skeleton follows the VRM standard rules (see https://github.com/vrm-c/vrm-specification/blob/master/specification/0.0/README.md#vrm-rule). It is assumed that the retargeted joints have no additional transform and the world transform has the rotation 0, 0, 0 in OpenGL style axis.

The set of joints/bones needed for rigid head motions retargeting is:

• spine

• chest

• upperChest

• neck

• head

The table below shows the motion ranges and the affected bones/joints.

These bones can be configured through the Humanoid System of the VRM extension

Eye Pupil Motions

The eye pupil motions are retargeted in the same way as the rigid head motions, by directly manipulating the transforms of the leftEye and rightEye bones. The range of motion is [-10, 10] degrees for both rotation axis (yaw and pitch), but this may vary depending on how the model is being built. The range was determined through an empirical process done on the sample models from VRoid Studio.

Procedural Behaviors

Animaze supports a wide range of procedural behaviors, including a ‘look at camera’ setting and natural eye movement. These behaviors are implemented through the core tracking interpretation module in Animaze, and are usually applied through skeletal animations and blendshapes that the avatar already features. In the case of VRM models, some of these behaviors need a custom implementation:

• VRM models import in T-Pose, although if you add the idle1 animation, it will be applied. We’ve added an Idle Pose to set the model in a usable state.

• VRM models import in a static T-Pose, so we’ve added a small procedural breathing animation to make the avatar more lively.

• Eyes look-at-camera behavior has been added and uses the Eye Pupil Motions technique for mapping the motion.

• Head look-at-camera behavior uses the Rigid Head Motion technique for mapping the motion, but overrides the tracking data.

Idle Pose

VRM models import in static T-Pose so we’ve procedurally rotated a set of bones to get the model’s hands down. The joint/bones used are: leftUpperArm, leftLowerArm, rightUpperArm and rightLowerArm. The upper arms are rotated 70 degrees around the Z Axis (roll motion), and the lower arms an additional of 10 degrees around the Z Axis.

Note that the procedural Idle Pose doesn’t account for mesh clipping and is currently not configurable

Idle Motion (Breathing)

The idle motion we’ve added is a very subtle breathing animation that affects the chest and upperChest joints/bones. The idle motion is a uniform scaling (all axis) animation in the interval 1 to 1.025, with the assumption that the joint scaling is set by default to 1, as the VRM rules state.

Eyes Look-at-Camera

The look-at-camera behavior uses the head joint position and orientation (so, assumed to follow the VRM rules) to compute the look angle to the render camera and is applied through the Eye Pupil Behavior retargeting technique.

Head Look-at-Camera

The Head Look-at-Camera behavior orientates the head to the render camera and uses the same head joint position and orientation as the Eyes Look-at-Camera, while the animation mapping is done through the Rigid Head Motion technique.

Special Actions & Poses

Special Actions and Poses are supported on .VRM sourced models in a limited way, they need to be imported for the specific avatar in use and there is currently no support for generic animations that can be shared between two or more .VRM sourced models.

There are two main ways of adding special actions or poses:

1. Provide animations with the .VRM model (the base .gltf format supports skeletal animation). After import these animations can be added as Special Actions or Poses through the appropriate Editor function.

        2. Import the .VRM model and then import the special animations as .FBX source files which are detected by the Special Actions & Poses menus in the Editor as animation sources. Note that these FBX animation sources should match 100% the .VRM skeletal hierarchy and have animation keyframes on all joints of the skeleton.