NB: this article was written in 2013 and is extremely out-of-date. Unity now has native support for blendshapes.

The humble blinking animation: Without it, our characters wouldn't look quite as alive.

This post is about my frustrated attempts to make a character blink, which eventually led me to make the Unity3d games engine a bit more animation friendly – which for me means: a bit more like Maya.

Blend-Shapes in Maya

In Maya, if you want blinking or any other facial animation, you use blend-shapes. This is known as 'morphing' or 'shape interp[olation]' in other 3d packages. When blend-shapes are used, the mesh you see is a product of several different meshes. The influence of each other mesh on your base mesh (usually expressionless, if a face) is determined by a weight value, which can be key-framed. This allows the animator to smoothly morph between different intensities of different facial expressions, even mixing expressions together.

I used Maya's blend-shapes for this animation:

Blow-In 2 (without subs) from Simon Crowe on Vimeo.

All of the blend shapes were modified duplicates of the main head mesh:

Alternatives to Blend Shapes in Unity3d

Unfortunately, theUnity3d games engine, to my knowledge at least, lacks support for blend-shapes. The alternative is using 'bones' (a hierarchical skeleton of transform objects) to animate faces – these are called joints in Maya. For most animal models, the bones correspond to actual bones:

I stole this image from Scott Petrovic's blog. (click image to visit) I hope he doesn't mind!

 

This system of animation is very well suited to limbs and spines, but is not at all suited to the complex muscle structures of the face, where movement does not primarily result from the use of bones as levers.

Wait! Maybe I'm digressing a little. A simple action like blinking would be possible with bones, right? Well, I attempted to set up my character's eyelashes in Blender, and gave up after Blender didn't do what I expected it to do (i.e. What Maya would have done.) My idea was to group the vertices of the top eyelash to a bone centred inside the eyeball, which would rotate up and down to make the character blink. While this might have worked, I suspect it would have been incredibly frustrating to get it to look right.

Making Unity3d (Sort-of) Do Blend Shapes

I turned my attention to implementing blend shapes in Unity – something I'd worked out how to do  in theory, at least – years ago. I didn't do the sensible thing and look for other people's blend shape scripts and plug-ins, I dived right in!

To my credit, my idea worked. All the problems were Unity-specific. I put the nuts and bolts of the system in an abstract class, so all actual implementations could just focus on changing the weights of each blend shape. Here is that class, with a few comments to attempt to explain the logic of the code:

using UnityEngine;
[RequireComponent (typeof (MeshFilter))]
public abstract class BlendShape : MonoBehaviour {
	public Mesh[] shapes;
	public float[] weights;
	protected Vector3[][] differenceVerts;
	protected Vector3[] baseVerts;
	protected Mesh baseMesh;
	
	void Start () {
		MeshFilter baseMeshFilter =
			(MeshFilter)this.GetComponent("MeshFilter");
		baseMesh = baseMeshFilter.mesh;
		baseVerts = new Vector3[baseMesh.vertices.Length];
		for (int vertIdx = 0; vertIdx < baseVerts.Length; vertIdx ++) {
			baseVerts[vertIdx] = baseMesh.vertices[vertIdx];    
		}
		differenceVerts = new Vector3[shapes.Length][];
		for (int shapeIdx = 0; shapeIdx < shapes.Length; shapeIdx ++) {
			differenceVerts[shapeIdx] =
				new Vector3[shapes[shapeIdx].vertices.Length];
			for (int i = 0; i < differenceVerts[shapeIdx].Length; i ++) {
				differenceVerts[shapeIdx][i] =
					shapes[shapeIdx].vertices[i] - baseVerts[i];
				// For each vertex in each blend shape mesh, we calculate
				// and store the difference between that vertex's position
				// and the position of the corrosponding vertex in the base
				// mesh.
			}
		}
		Initialize();     // The Initialize method must be overriden in any
				  //class that inherits from this one.
	}
	void Update () {
		ProcessWeights();     // ProcessWeights can be overriden and filled
				      // with anything that needs to be done every
				      // frame to ensure wieghts are correct.
		Vector3 addendVert;
		Vector3[] newVerts = new Vector3[baseMesh.vertices.Length];
		for (int vertIdx = 0; vertIdx < baseVerts.Length; vertIdx ++) {
			addendVert = Vector3.zero;
			for (int shapeIdx = 0;
				shapeIdx < shapes.Length; shapeIdx ++) {
				addendVert += differenceVerts[shapeIdx][vertIdx]
				* weights[shapeIdx];
				// The differences in poition we calcualted earlier are
				// added together for each vertex - each multiplied by
				// a particular blend shapes's weight.
			}
			newVerts[vertIdx] = baseVerts[vertIdx] + addendVert;
			// The sum of all weighted differences are added to the base
			// vertices, producing blended vertex positions!
		}
		baseMesh.vertices = newVerts;
	}
	protected abstract void ProcessWeights();
	protected abstract void Initialize();
}

 

If we make sure that all our blend shapes are copies of the original mesh with the same number of vertices and assume that, when Unity imports the meshes, this will remain the case – this should work. The last assumption is a big one, and can easily be wrong. Without going into details of how I managed to get my eyelash mesh to work with my script, here is the code I used to animate the blink:

using UnityEngine;
using System.Collections;
 
public class BlinkBlendShape : BlendShape {
	float lastBlinkEnd = 0;
	float blinkStart;
	float lerpEnd;
	float currentWait;
	bool blinking;
	bool hasStarted;
	bool hasPaused;
	public float wait = 6;
	public float waitRandomMin = -0.75f;
	public float waitRanomMax = 0.75f;
	public float startDuration = 0.08f;
	public float pauseDuration = 0.05f;
	public float endDuration = 0.075f;
	
	protected override void Initialize() {
		currentWait = wait + Random.Range(waitRandomMin, waitRanomMax);
	}
	protected override void ProcessWeights () {
		if (blinking) {
			if (!hasStarted) {
				if (Time.time < lerpEnd) {
					weights[0] = 1 -
						((lerpEnd - Time.time) / startDuration);
				} else {
					hasStarted = true;
					lerpEnd = Time.time + pauseDuration;
				}
			} else if (!hasPaused) {
				if (Time.time > lerpEnd) {
					hasPaused = true;
					lerpEnd = Time.time + endDuration;
				}
			} else {
				if (Time.time < lerpEnd) {
					weights[0] = (lerpEnd - Time.time) / endDuration;
				} else {
					blinking = false;    
					lastBlinkEnd = Time.time;
					currentWait = wait
					    + Random.Range(waitRandomMin, waitRanomMax);
					hasStarted = false;
					hasPaused = false;
				}
			}
		} else {
			if (Time.time >= lastBlinkEnd + currentWait) {
				blinking = true;
				lerpEnd = Time.time + startDuration;
			}
		}
	}
}

 

Blinking should be a simple action, but my BlinkBlendShape class is actually longer than its parent: Blendshape. One reason for this is that I used random time values to make it more realistic. No-one blinks exactly every five seconds!

I shudder at the thought of how long and complex a script, or set of scripts, would have to be to allow for key-framed blend shapes like those in Maya. Perhaps something with simple methods like StartSmiling(), StopSmiling() or SetSmileWeight(float weight, float transitionDuration), would be more suitable for a non-linear game.

A Random Thought: Besides Facial animations, What Could We Use Blend Shapes for in a Game?

  • Changing levels of fluid in containers.
  • Wobbling viscous projectiles.
  • Rapidly growing plants.
  • Parasites crawling beneath skin.
  • Blooming flowers.
  • Boiling liquids.
  • Gas clouds.
  • Pulsating organs.
  • Undulating Surfaces.
  • Flowing, oozing, flopping substances.
  • Porcupine-like bristles.
  • Sphincters.
  • Soap bubbles.
  • Springs, maybe...
  • Massive slow-moving raindrops, like on the moon Titan.
  • Shape-shifting lizards! (Within reason – depending on how well the changing vertex positions worked with a skinned mesh.)

To conclude...

The image at the top of the page was the result of my own blend shape script. It works, at least for blinking, with a few limitations:

  • When unity imports a mesh, there is no guarantee that the number of vertices will be the same as your original model. One reason for this is that if a mesh has hard edges, Unity will use two separate vertices and edges to make the edge render as hard and sharp. Another is that if your mesh has UVs, Unity has to make two vertices for every one that is a seam in the uv map.
  • Two sided meshes (where each outward facing vertex has an inward-facing counterpart) like my original eyelash mesh, import with unpredictable vertex numbering. There are probably other types of meshes that do this.
  • It probably wouldn't be that efficient when working with multiple head meshes, rather than one mesh containing 100 vertices of one-sided eyelash. It's a script when it should really be a plug-in; It uses C# – a high-level language – do something that could be more efficient in a lower-level language like C++.

This post was more of an update on how I'm getting along with a project, than a feasible solution to a coding problem. Perhaps someone will use this as a starting point for a better implementation or direct me to a highly mature Blend-Shapes-in-Unity project with far fewer limitations.