Creating an animated 3D eCard using WebGL + React Three Fiber + GLTF models with animations

This post serves as a companion to the project code in this repo on Github: https://github.com/firxworx/fx-r3f-congrats.

I review how I used use React Three Fiber (@react-three/fiber aka “R3F”) and the popular helper library DREI (@react-three/drei) to create an animated 3D scene featuring GLTF models and 3D text.

Screenshot:

Project Preview Screenshot

React Three Fiber is the React Renderer for ThreeJS, the popular JavaScript library for working with WebGL.

The intended audience for this post is developers who are familiar with React and are interested in working with webGL via React Three Fiber.

Topics include:

  • adding a new 3D model to a scene
  • loading multiple instances of a GLTF model
  • animating a model using Adobe Mixamo
  • playing animations
  • using matcap textures via DREI’s useMatcapTexture() hook

How this came about

I recently started experimenting with 3D + WebGL + R3F for a side-project involving a unique React-powered UI designed for a user with special needs. An example on codesandbox inspired me to create a 3D eCard for a colleague who just had their first child. It was the perfect exercise to figure out the the process of animating models with Mixamo.

My concept was to display a personal message with a dancing baby in front of it, with the lot being encircled by flying brand/logo marques of my colleague’s company. I happened to have an animated GLB model of the company’s logo marque on-hand, originally created by a provider on https://fiverr.com.

When I decided to release the code I genericized the models + caption, pushed it to github, and wrote this blog post. I replaced the proprietary logo marque with an animated stork that I found on Sketchfab.

Getting Started

Although I usually prefer NextJS for professional projects, I went with Create React App for this one to keep things simple.

To bootstrap a new React + TypeScript project with CRA:

yarn create react-app PROJECT_NAME --template typescript

I cleaned up the CRA boilerplate to my liking — I removed some cruft and refactored so that all components including App.tsx were under the src/components/ folder.

To install R3F + DREI + ThreeJS:

yarn add @react-three/fiber @react-three/drei three
yarn add @types/three --dev

Canvas & Scene Setup

The App component lays out the entire scene.

The components in the scene are children of the <Canvas> tag. CSS via the style prop sizes the canvas to fit the screen and the camera prop is used to position the default camera. A more production-grade version might account for the possibility of the user’s window/screen size changing, however this approach served my purposes.

The <ambientLight /> and <directionalLight /> tags are used to add lighting to the scene. The ambient light provides general lighting while the directional light helps ensure that the text is readable.

Learn more about ambient + directional lighting from the ThreeJS docs:

The contents (models, etc.) comprising the scene are added as children of a top-level <group>...</group>.

Finally, <OrbitControls /> are added thanks to DREI. OrbitControls provide mouse interactivity and enables the user to manipulate the camera to orbit the scene. Check out the example usage in the DREI StoryBook:

Scene group

This section breaks down the children of the top-level <group>...</group>.

React.Suspense

Everything inside the parent <group>...</group> is wrapped in a React.Suspense component with the fallback set to a “Loading…” message. The fallback is presented to the user while the scene’s assets are loading.

DREI provides a useful <Html> helper component that I use inside the fallback.

If you want to implement a loader that provides the user with more detailed progress feedback, you can look into DREI’s useProgress() hook.

For more details on DREI’s various helpers and links to their Storybook, refer to the DREI README on Github.

Sky

DREI’s <Sky /> component is used to add a nice Sky to the scene.

A great alternative that also ships with DREI is its <Stars /> component.

Text & TextBlock

I borrowed the font *.blob and the general implementation of the Text component from the codesandbox inspiration.

I quickly discovered that the font data didn’t include a space character so I hacked one in: I formatted the JSON data so I could see what was going on and manually added a glyph for " ".

Next, I changed the Text material from the rainbow-looking meshNormalMaterial to meshPhongMaterial and selected a blue colour because my colleague had a baby boy.

You can learn about the different types of ThreeJS materials here: https://threejsfundamentals.org/threejs/lessons/threejs-materials.html.

I implemented the TextBlock component (found inside App.tsx) similar to the Jumbo component in the inspiration’s codesandbox, modifying the animation to be more of a gentle rocking motion.

The rocking motion is implemented using the R3F library’s essential useFrame() hook.

Invoking the useFrame() hook in a component subscribes it to the render loop. The hook calls back every frame, passing the state (this is the same state as per the useThree() hook) and a clock delta.

The rocking motion is achieved by setting the group’s x + y + z rotation to the sin() of the clock’s elapsed time multiplied by a factor.

GroundPlane

I decided that a breakdancing baby would benefit from some ground to dance on so I added a plane.

The GroundPlane.tsx component is a good example of a simple mesh with a straightforward plane geometry and a material.

A mesh in ThreeJS refers to a polygonal mesh. A geometry represents its structure (in this case, planeBufferGeometry is used) and a material represents its appearance.

Creatively speaking, I was aiming for the visual “suggestion” of a ground that wouldn’t distract from the rest of the scene, so I chose a meshPhongMaterial in a light blue colour for the “baby boy” theme, and I set the complementary transparent and opacity props, choosing an opacity value of 0.5.

DancingBaby

I sourced the Dancing Baby model from SketchFab. This model is the author’s take on the original Dancing Baby meme that went viral in the early days of the web (1996). That was a time when everyone was blown away by a 3D dancing baby: the word meme was not part of the anyone’s general vocabulary, things that went “viral” appeared on cable television news, and the Motorola StarTAC was the hottest mobile device.

The model had no animations but was in a T-Pose. This pose is important for rigging and animating character models.

I downloaded the Dancing Baby in its original format (OBJ) vs. GLTF (preferred for WebGL) because I planned to animate it in Mixamo and Mixamo doesn’t support GLTF models.

There are a few good blog posts out there to help you navigate the animation process with Mixamo + Blender, so I won’t go into the specific settings and technical details in this post. You can learn more here:

Animating with Mixamo

I logged into Mixamo and uploaded the Dancing Baby model.

Tip: when you download models from Sketchfab, you may sometimes find that the zip file contains extra files + folders that aren’t directly related to the model itself. In these cases you may need look through the unzipped contents and find the part of the folder tree that corresponds to the actual model/data/textures, and then use that part with 3D tools like Mixamo.

Once in Mixamo I could see that there were some minor glitches with the Dancing Baby’s textures. They somehow didn’t look as polished as they did in the Sketchfab preview and I thought the eyes looked especially creepy. Since I was already contemplating going with a different look, I dropped the textures at this stage by re-uploading the OBJ file to Mixamo without them.

After the upload and initial processing was completed, Mixamo’s auto-rigger UI prompted me to identify various joints (elbows, knees, etc.) so that it could create a rig (“skeleton”) for the model. If your model comes pre-rigged by its author (this is usually ideal for the best-looking animations) and Mixamo recognizes it and determines it can work with it, then it may skip this step.

Once the auto-rigging process was complete, I navigated through Mixamo’s animation library, found the break-dancing animation I wanted to use (I was specifically searching for something loop-able), and applied it to the model.

Thankfully the auto-rigger did a good enough job that most of the available animations looked pretty good for my needs.

Blender for Conversion

I exported + downloaded the animated model and imported it into Blender, the free + open-source 3D software.

Using Blender, I exported the model to the preferred format for WebGL: GLTF/GLB.

I put the GLB model file in the project’s public/ folder.

Generating a TSX Component

I used gltfjsx to generate a TSX component for the dancing-baby.glb model. The --types flag output TypeScript/TSX vs. JavaScript/JSX and the newer --transform flag will produce a compressed, texture-resized, deduped, and pruned asset:

npx gltfjsx dancing-baby.glb --types --transform

The generated TS/TSX code isn’t always perfect so I had to modify it to get it working with the latest version of TypeScript.

While there has been progress, at the time of writing many of the libraries in the react-three-fiber ecosystem do not have full TypeScript support yet. Thus you may find that you need to revise the code and perhaps add a @ts-ignore/@ts-expect-error here and there to get a model component working.

Playing the Animation

I use DREI’s useAnimations() hook to obtain the action and animation names from the model loaded via useGLTF().

I then added a useEffect() hook to the component to play the first (and only) animation.

MatCap Textures

I used MatCaps to give the baby a metallic/chrome/steel appearance.

MatCap is short for Materials Capture and represents a technique for encoding the colour and shading of materials.

DREI provides the super-convenient useMatcapTexture() hook that accepts the ID of a texture from this library: https://github.com/emmelleppi/matcaps. I simply browsed the previews in the library’s github repo and noted the ID’s of textures I thought would work well for the different meshes that comprised the Dancing Baby.

I revised the component’s TSX and replaced each mesh’s material with a meshMatcapMaterial and passed the corresponding MatCap via the matcap prop:

// using the useMatcapTexture hook at the top of the component 
const [matcapDiaper] = useMatcapTexture('E8E5DE_B5AFA6_CCC5BC_C4C4BB', 1024)]

// ...

// inside the corresponding <mesh>...</mesh> within the TSX
<meshMatcapMaterial attach="material" matcap={matcapDiaper} />

FlyingStorkFlock

The FlyingStorkFlock component renders a bunch of FlyingStork components that each start at a random position and y-axis rotation, and sets the rotate prop (which include speed and factor props) to reflect one of three randomly-selected flight characteristics that I refer to as the vibe of the stork.

Mixing up the flight characteristics of the storks lends some interest and variance to the scene and helps the flock appear more natural.

A key difference from the codesandbox inspiration is that I do not pass a filename as props to component for an individual bird, and in the implementation of the bird component (FlyingStork) I use DREI’s useGLTF() hook to load the *.glb file vs. React Three Fiber’s more general useLoader() hook.

The inspiration’s implementation has the convenient effect of ensuring that each instance of a Bird is loaded from file by the component’s invocation of the useLoader() hook, and this supports rendering multiple instances of the same GLTF on the canvas.

I use a different approach to support rendering multiple storks: after the *.glb file is loaded via the useGLTF() hook, instances of the FlyingStork component store a memoized clone of the scene. I detail this approach further below.

FlyingStork Model

I sourced the Flying Stork from SketchFab. This time, the model included a flying animation so I downloaded it pre-converted to GLTF.

After unzipping the model file, I used gltf-pipeline to compile it to a compressed GLB version that will deliver better performance:

npx gltf-pipeline -i scene.gltf -o model.glb --draco.compressionLevel=10

I moved the GLB output to the public/ folder with the filename flying-stork.glb.

Next I used gltfjsx to generate the React component boilerplate:

npx gltfjsx flying-stork.glb --types --transform

I found once again that I had to customize the TSX code to get it to compile with the latest version of TypeScript.

Supporting Multiple Instances of a GLTF Model

To support rendering multiple storks on-screen from the same GLTF model file, I cloned the stork scene that was returned by the useGLTF() hook when loading the *.glb model.

I learned from the ThreeJS forums that SkeletonUtils.clone(scene) was required vs. gltf.scene.clone() otherwise there would be issues with skinned meshes. React’s useMemo() hook is used to memoize the cloned scene.

React Three Fiber’s useGraph() hook is used to obtain the nodes from the cloned scene. The various nodes are referenced within the props of the different meshes defined in the component’s TSX.

I clone the animations for good measure via animation.clone() and then use the useAnimations() hook similar to the dancing baby case, except this time I pass the clone instead of the original.

Playing Animations

Similar to the dancing baby, I added a useEffect() hook to play the animations.

Circling/Flying Birds

I revised the generated FlyingStork component to accept a rotate prop that accepts an object with speed and factor properties (both of type number).

The FlyingStorkFlock component sets the rotate prop on each instance of FlyingStork that it creates.

In the FlyingStork component, the useFrame() hook is applied in a similar way to the Birds component in the codesandbox inspiration, except it uses the speed and factor values from the rotate prop.

The circling behaviour is implemented by updating the groupRef.current.rotation.y value on every frame:

groupRef.current.rotation.y -= Math.sin((delta * rotate.factor) / 2) * Math.cos((delta * rotate.factor) / 2) * 1.5

Conclusion

And that’s it!

If this post helped you on your path to building something awesome with React Three Fiber let me know in the comments :).

If you fork this project and create your own congrats SPA for your friend or colleague, please share the URL!