[Tutorial] Creation and Setup of Realistic Rocks for AAA Games in Unreal Engine

Revised on June 2023, Original Post in Arstation, 2020

Hello there!

I would like to share today a tutorial showing my workflow to create realistic rocks for video games. This tutorial is based on this project https://www.artstation.com/artwork/Oox8WJ where the creation of the rocks was the key to the art piece. I would like to explain the step-by-step process, and hopefully help more artists to create if they find it useful. If you have a different workflow or something to improve let me know, we are here to learn from each other! In my experience some resources and tips could be difficult to find, this is a mix -or compilation- of all I learned about this subject and the best way to develop them, so let's have a look at this... ROCKS!

(If you don't want to follow the whole process and you are only interested in the shader to apply to your rocks you can download it here ready to use: https://gum.co/UXFCf)

0. Research

One of the most important (if not the most) of the whole process. But, what are we looking for exactly? rocks are rocks! Well, searching for references, in this case, helps to see what kind of rocks you want to do: how the shapes or deformation of these rocks are (pebbles, big rocks, cliffs, broken rocks) how weather, terrain, humans or time has affected them... There are a lot of variables depending on all of these factors, so you should have a clear idea of what you are looking for. In my case, I was looking to do a snow-path forest, just a small piece. Going small helps me to focus more on the few props I did instead of thinking about the whole environment. Less stuff, more dedication.

1. Sculping rocks

Let's start with the sculpting process. although I don't want to focus too much on this point since you can find hundreds of tutorials about sculpting much better and up-to-date, I prefer to focus on optimization and setup, but I'll summarize my workflow.

I start with blocking in Blender. I know some people start directly in Zbrush from a sphere or cube but I think it's better if you do the blocking with the main shapes first, it's much faster and you have more control of the mesh, so let's do it. After that, we import it to Zbrush.

I activate Dynamesh first, set it up at 128 resolution or even 64 should be enough, don't go too high at the beginning as we only need to shape the main shapes (deactivate it after using it). I like to start with the ClayBuildup brush to create shapes and TrimDynamic to refine them. I recorded a small example to show how it goes:

These are the brushes that I use in each step, and how I use them

I change the default material to other ones which allow me to see better the geometry and details (see a few examples below). Starting to sculpt could be a bit frustrating at first —  it can feel very random since rocks are 'abstract'. That's why you'll need references and studying them to see how they are and see their 'anatomy', practice, and practice! I'm still doing that, I still have a lot to learn.

After finishing the main shapes, we can subdivide them to add more geometry and keep working on secondary shapes. Sometimes you'll find small lines or cracks that can break the geometry in the future, but before that happens we are gonna fix it with Dynamesh. It will create a new topology and will fix any problems in the geometry which is necessary before continuing to add more density and details to our geometry. I picked some pictures of the process to show the process.

For the last tweaks, I use alphas for the finished details, adjusting the Z Intensity and working on them from Morph brush or Flatten/TrimDynamic brushes (you'll need a dense geometry already for it). Some alphas I used are this incredible collection of Jronn (https://www.artstation.com/artwork/KazOQ9) or the famous Orb Cracks (https://gumroad.com/l/nOkHw). That said, take your time for the final touches. One thing I do to save geometry if I'm not gonna use the 360º of the rock is removing the bottom, it's something you can do from the beginning, but I recommend doing it at the end, before exporting the high poly.

As I said before, I swap the default material of Zbrush with another one which allows seeing the shapes and details of the rocks better. Here are some materials I used. My all-time favorite: ok_cool_dawn, see more great Matcaps here: https://pixologic.com/zbrush/downloadcenter/library/#prettyPhoto

Once you have the high poly ready, you can export it. I prefer to do it as. FBX, so you can go to Zplugin > FBX ExportImport plugin > Export.

2. Optimization in Zbrush. Decimation Master

We worked hard and we got our rock, now it's the time for some magic, also called auto-retopo. Decimation Master, in a few words, is a tool that computes -with magic- all the geometry of your mesh to optimize the polys in the best way possible, as low as you want, with certain limits, of course. You can find the tool in Zplugin > Decimation master, now let's see how that works.

First of all, we have to Pre-process our mesh or meshes (in case you have more subtools) so let's go to Decimation Master and press 'Pre-process Current' or 'All' depending on the case. It can take a bit of time, depending on your computer and the thousands/millions of polys of the mesh, you will see an orange bar of the process on the top.

After the process is done, we go back to Decimation Master where we can see a few options to adjust the density of our new retopo, you can type the maximum polys you want for the mesh or a percentage. I usually set it up to have between 3-7k depending on the rock. In this case, I would like to have something around 3K, so we type it and click on 'Decimate Current' or 'All', depending on the case. This process is really fast and in a few seconds, we'll have our retopology almost ready. (I'm hurting character artists' hearts here). You can have a look with Shift + F. Have a look at the whole piece to make sure everything looks correct and we don't have anything weird or strange, also make sure the shape is correct and not deformed in comparison to the high poly version, otherwise, we'll have problems baking it later, find the balance. I illustrated in the picture below a good balance for it. The next step is to export the mesh in  a .FBX as we did with our high poly rock and go to our 3D software to clean up and keep optimizing it.

It's time for clean up! And I'll use Blender in this case. You can follow the same process in 3ds Max or Maya but the tool could be different, so if you are using a different software have a look at how it is done there.

Decimation Master is magic, but not perfect, so we can find some imperfections or problems in our model, that's why we have to do a cleanup pass now. Let's select the whole rock in Edit Mode and let's open the Merge menu by pressing M in Blender 3.2 (Alt+M in 2.89 version or before). We'll see our Merge menu where we want to merge the tris by proximity, so choose 'By Distance' and then we'll see a small bar in the bottom left corner to choose the merge distance. The distance that we want will depend on the size of the rock, sometimes 0.1mm is enough, with big rocks I have to put 3-5m or more. When we do that, we'll see how many polys will be removed in the bottom right corner, adjust the distance to clean as many polys as you need to have a beautiful piece of rock, Sometimes it's good to do it even if you don't see anything weird, just to make sure that we don't have overlapped tris or faces. A video with the process:

3. The UVs

The way to do UVs here is really easy. We have to chop the rock in segments in such a way that we should have big parts as 'flat' as we can. When we finish with the UV the best way to organize our UV Map is like if we tried to put them together again, it will help later to have a better performance when we start to set up the shader and textures in our rock in Unreal.

Before we finish, add a check material to the mesh to see that everything wraps correctly. I use the plugin Texel Density Checker.

After that, create a standard material, and export it as .FBX. Let's bake it!

4. Baking the rock

There are multiple options here, I use Substance Painter, but you can make the bake using XNormal for example (https://xnormal.net/) since we only need our normal map from the high poly rock. I'm not gonna spend time here, but if you need more information about baking you can see this tutorial: https://www.youtube.com/watch?v=QJQBaXDKn4w

Some of the adjustments I do before baking it are;

Set up Antialiasing Subsampling by 8X8 and increase Secondary Rays from Ambient Occlusion and Curvature. When you get it you can export all the textures as Unreal Engine Textures if you want, but we only need the normal map since we are gonna create kind of procedural textures in Unreal.

5. Setup our rock in Unreal Engine - Normal Maps

We are going to create a series of nodes to set up the textures of our rock. Before starting, we'll need an Albedo and a normal map that you can download from Quixel Bridge for example, which is free. They have hundreds of textures! https://quixel.com/bridge Once you have it all, Import the material/texture and your normal map into Unreal.

Let's start creating a new material and dropping our original normal map there. We aren't gonna use a Metallic or Roughness map to save resources so we just set them up in 0 and 0.5-0.7 respectively. To have more control of our normal map we are gonna use a Component Mask to separate the R and G channels from B. We do that to increase the R and G channels while keeping B. We create a parameter and connect both to Multiply, which goes to Append (A). In Append (B), we connect the B Channel of our Normal Map. We can control the intensity of the map easily with the Material Instance later. It's always better with a visual ref:

Now we are gonna add extra detail to our rock, it helps to have better detail and avoid blur when we are close (Isn't it great?). Take a second Normal Map (the one you downloaded from the material in Quixel before) and create a TextCoord and a Parameter as I show below to multiply the texture since we don't want big shapes, only adding extra details. Then we create (or copy/paste) exactly the same nodes as our previous Normal Map with the exception of connecting B Channel in Append (B) since we don't need to Intensify the texture, only multiply it. Then create an Add node and connect BOTH Normal maps, the Main one to A, the second one to B. If you don't like the result (after playing with the values of the material instance, we'll see that in a bit) you can download and test different normal maps, the combinations are unlimited.

6. Setup our rock in Unreal Engine - Adding Snow/Grass

Now we have our normal map, which can be increased, and a secondary one to add extra detail. It's time to add some snow to our rock. We can use grass instead or anything really, but in this case, we are gonna use snow. As before, we'll need a Snow (Albedo texture) that you can download in Bridge Quixel for free.

The best thing about this setup is that the snow or grass will be always on top of the mesh, no matter if you rotate or transform the mesh. It's an amazing tool to reuse our rocks and place them really quickly with great results.

We are gonna import our Albedos, one for the snow in this case, and another one for the color of the rock itself and we are gonna add a Vertex Color and WorldAlignedBlend nodes. This allows us to choose where we are gonna add the snow texture, by default it will be on top but we can also change the position and add it to the sides, as I did to add the snow to the trees. We'll need two parameters, one to increase the snow, and another one to adjust the falloff or dissolve, as you can see in the picture.

After that, we just have to create a Lerp node to connect our Albedo rock texture to Lerp A, our snow to B, and WorldAlignedBlend (Explicit Normal output) to Alpha. Finally, connect the Lerp node to the Base Color, and everything is done! I show here the final material setup ready to use (You can also download it at the beginning of the post)

Now we should create an instance material and drop it into our rock mesh, from there, we can adjust all the parameters there, as I show you below. You have to understand the shader so take your time to set it up, sometimes the secondary normal map may not be good enough, or maybe the Albedo doesn't look good, so try different maps. We can create faster combinations for our amazing rocks now! And not only rocks, as I mentioned earlier, I use these nodes to add Snow to the trees too.

That's all!

I hope you find this tutorial useful, if you have any questions about the process let me know - I'm trying to talk more about environments for video games and Environment art in this blog. With time, you will find more information and resources here.

See you soon!

Fran