Throughout the years, I have been trying to understand normal mapping and the problems that usually appear when working with them.
Most explanations I found are usually too technical, incomplete or too hard to understand for my taste, so I decided to give it a try and explain what I gathered. I recognize that these explanations may be incomplete or not 100% accurate, but I want to try anyway.
Normals
The first 3D models ever made looked something like this:
This was great, but it had an obvious limitation: it looked too polygonal.
The first obvious solution was to add more polygons, making the surface more even and smooth, up to a point where the polygons would look like a single, smooth surface. Turns out this needs a huge amount of polygons (specially at the time) in order to make surfaces such as a sphere look smooth.
Another solution was needed, and thus normals were invented. (Not really, but its easier to explain and understand.)
Let's trace a line from the center of a polygon, completely perpendicular to its surface. We will call this line the super confusing name of : normal.The purpose of this normal is to control where a surface is pointing, so that when light bounces from this surface, it will use this normal to calculate the resulting bounce.When light hits a polygon, we compare the angle of the light ray to the normal of the polygon. Light will get bounced back using the same angle relative to the normal direction:
In other words, the light bounce will be symmetrical relative to the polygon normal. This is how most bounces work in the real world.By default, all polygons bounce light rays completely perpendicular to their surface (as they would in real life), because polygon normals are, by default; perpendicular to the polygon surface. If there are gaps in the normals, we will see them as separate surfaces, since light will bounce in either one direction or the other.
Now, if we have two faces connected, we can tell the computer to smooth the transition between the normal of one polygon to the other so the normal gradually aligns with the closest polygon normal. This way, when light hits one polygon directly in its center, light will bounce straight, following the normal direction. But, in between polygons, this normal direction is smoothed, bending how light bounces.
We will percieve the transition as a single surface, as light will bounce from one polygon to the other in a smooth fashion, and there will be no gaps. Effectively, the light bounces from these polygons smoothly, as it would do if it had a ton of polygons.
This is what we are controlling when we set smoothing groups (3ds Max, Blender) or set edges to be hard or smooth (Modo, Maya): we are telling the program which transitions between faces we want to be smooth and which we want to be hard.
Here's a comparison of the same sphere with hard and smooth transitions, both with 288 polys:
We could potentially set something like a box so that all its vertices have averaged normals. The 3D software will try to smooth its surface, so it looks like a single, smooth surface. This makes perfect sense for the 3D program, but it looks very weird because we have something that should have several separate surfaces (each face of the box) but the program is trying to show it as a single, smooth surface.
This is why we usually have a smoothing angle setting in 3D software: if we have 2 connected polygons in an angle greater than this smoothing angle, their transition will be soft, and polygons connected in an angle smaller than the smoothing angle will be hard. This way, extreme angles between surfaces will be shown as different surfaces, as they would in the real world.
So, we used normals to control the transitions between faces in our model, but we can take this a step further.
Since we are changing how light bounces from an object, we can also make a very simple object bounce light as a complex one would. That is a normal map. We use a texture to bend the direction of the light that bounces from a 3D object, making it look more complex than it really is.
A real life example of this would be those holograms that were gifted back in the day with potato chips (at least here in Spain). These are completely flat, but bounce light in a way that's similar to how a 3D object would, making it look more complex than it really is. In the 3D world these work even better, but still have some limitations (since the surface would still be flat).
Vertex normals
While we do use the normals of the polygons for some other black magic related things, we don't actually control the smoothing of our model surface using the polygon normals. We use the vertex normals to control the smoothing of our normals. This is basically the same idea, but a little bit more complex.
Each vertex can have one or more normals associated. If it has a single normal, we call it an averaged vertex normal and if it has more than one, we call it a split vertex normal.
Let's take two polygons connected by an edge. If the transition between the two faces is smooth (we set it to smooth in Maya/Modo, or both have the same smoothing group in Max/Blender), each vertex has a single normal, which is the average of the polygon normals (this is why its called averaged vertex normal).Important note: up until very recently, each 3D program used its own method of calculating averaged vertex normals, which meant that normal maps calculated on one program may look completely different in another 3D program. I explain more about this in the second part of this tutorial.
If the transition is hard (hard edge, or the smoothing groups are different), each vertex has several normals: one for each connected vertex, and aligned with their normals. This leaves a gap in the normals, which looks as 2 different surfaces. This is what we call a split vertex normal.
As you can probably guess, controlling the vertex normals is vital if we want to control our normal maps. Fortunately, we don't really need to directly modify the normals or even see them, but knowing how this works will help us understand why we do things the way we do and understand more about the problems we may see.
Baking normal maps
When baking a normal map, we are basically telling the baking program to modify the direction that the low poly normals follow so that the match the direction of a highpoly model; so the lowpoly model is bouncing light as the highpoly would. All this information is stored on a texture called normal map. Let's see an example.
Let's say that we have a lowpoly like this. A flat plane with 4 vertices and a UV set, that our baking program will use to create the normal map.
And it has to recieve the normal information from this highpoly, whose normals are more complex.
Keep in mind that we are only transfering normal information, so the UVs, material, topology, transformations, etc... are completely irrelevant. Rule of thumb: if your highpoly looks good, it means that its normals are good and should be fine enough for baking.
Our baking program will take the lowpoly and cast rays following the normal directions of the lowpoly (this is why we need to control the lowpoly normals). Those rays have a limited lenght, to avoid getting normal information from far away faces (usually named bake distance or cage distance). When those rays collide with the highpoly, the baker calculates how to bend those rays so they follow the same normal direction as the highpoly, and stores that information into a normal map.
Here's the resulting bake from this example:
We have a texture that our engine uses to modify the lowpoly normals, so light bounces on this lowpoly model as it would if it was the highpoly version. Keep in mind that this is a texture, and can't affect the silouette of your lowpoly (you can't modify how light bounces from your model if light doesn't hit your model).
Normals can be stored as textures
While its certainly possible to "read" how our lightpoly was from looking at the normal map, its obvious that normal maps are not regular textures. This is because they don't carry color information, but normal information. This also means that normal maps should not be treated as regular textures and they have special compression and gamma correction settings, as we will see.
You can think of a normal map as a set of 3 greyscale textures, stored on a single image:
- The first image is telling the engine how this model should bounce light when lit from the right side, and its stored on the red channel of the normal map texture.
- The second image is telling the engine how the model should bounce light when lit from below*, and its stored on the green channel of the normal map texture.*Some programs use above instead of below, so we can have "left-handed" and "right handed" normal maps, and this can cause some problems as we will see later.
- The third image is telling the engine how the model should bounce light when lit from the front, and is stored on the blue channel of the normal map texture. Since most things look white when lit from the front, normal maps usually look blueish.
When we combine al three images on single one, we have a normal map. Please keep in mind that this explanation is not 100% accurate, but it will hopefully help you understand the information inside a normal map and have a better understanding of what it does.
In conclusion
Normals are vectors that we use to define how light bounces from a surface. They can be used to control the transition between faces (by averaging the normals of connected vertices to make a smooth transition, or splitting them to make a hard transition), but they can also be reoriented, to make a lowpoly model bounce light as a much more complex model would.This information is stored on 3 separate channels of an image, and the 3D program reads it in order to understand which direction the surface of the model should look towards.
Now that we have a general idea of what normals are, and how a normal map works, let's talk about how we can bake these details from the highpoly into the lowpoly.