Category: FASTBuild

FASTBuild Caching setup

How it works

Caching is a key feature in FASTBuild that allows machines to share the result of previous builds with each other avoiding unnecessary recompilation of the same code and resulting in a drastic total build time reduction. If you are familiar with ccache… this may sound like old news!

In a typical scenario we can imagine a central build machine that continuously builds a project and writes the result changed object files into the shared cache. Later when the team members need to re-compile that same version of the source files, instead of recompiling them FASTBuild will detect the cache-hits and transfer the cached results which can be is significantly faster.

Here is an example of build times both with and without the FASTBuild caching feature from the FASTBuild website:

Note that the build caching is not offered on commercial products like Incredibuild, making it a unique feature to FASTBuild!

Caching setup

The setup is straightforward and consists of 3 steps:

1) Create a shared cache folder.

2) Set the CachePath attribute in the Unreal FastBuild.cs to the network location selected in step 1).

3) Launch a new build and observe the cache folder being populated.

Assuming we’re running on Windows, here are a few more details about each step:

1. Create the shared cache location:

a) Choose a computer that is available on the local network, i.e. it is accessible by all computers participating in the build.

b) Create a new folder named something like FASTBuildShared. This folder will contain all FASTBuild shared files. Enable Windows sharing on that folder.

c) Make sure to give Read/Write permissions to all users/machines that need to access the central cache.

FBuild-Cache-Sharing

d) Create a sub-folder called Cache, this will contain our FASTBuild cache structure.

Tips and Troubleshooting:

  • Distributing builds through WIFI can result into network saturation, slow transfers and timeouts which can negatively impact build times.
  • The shared machine needs to have enough available storage space to host the cache. The size of the cache depends on multiple factors like: the size of the codebase, the number of files changed in each new build that is pushed to the cache, the number of builds that are pushed (written to) the cache and the platform that we are building against. For instance a full  Windows UnrealTournament build of a single configuration uses approximately 2GB.
  • The cache machine should have good hard drive IO performance, or a very large amount of RAM, particularly if it will serving many files to many computers simultaneously.
  • If some computers are not using the cache as expected, be sure to test the access to the shared brokerage folder from other computers: for example try to create and delete a folder to verify that you have full access rights.
2. Specify the cache location on the build clients:

There are 2 ways of specifying the cache location:

a) Local option: Through the Unreal FastBuild.cs options:

You can simply edit the CachePath attribute and specify the network location created in step 1).

private static string CachePath = "\\\\DESKTOP-BEAST\\FastBuildShared\\Cache"; // Optional: Location of the FASTBuild shared cache (local or network location).



b) Global option: Through system environment variables:


Open the Windows Environment Variables Settings (System->Advanced system settings->Environment Variables) and add a new entry called FASTBUILD_CACHE_PATH. The value needs to be set to the network location created in step 1).


In our example: \\DESKTOP-BEAST\FastBuildShared\Cache


3. Build with the FASTBuild cache enabled:


Now that we have created the central shared cache in Step 1 and configured its location on every build client in Step 2, we can start a new build.


If everything is OK you should see the cache structure being gradually populated with directories and files:


FBuild-Cache-directory


After the first build is completed, try to rebuild the same code once again and you should notice that the build client is able to retrieve build results from the cache. In the Visual Studio output window you should start seeing logs that look like:


5>  4> Obj: D:\Dev-Projects\Unreal4\UnrealTournament\Engine\Intermediate\Build\Win64\UE4\Development\MediaAssets\Module.MediaAssets.cpp.obj <CACHE>


The FASTBuild Summary is useful to inspect how the cache performed in the last build session:


FBuild-Cache-VSOutput


Tips:


    

  • The CacheMode setting is used to specify the behavior of the current FBuild.exe instance  in regards to the cache. There are 3 modes: ReadOnly, WriteOnly and ReadWrite. Depending on the role of each machine in your distributed build system you might want to set the CacheMode accordingly. In a typical scenario the central build machine would run in a ReadWrite mode while other build clients would run in ReadOnly.
    • 

  • The cache mode can also be set through the FASTBUILD_CACHE_PATH environment variable (refer to the FASTBuild documentation for more details).
    • 

  • On the Windows platform (for different technical reasons that are explained in the FASTBuild documentation), running in Write cache mode makes the build slower as its not able to use some build time optimizations (mainly because of the preprocessor cost and not being able to use PCHs). Consequently it is recommended that only a limited number of machines run in Write mode (i.e. central build machines) while all other clients (i.e. programmer’s machines) would run in Read-Only mode allowing them to use the latest cached files and a faster compilation of the non cached files.
    •

FASTBuild with Unreal Engine 4 setup

So you want to build Unreal Engine 4 with FASTBuild… Or maybe you don’t want to pay for Incredibuild but rebuilding Unreal in 45 minutes doesn’t seem promising. You are in the right spot. Here were our steps to building UE4 with FASTBuild on Windows.

  1. Download FASTBuild from here, the latest x64 Windows version most likely. Move the extracted folder to the Engine/Extras folder in Unreal.
  2. Download FASTBuild.cs from here and place under Engine/Source/Programs/UnrealBuildTool/System/. Either add it to the UnrealBuildTool project yourself or regenerate the projects.
  3. Modify the ActionGraph.cs file, specifically the ExecuteActions method, to call FASTBuild.ExecuteActions like so:

    For Unreal Engine before 4.15, it looked more like this:
    In both cases, you can of course also add a bAllowFastBuild bool to BuildConfiguration.cs to control who attempts to use FASTBuild.
  4. Rebuild UnrealBuildTool from Visual Studio.
  5. Build a smaller project, like say ShaderCompileWorker to test it out! It should use FASTBuild, but it won’t build across the network or use the cache without a little bit more setup.

Next you’ll want to set up caching and distribution, as that’s where you’ll actually see the benefits of using FASTBuild!

Using FASTBuild with Unreal Engine 4

Near the end of 2015 we were placed in the unpleasant situation of needing to compile two versions of Unreal Engine without the benefit of the ubiquitously speedy Incredibuild.

To make things worse we on platforms using the lightly poky Microsoft C++ compiler, meaning that changing a core header in Unreal was resulting in compile times of over half an hour.

It did not take many of these changes before we decided something had to be done. Being in Montreal we knew that some studios in the city were using the aptly named distributed build and caching solution FASTBuild. As the universe tends to do, as we were cleaning up our solution it appears someone else was also on the trail of making FASTBuild work with unreal! If we had a time machine… FASTBuild with Unreal would have saved us a bit of trouble.

The end goal for what we did as well as what Carl did was the same: produce a .bff file from the UnrealBuildTool actions.

The Unreal FASTBuild duo has some quirks worth mentioning.

Unreal enjoys its precompiled headers (PCHs), this works as expected in FASTBuild except that Unreal also sometimes links with the PCH object file, and the object file FASTBuild generates does not have the appropriate name. So we have a small modification to FASTBuild which allows us to override the PCH object file name.

Another quirk was that some of the obj files (occasionally in the hundreds of MB) generated are so big that the clients were timing out. Yassine did some profiling and somewhat surprisingly it looks like it’s happening at the Windows socket level with the send call telling us the buffer had no room for very small sends of 8 bytes, just after attempting some sends in the tens of megabytes. Bumping up the timeout from 5 seconds to over a minute got rid of all of these issues, but it was a bit of an annoyance.