{"id":29,"date":"2017-03-07T15:28:41","date_gmt":"2017-03-07T20:28:41","guid":{"rendered":"http:\/\/knownshippable.com\/blog\/?p=29"},"modified":"2022-12-05T11:41:36","modified_gmt":"2022-12-05T16:41:36","slug":"fastbuild-caching-setup","status":"publish","type":"post","link":"http:\/\/knownshippable.com\/blog\/2017\/03\/07\/fastbuild-caching-setup\/","title":{"rendered":"FASTBuild Caching setup"},"content":{"rendered":"<h3>How it works<\/h3>\n<p>Caching is\u00a0a key feature in FASTBuild that allows machines to share the result of previous builds with each other avoiding unnecessary recompilation of the same code and resulting in a drastic total build time reduction. If you are familiar with ccache&#8230; this may sound like old news!<\/p>\n<p>In a typical scenario we\u00a0can imagine a central build machine that continuously builds a project and writes the result changed object files into the shared cache. Later when the team members\u00a0need to re-compile that same version\u00a0of the source\u00a0files, instead of recompiling them FASTBuild will detect the cache-hits and transfer the cached results which can be is significantly faster.<\/p>\n<p>Here is an example of build times both with and without the FASTBuild caching feature from the FASTBuild website:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-98 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2017\/03\/home_timings.png\" alt=\"\" width=\"900\" height=\"250\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2017\/03\/home_timings.png 900w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2017\/03\/home_timings-300x83.png 300w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2017\/03\/home_timings-768x213.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>Note that the build caching is not offered\u00a0on commercial products like Incredibuild, making it a unique feature to FASTBuild!<\/p>\n<h3>Caching setup<\/h3>\n<p>The setup is straightforward and consists of\u00a03 steps:<\/p>\n<p>1) Create a\u00a0shared cache folder.<\/p>\n<p>2) Set\u00a0the CachePath attribute\u00a0in the Unreal\u00a0<a href=\"https:\/\/github.com\/liamkf\/Unreal_FASTBuild\/blob\/master\/FASTBuild.cs\">FastBuild.cs<\/a>\u00a0to the network location selected\u00a0in step 1).<\/p>\n<p>3) Launch a new build and observe the cache folder being populated.<\/p>\n<p>Assuming we\u2019re running on Windows, here are a few more details about each step:<\/p>\n<h5>1. Create the\u00a0shared cache location:<\/h5>\n<p>a) Choose\u00a0a computer that is available on the local network, i.e. it is accessible by all computers participating in the build.<\/p>\n<p>b) Create a new folder named something like\u00a0<strong>FASTBuild<\/strong><strong>Shared<\/strong>. This folder will contain all FASTBuild shared files. Enable Windows sharing on that folder.<\/p>\n<p>c) Make sure to give Read\/Write permissions to all users\/machines that need to access the central cache.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-39 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-Sharing.png\" alt=\"FBuild-Cache-Sharing\" width=\"781\" height=\"722\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-Sharing.png 781w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-Sharing-300x277.png 300w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-Sharing-768x710.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p>d) Create a sub-folder called C<strong>ache<\/strong>, this will contain our FASTBuild cache structure.<\/p>\n<p>Tips and Troubleshooting:<\/p>\n<ul>\n<li>Distributing builds through WIFI can result into network saturation, slow transfers and timeouts which can negatively impact build times.<\/li>\n<li>The shared machine needs to\u00a0have\u00a0enough available storage space to host the cache. The size of the cache depends on multiple factors like: the size of the codebase, the number of files changed in each new\u00a0build that is pushed to the cache, the number of builds that are pushed (written to) the cache and the platform that we are building against. For instance a full\u00a0 Windows UnrealTournament build of a single configuration uses approximately 2GB.<\/li>\n<li>The cache machine should have good hard drive IO performance, or a very large amount of RAM, particularly if it will\u00a0serving many files to many computers simultaneously.<\/li>\n<li>If some computers are not using the cache as expected, be sure to test the access to the shared brokerage\u00a0folder from other computers: for example try to create and delete a folder to verify that you have full access rights.<\/li>\n<\/ul>\n<h5>2. Specify the cache location on the build\u00a0clients:<\/h5>\n<p>There are 2 ways of specifying the cache location:<\/p>\n<p>a) Local option: Through the Unreal <a href=\"https:\/\/github.com\/liamkf\/Unreal_FASTBuild\/blob\/master\/FASTBuild.cs\">FastBuild.cs<\/a> options:<\/p>\n<p>You can simply edit the <strong>CachePath<\/strong>\u00a0attribute\u00a0and specify the network location created in step 1).<\/p>\n<pre><pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\nprivate static string CachePath = \"\\\\\\\\DESKTOP-BEAST\\\\FastBuildShared\\\\Cache\"; \/\/ Optional: Location of the FASTBuild shared cache (local or network location).\n<\/pre>\n<p>b) Global option: Through system environment variables:<\/p>\n<p>Open the Windows Environment Variables Settings (System-&gt;Advanced system settings-&gt;Environment Variables) and add a new entry called\u00a0<strong>FASTBUILD_CACHE_PATH<\/strong><strong>.<\/strong> The value needs to be set to the network location created in step 1).<\/p>\n<p>In our example:\u00a0<strong>\\\\DESKTOP-BEAST\\FastBuildShared\\Cache<\/strong><\/p>\n<h5>3. Build with the FASTBuild cache enabled:<\/h5>\n<p>Now that we have created the central shared cache in Step 1 and configured\u00a0its location on every build client in Step 2, we can start a new build.<\/p>\n<p>If everything is OK you should see the cache structure being gradually populated with directories and files:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-47 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-directory.png\" alt=\"FBuild-Cache-directory\" width=\"663\" height=\"633\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-directory.png 663w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-directory-300x286.png 300w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p>After\u00a0the first build is completed, try to rebuild the same code once again and you should notice that the build client is able to retrieve build results from the cache. In the Visual Studio output window you should start seeing logs that look like:<\/p>\n<p>5&gt; \u00a04&gt; Obj: D:\\Dev-Projects\\Unreal4\\UnrealTournament\\Engine\\Intermediate\\Build\\Win64\\UE4\\Development\\MediaAssets\\Module.MediaAssets.cpp.obj &lt;<span style=\"color: #ff6600;\"><strong>CACHE<\/strong><\/span>&gt;<\/p>\n<p>The FASTBuild Summary is useful to inspect how the cache performed in the last build session:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-49 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-VSOutput.png\" alt=\"FBuild-Cache-VSOutput\" width=\"559\" height=\"406\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-VSOutput.png 559w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-VSOutput-300x218.png 300w\" sizes=\"auto, (max-width: 559px) 85vw, 559px\" \/><\/p>\n<p>Tips:<\/p>\n<ul>\n<li>The <em>CacheMode<\/em> setting is used to specify the behavior of the current FBuild.exe instance \u00a0in regards to the cache. There are 3 modes: <em>ReadOnly<\/em>, <em>WriteOnly<\/em> and <em>ReadWrite<\/em>. Depending on the role of each machine in your distributed build system you might want to set the\u00a0<em>CacheMode<\/em> accordingly. In a typical scenario the central build machine would run in a <em>ReadWrite<\/em>\u00a0mode while other build clients would run in <em>ReadOnly<\/em>.<\/li>\n<li>The cache mode can also be set through the\u00a0FASTBUILD_CACHE_PATH environment variable (refer to\u00a0the\u00a0<a href=\"http:\/\/www.fastbuild.org\/docs\/features\/caching.html\">FASTBuild documentation<\/a>\u00a0for more details).<\/li>\n<li>On the Windows platform (for different technical reasons that are explained in the <a href=\"http:\/\/www.fastbuild.org\/docs\/features\/caching.html\">FASTBuild documentation<\/a>), running in Write cache mode makes the build slower as its not able to use some build time optimizations (mainly because of the preprocessor cost and not being able to use PCHs). Consequently it is recommended that only a limited number of machines run in Write mode (i.e. central build machines) while all other clients (i.e. programmer&#8217;s machines) would run in Read-Only mode allowing them to use the latest cached files and a faster compilation of the non cached files.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>How it works Caching is\u00a0a key feature in FASTBuild that allows machines to share the result of previous builds with each other avoiding unnecessary recompilation of the same code and resulting in a drastic total build time reduction. If you are familiar with ccache&#8230; this may sound like old news! In a typical scenario we\u00a0can &hellip; <a href=\"http:\/\/knownshippable.com\/blog\/2017\/03\/07\/fastbuild-caching-setup\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;FASTBuild Caching setup&#8221;<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,3],"tags":[],"class_list":["post-29","post","type-post","status-publish","format-standard","hentry","category-fastbuild","category-unreal-engine"],"_links":{"self":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts\/29","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/comments?post=29"}],"version-history":[{"count":10,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts\/29\/revisions"}],"predecessor-version":[{"id":100,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts\/29\/revisions\/100"}],"wp:attachment":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/media?parent=29"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/categories?post=29"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/tags?post=29"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}