{"id":63,"date":"2017-03-07T15:10:30","date_gmt":"2017-03-07T20:10:30","guid":{"rendered":"http:\/\/knownshippable.com\/blog\/?p=63"},"modified":"2017-03-07T15:10:30","modified_gmt":"2017-03-07T20:10:30","slug":"fastbuild-distribution-setup","status":"publish","type":"post","link":"http:\/\/knownshippable.com\/blog\/2017\/03\/07\/fastbuild-distribution-setup\/","title":{"rendered":"FASTBuild Distribution setup"},"content":{"rendered":"<h3>How it works<\/h3>\n<p>FASTBuild was designed from the ground up to support efficient parallel compilation and build distribution. Its self-contained architecture and the absence of external dependencies insure an easy deployment across build machines; so basically you don&#8217;t need to install specific development tools on the build machines, FASTBuild takes care of sending the compiler toolchain to the build workers and run the remote compilation work in an isolated environment.<\/p>\n<p>Here is an example of build times using FASTBuild distribution from the FASTBuild website:<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/fastbuild.org\/docs\/timing_dist.png\" \/><\/p>\n<p>Tips:<\/p>\n<ul>\n<li>Only source code compilation can be distributed across machines at the moment.<\/li>\n<li>There are some restrictions as far as supported compiler settings and build distribution, you can refer to the FASTBuild offical <a href=\"http:\/\/fastbuild.org\/docs\/features\/distribution.html\">documentation <\/a>for more details.<\/li>\n<\/ul>\n<h3>Distribution setup<\/h3>\n<p>The setup is straightforward and consists of 4 steps:<\/p>\n<p>1) Creating a\u00a0shared brokerage folder: this is where build workers register themselves<\/p>\n<p>2) Setting the\u00a0FASTBUILD_BROKERAGE_PATH environment variable on all build machines, the path must point to the location created in the first step.<\/p>\n<p>3) Starting the <strong>FBuildWorker.exe<\/strong> on all build machines.<\/p>\n<p>4) Launching a distributed\u00a0build.<\/p>\n<p>Assuming we&#8217;re running on Windows, here are a few more details about each step:<\/p>\n<h5>1. Create the shared brokerage folder:<\/h5>\n<p>Workers are discovered through a brokerage location. The brokerage location is a network path accessible from both the host and workers.<\/p>\n<p>a) Choose\u00a0a computer that is available\u00a0through the local network, i.e. it is accessible by the other computers participating in the build. We had noticeably fewer problems when all machines were connected through a wired LAN, as wireless performance (and thus build times) tended to degrade rapidly for large builds.<\/p>\n<p>b) Create a new folder named something like\u00a0<strong>FASTBuild<\/strong><strong>Shared<\/strong>. This folder will contain all FASTBuild shared files, see the <a href=\"http:\/\/knownshippable.com\/blog\/2016\/08\/16\/fastbuild-caching-setup\/\">caching instructions<\/a> for another usage of this folder. Enable Windows sharing on this folder.<\/p>\n<p>c) Give Read\/Write permissions to all users and machines that need access the shared location.<\/p>\n<p>Tips\/Troubleshooting:<\/p>\n<ul>\n<li>As mentioned above, distributing builds through WIFI can result in network saturation, slow transfers and timeouts which can negatively impact build times<\/li>\n<li>If some computers aren&#8217;t contributing as expected, be sure to test the access to the shared brokerage\u00a0folder from other computers: for example try to create and delete a folder to verify that you have full access rights.<\/li>\n<\/ul>\n<h5>2. Setting the FASTBUILD_BROKERAGE_PATH environment variable:<\/h5>\n<p>Build hosts and build workers need to know about each other in order to communicate commands and build tasks.<\/p>\n<p>Every active build worker (machines running FBuildWorker.exe), writes a file to the brokerage folder created in Step 1. Upon starting a build, the build host (running Fbuild.exe) scans the files inside the brokerage folder and registers the associated machines\u00a0as build workers it can use to distribute jobs.<\/p>\n<p>FBuild.exe and FBuildWorker.exe processes know about the brokerage location via the\u00a0FASTBUILD_BROKERAGE_PATH environment variable which must be set on all machines participating in the build distribution, The value must be the path created in step 1.<\/p>\n<p>In our example it looks something like:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-40 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-EnvVars.png\" alt=\"FBuild-Cache-EnvVars\" width=\"619\" height=\"589\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-EnvVars.png 619w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-Cache-EnvVars-300x285.png 300w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h5>3. Starting FBuildWorker.exe on all build machines:<\/h5>\n<p>Start FBuildWorker.exe on all machines participating in the build distribution.<\/p>\n<p>You should see\u00a0worker UI that offers a couple of customization settings in order to manage its work load.<img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-68 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-starting-fbuilworker.png\" alt=\"FBuild-starting fbuilworker\" width=\"686\" height=\"294\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-starting-fbuilworker.png 686w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-starting-fbuilworker-300x129.png 300w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p>To verify that the build workers\u00a0have successfully written their file inside the brokerage folder you should be able to see their\u00a0corresponding machine name as files inside that location. If we take our previous example, the files can be found inside <strong><em>\\\\FastBuildShared\\main\\14<\/em><\/strong><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-69 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-brokerage-path.png\" alt=\"FBuild-brokerage-path\" width=\"640\" height=\"163\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-brokerage-path.png 640w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-brokerage-path-300x76.png 300w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h5>4. Launching a distributed build:<\/h5>\n<p>The last step before launching a distributed build is to make sure it is enabled inside the Unreal\u00a0<a href=\"https:\/\/github.com\/liamkf\/Unreal_FASTBuild\/blob\/master\/FASTBuild.cs\">FastBuild.cs<\/a>. Verify that the\u00a0<em>bEnableDistribution<\/em> attribute is set to\u00a0<em>true.<\/em><\/p>\n<p>Now start a new build and notice some important information logged in the build output window that shows that the build distribution is active:<img loading=\"lazy\" decoding=\"async\" class=\"wp-image-73 size-full aligncenter\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers2-1.png\" alt=\"FBuild-output workers2\" width=\"1119\" height=\"412\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers2-1.png 1119w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers2-1-300x110.png 300w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers2-1-768x283.png 768w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers2-1-1024x377.png 1024w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/>\u00a0<img loading=\"lazy\" decoding=\"async\" class=\"wp-image-72 size-full aligncenter\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers.png\" alt=\"FBuild-output workers\" width=\"1121\" height=\"387\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers.png 1121w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers-300x104.png 300w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers-768x265.png 768w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers-1024x354.png 1024w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>The build report should also tell us some important stats on how the build was distributed and how much time we saved in the process:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-78 size-full\" src=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers3-1.png\" alt=\"FBuild-output workers3\" width=\"558\" height=\"404\" srcset=\"http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers3-1.png 558w, http:\/\/knownshippable.com\/blog\/wp-content\/uploads\/2016\/08\/FBuild-output-workers3-1-300x217.png 300w\" sizes=\"auto, (max-width: 558px) 85vw, 558px\" \/><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>How it works FASTBuild was designed from the ground up to support efficient parallel compilation and build distribution. Its self-contained architecture and the absence of external dependencies insure an easy deployment across build machines; so basically you don&#8217;t need to install specific development tools on the build machines, FASTBuild takes care of sending the compiler &hellip; <a href=\"http:\/\/knownshippable.com\/blog\/2017\/03\/07\/fastbuild-distribution-setup\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;FASTBuild Distribution setup&#8221;<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-63","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts\/63","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/comments?post=63"}],"version-history":[{"count":10,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts\/63\/revisions"}],"predecessor-version":[{"id":89,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/posts\/63\/revisions\/89"}],"wp:attachment":[{"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/media?parent=63"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/categories?post=63"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/knownshippable.com\/blog\/wp-json\/wp\/v2\/tags?post=63"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}