Setting up Perforce for Unreal Engine

Using a properly setup Perforce to handle Unreal Engine streams can be a boon to productivity. This article explores how to set this up using Perforce best practices and what I've learned over the years. Please email me any questions and/or clarifications that you require.

The example used here is UE4, but the same principles apply to UE5. The only difference is trimming the footprints at the bottom of this article.

Setting up the Depot

The best process I've found here is to go two deep when creating the depot. This allows categorization of the streams and prevents an explosion of streams in the 'root' folder.

From Helix Admin:


The key part to note is 'Stream Depth' needs to be at least 2. For the example given of UE4/1/2
  • UE4 is the name of the depot
  • The category of a branch. e.g. 'releases', 'main', 'tasks'
  • The name of a specific branch. e.g. 'Raleigh'

  • Whenever you create a new branch, add it under the correct category. To my knowledge you cannot change the category of an existing branch after it has been created. The only way to move it is to obliterate and then recreate.
    After some months of work and general development, you should end up with something like the following:

    You'll note that the 'Task-Linux-GL' branch was created in the wrong location.

    Tips for branch creation:
  • Always double check you've created it in the correct location.
  • Have more generic names for branches, such as office location. This means branches can be reused to reduce clutter in the Perforce UI and save on sync times (most of the content won't change).
  • Once a branch has been created, it will never go away until a new depot is created.
  • Adding UE4 to Perforce

    The default install starts off hidden in the bowels of Windows and the projects are created in the 'My Documents' folder. This does not facilitate storing in Perforce or a reasonable branching strategy. The only way this is feasible is if you never make any engine code changes and only make content.

    Set up a decent typemap. Typemap (GitHub link)

    Make a new workspace (using your own conventions) and put the workspace root at the root of one of your drives. My convention is to use the machine name and branch name with the root folder being named the same. e.g. the MACHINE-Main workspace resides in d:\MACHINE-main.

    Copy the contents of the UE4/Engine folder and the files in the root of the UE4 folder to your workspace root after setup.bat has been run, but before any compilation has been done. Mark these files for add and submit them. Generate project files, compile and run the engine. Generate a new project for your content and store it in the root of your workspace. I'd also recommend adding a TestBed project for experimentation and throwaway content. After all this, you should end up with something like:

      
      	d:\MACHINE-Main
      	d:\MACHINE-Main\Engine\
      	d:\MACHINE-Main\TestBed\
      	d:\MACHINE-Main\CoolNewGame\
      	d:\MACHINE-Main\GenerateProjectFiles.bat
      	d:\MACHINE-Main\UE4Games.uprojectdirs
      

  • Make sure you don't have any Intermediate folders checked in. This can't be done as part of the stream definition as that does not allow subfolders to be ignored.
  • I don't copy the FeaturePacks or Templates that come with the engine. I'd recommend that you create your project from the appropriate template, but don't have those files source controlled.
  • Try not to use Engine content directly. If you do end up using (say) Engine/Content/BasicShapes/Cube, make a copy in your own project and use that. This prevents any accidental editing of the asset and makes engine upgrades a little less painful.
  • Workflow

    Perforce best practices state that you should merge down and copy up. There is plenty of literature online about this so I won't delve too deep here.

    Introductory Video

    Given:


  • Most (if not all) development work should be done in a branch.
  • Every day (or more frequently), merge down from main (the green arrow) and resolve any differences. The more often this is done (the more iteratively), the less problems there will be.
  • Once a feature is finished, copy up from the branch to main (the red arrow). As the merging down has been done frequently, the branch and main should be identical.
  • When QA and management sign off on a release, copy the main (a.k.a. Trunk) to a release branch. This is where any hot-fixes to the releases should be performed.
  • Trimming the Perforce footprint (UE4.27.2)

    UE4 comes with all the code for all editors and all platforms (except the targets that require a special license). I have no intention of targeting certain platforms or using a Mac editor, so I trim these source files out. This cuts down the the number of files in the depot by about a third, speeds up editor loading time, and ensures no-one goes down any blind alleys. Not everyone has an Epic machine, so every little bit of performance helps. The downside is that engine changes are not tested on platforms you've disabled, but if you're never going to target that platform, who cares?

    This method uses Perforce Virtual Streams to mask out the unwanted files gracefully. The UnrealBuildTool project file is changed significantly, but otherwise there is only an accessor added to the code. Several build cs files also had to be updated to make the changes forward and backward compatible.

    Setup virtual streams

    Virtual streams allows restriction of the workspace view i.e. allows masking out of certain files. Virtual Streams. To this end, they can be set up to mask out platform specific files.

    Sample virtual stream ignore set for HoloLens (GitHub link), Apple (GitHub link), and Android (GitHub link)

    Implement code changes

    All changes are wrapped in the comments // ETERNAL BEGIN and // ETERNAL END

    Add the following five files to //Engine/Source/Programs/UnrealBuildTool/Platform

  • StubAndroid.cs (GitHub link)
  • StubHoloLens.cs (GitHub link)
  • StubIOS.cs (GitHub link)
  • StubLumin.cs (GitHub link)
  • StubMac.cs (GitHub link)
  • Update the UnreaBuildTool.csproj file
  • UnrealBuildTool.csproj (GitHub link)
  • Add the accessor to UEBuildPlatform.cs with this change
  • UEBuildPlatform.cs.add (GitHub link)
  • Call this accessor in the various build.cs files as required. You may find these changes quicker to implement by yourself, but here they are anyway.
  • AllDesktopTargetPlatform.Build.cs (GitHub link)
  • CrashDebugHelper.Build.cs (GitHub link)
  • GoogleVRController.Build.cs (GitHub link)
  • TargetPlatform.Build.cs (GitHub link)
  • Slate.Build.cs (GitHub link)
  • UnrealEd.Build.cs (GitHub link)
  • The NoApple and NoHoloLens branches will generate project files, compile, and run without issue.

    The NoAndroid branch requires the following C++ changes to generate project files, compile, and run. The changes purely comment out Android specific code.

  • NoAndroid.patch (GitHub link)
  • The end result of masking out these five platforms in reducing the engine folder from about 36GB to about 19GB.

    File size File size delta File count File count delta
    Vanilla 36,331,715,884 0 178,257 0
    No Apple 24,980,271,332 11.3GB 168,625 9,632
    No Android 32,100,783,594 4.2GB 174,034 4,223
    No HoloLens 35,132,865,156 1.2GB 177,490 767

    Trimming the Perforce footprint (UE5.2.1)

    UE5 has made some noticeable improvements to UBT:

  • It uses Net6
  • It uses nullable types
  • It populates the project file based on the files it finds - no more manual editing of the project file!
  • HoloLens support is automatic
  • Lumin (whatever that was) no longer exists
  • Setup virtual stream

    Virtual streams allows restriction of the workspace view i.e. allows masking out of certain files. Virtual Streams. To this end, they can be set up to mask out platform specific files.

    Sample virtual stream ignore set for Mac, iOS, Android, HoloLens, TVOS (GitHub link)

    Implement code changes

    All changes are wrapped in the comments // ETERNAL BEGIN and // ETERNAL END

    Add the following three files to //Engine/Source/Programs/UnrealBuildTool/Platform (Lumin is deprecated and HoloLens is automatic)

  • StubAndroid.cs (GitHub link)
  • StubIOS.cs (GitHub link)
  • StubMac.cs (GitHub link)
  • Update the following 8 files to get UnrealBuildTool to compile (this is less than ideal)

  • UEBuildFramework.cs (GitHub link)
  • UEBuildPlatform.cs (GitHub link)
  • UEBuildTarget.cs (GitHub link)
  • UEBuildLinux.cs (GitHub link)
  • RiderProjectFile.cs (GitHub link)
  • VSCodeProjectFileGenerator.cs (GitHub link)
  • AppleToolChain.cs (GitHub link)
  • RemoteMac.cs (GitHub link)
  • Fix the engine compile issues with the following 4 files

  • LocationServicesAndroidEditor.Build.cs (GitHub link)
  • CookedEditor.Build.cs (GitHub link)
  • CookedEditorPackageManager.h (GitHub link)
  • PIEPreviewDevice.cpp (GitHub link)
  • The engine should now compile fine.

    To get the tools to compile

  • Make sure the typemap has been set so the relevant binaries are binary+w
  • Remove '+ProgramEnabledPlugins="XcodeSourceCodeAccess"' from the DefaultEngine.ini files of UnrealFrontEnd and UnrealInsights
  • Apply this fix if necessary Unsync Compile Fix
  • Apply this fix for CrashDebugHelperCrashDebugHelper.Build.cs (GitHub link)
  • The end result of masking out these five platforms in reducing the engine folder from about 57GB to about 44GB.

    File size File size delta File count File count delta
    Vanilla 56.8GB 0 255,734 0
    Trimmed 43.5GB 13.3GB 232,287 23,447

    There seems to be some effort to automatically populate the UnrealTargetPlatform enum in UBT with discovered platforms. This would be awesome! For example, if UnrealTargetPlatform.HoloLens existed in UBT, it could be checked in the various Build.cs files to include only the relevant files and reduce the number of include folders that needed to be searched. It could also be used to generate a preprocessor define (e.g. UE_BUILD_TARGET_HOLOLENS) that could be used to add HoloLens specific UI only if that target was available. Eventually, Unreal Engine could have an installer that allows you to only install the components you wish (a bit like the Visual Studio installer). Which editors do you want? (Windows, Mac, Linux) and which targets do you want? (Windows, Xbox8, Mac, iOS, Android, PS9, etc.)

    In an ideal world

  • Better platform naming conventions e.g. the Mac masking can grab 'macros.h'. Each platform should be distinct and unique.
  • Put platform specific third party libraries and resources under folders named after the platform.
  • Future work

  • Make a NoLinux branch
  • Improve the masking to mask out more files - any tips here would be greatly appreciated!