UE 内置的 Release/Patch 分析

UE 的 Patch 有一些缺点:

  • 无法精确地控制 Patch 包含的内容且不方便拆分。
  • 无法进行迭代 Patch,不能直观预览资源信息。
  • 无法方便地管理工程和 Patch 版本。
  • 开发阶段无法方便地进行测试补丁打包。

具体操作是在 Project Launcher 中进行 Release 打包:

Cook 时的命令:

1
UE4Editor-Cmd.exe "D:\ThirdPerson425\ThirdPerson425.uproject" -run=Cook  -TargetPlatform=WindowsNoEditor -fileopenlog -unversioned -createreleaseversion=1.0.0.0 -compressed -abslog="C:\Program Files\Epic Games\UE_4.25\Engine\Programs\AutomationTool\Saved\Cook-2021.03.19-19.00.30.txt" -stdout -CrashForUAT -unattended -NoLogTimes  -UTF8Output

当打包成功后会在项目的根目录中创建出一个 Releases 目录,存储以下文件:

1
2
3
4
5
6
7
8
9
D:\ThirdPerson425\Releases>tree /a /f
D:.
\---1.0.0.0
\---WindowsNoEditor
| AssetRegistry.bin
| ThirdPerson425-WindowsNoEditor.pak
|
\---Metadata
DevelopmentAssetRegistry.bin

可以看到它的实现实际上是备份了当前打包版本的 AssetRegistry 文件,但经过分析发现,它并不会用来和后续的版本做比对,使用的是另一种方式。

在基于某个 Release 进行 Patch 时会自动把 AssetRegistry.bin 和 ushaderbytecode 包含到 pak 中(并且 shaderbytecode 并没有进行 patch):

当基于某个基础版本进行 Patch 的时候,在 Project Launher 的设置如下:

执行流程中首先需要对项目进行 Cook,但是 Cook 时不需要指定任何版本信息:

1
2
3
4
5
6
7
8
9
10
11
12
UE4Editor-Cmd.exe
"D:\Project\ThirdPerson425.uproject"
-run=Cook
-TargetPlatform=WindowsNoEditor
-fileopenlog
-unversioned
-abslog="C:\Program Files\Epic Games\UE_4.25\Engine\Programs\AutomationTool\Saved\Cook-2021.03.22-15.29.04.txt"
-stdout
-CrashForUAT
-unattended
-NoLogTimes
-UTF8Output

当 Cook 完毕,执行 Pak 打包的时候,需要传入版本信息:

1
2
3
4
5
6
7
8
9
10
11
12
UnrealPak.exe
"D:\Projects\ThirdPerson425.uproject"
"D:\Projects\Package\1.0.0.0_patch1\WindowsNoEditor\ThirdPerson425\Content\Paks\ThirdPerson425-WindowsNoEditor_0_P.pak"
-create="C:\Users\lipengzha\AppData\Roaming\Unreal Engine\AutomationTool\Logs\C+Program+Files+Epic+Games+UE_4.25\PakList_ThirdPerson425-WindowsNoEditor_0_P.txt"
-cryptokeys="D:\Projects\Saved\Cooked\WindowsNoEditor\ThirdPerson425\Metadata\Crypto.json"
-order="D:\Projects\Build\WindowsNoEditor\FileOpenOrder\CookerOpenOrder.log"
-generatepatch="D:\Projects\Releases\1.0.0.0\WindowsNoEditor\ThirdPerson425-WindowsNoEditor*.pak"
-tempfiles="C:\Program Files\Epic Games\UE_4.25\TempFilesThirdPerson425-WindowsNoEditor_0_P"
-patchpaddingalign=2048
-platform=Windows
-multiprocess
-abslog="C:\Users\lipengzha\AppData\Roaming\Unreal Engine\AutomationTool\Logs\C+Program+Files+Epic+Games+UE_4.25\UnrealPak-ThirdPerson425-WindowsNoEditor_0_P-2021.03.22-15.29.17.txt"

注意:

  1. -create=传递进去的 Response 的 txt 中是包含整个项目的资源列表的,并非是差异的文件列表。(-create= 也可以传递 pak 文件,用来生成差异)
  2. 需要通过 -generatepatch= 传递基础包中的 pak 文件
  3. 读取基础版本包 pak 中所有文件,计算出每个文件的 hash 值
  4. 与 Response 中的资源进行比对,得到新加或者与基础包 pak 中 Hash 文件不同的文件列表
  5. 把差异部分打包至新的 pak 中。

执行时会加载基础包中 pak 的文件,计算 pak 中资源的 hash 值:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
LogPakFile: Display: Parsing crypto keys from a crypto key cache file
LogPakFile: Display: Loading response file C:\Users\lipengzha\AppData\Roaming\Unreal Engine\AutomationTool\Logs\C+Program+Files+Epic+Games+UE_4.25\PakList_ThirdPerson425-WindowsNoEditor_0_P.txt
LogPakFile: Display: Added 1559 entries to add to pak file.
LogPakFile: Display: Loading pak order file C:\Users\lipengzha\Documents\Unreal Projects\ThirdPerson425\Build\WindowsNoEditor\FileOpenOrder\CookerOpenOrder.log...
LogPakFile: Display: Finished loading pak order file C:\Users\lipengzha\Documents\Unreal Projects\ThirdPerson425\Build\WindowsNoEditor\FileOpenOrder\CookerOpenOrder.log.
LogPakFile: Display: Generating patch from C:\Users\lipengzha\Documents\Unreal Projects\ThirdPerson425\Releases\1.0.0.0\WindowsNoEditor\ThirdPerson425-WindowsNoEditor*.pak.
LogPakFile: Display: Generated hash for "Engine/Content/Animation/DefaultAnimBoneCompressionSettings.uasset"
LogPakFile: Display: Generated hash for "Engine/Content/Animation/DefaultAnimCurveCompressionSettings.uasset"
LogPakFile: Display: Generated hash for "Engine/Content/Animation/DefaultAnimBoneCompressionSettings.uexp"
LogPakFile: Display: Generated hash for "Engine/Content/Animation/DefaultAnimCurveCompressionSettings.uexp"
LogPakFile: Display: Generated hash for "Engine/GlobalShaderCache-PCD3D_SM5.bin"
LogPakFile: Display: Generated hash for "Engine/Content/Functions/Engine_MaterialFunctions01/Opacity/CameraDepthFade.uasset"
LogPakFile: Display: Generated hash for "Engine/Content/Functions/Engine_MaterialFunctions01/Opacity/CameraDepthFade.uexp"
LogPakFile: Display: Generated hash for "Engine/Content/EngineMaterials/T_Default_Material_Grid_M.uasset"
LogPakFile: Display: Generated hash for "Engine/Content/EngineMaterials/T_Default_Material_Grid_N.uasset"
LogPakFile: Display: Generated hash for "Engine/Content/EngineMaterials/WorldGridMaterial.uasset"
LogPakFile: Display: Generated hash for "Engine/Content/EngineMaterials/DefaultMaterial.uasset"

生成 hash 的函数在 GenerateHashesFromPak 中,位于 PakFileUtilities/Private/PakFileUtilities.cpp 中。
其作用是:读取基础包中的文件,计算 hash 值,用作与新 Cook 之后的文件进行比对。

核心的代码在以下部分:

PakFileUtilities/Private/PakFileUtilities.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
bool ExecuteUnrealPak(const TCHAR* CmdLine)
{
// ...

// List of all items to add to pak file
TArray<FPakInputPair> Entries;
FPakCommandLineParameters CmdLineParameters;
ProcessCommandLine(CmdLine, NonOptionArguments, Entries, CmdLineParameters);

// ...

if(NonOptionArguments.Num() > 0)
{
CheckAndReallocThreadPool();

// since this is for creation, we pass true to make it not look in LaunchDir
FString PakFilename = GetPakPath(*NonOptionArguments[0], true);

// List of all items to add to pak file
TArray<FPakInputPair> Entries;
FPakCommandLineParameters CmdLineParameters;
ProcessCommandLine(CmdLine, NonOptionArguments, Entries, CmdLineParameters);

FPakOrderMap OrderMap;
FString ResponseFile;
if (FParse::Value(CmdLine, TEXT("-order="), ResponseFile) && !OrderMap.ProcessOrderFile(*ResponseFile))
{
return false;
}

FString SecondaryResponseFile;
if (FParse::Value(CmdLine, TEXT("-secondaryOrder="), SecondaryResponseFile) && !OrderMap.ProcessOrderFile(*SecondaryResponseFile, true))
{
return false;
}

int32 LowestSourcePakVersion = 0;
TMap<FString, FFileInfo> SourceFileHashes;

if (CmdLineParameters.GeneratePatch)
{
FString OutputPath;
if (!FParse::Value(CmdLine, TEXT("TempFiles="), OutputPath))
{
OutputPath = FPaths::GetPath(PakFilename) / FString(TEXT("TempFiles"));
}

IFileManager::Get().DeleteDirectory(*OutputPath);

// Check command line for the "patchcryptokeys" param, which will tell us where to look for the encryption keys that
// we need to access the patch reference data
FString PatchReferenceCryptoKeysFilename;
FKeyChain PatchKeyChain;

if (FParse::Value(FCommandLine::Get(), TEXT("PatchCryptoKeys="), PatchReferenceCryptoKeysFilename))
{
LoadKeyChainFromFile(PatchReferenceCryptoKeysFilename, PatchKeyChain);
ApplyEncryptionKeys(PatchKeyChain);
}

UE_LOG(LogPakFile, Display, TEXT("Generating patch from %s."), *CmdLineParameters.SourcePatchPakFilename, true );

if (!GenerateHashesFromPak(*CmdLineParameters.SourcePatchPakFilename, *PakFilename, SourceFileHashes, true, PatchKeyChain, /*Out*/LowestSourcePakVersion))
{
if (ExtractFilesFromPak(*CmdLineParameters.SourcePatchPakFilename, SourceFileHashes, *OutputPath, true, PatchKeyChain, nullptr) == false)
{
UE_LOG(LogPakFile, Warning, TEXT("Unable to extract files from source pak file for patch"));
}
else
{
CmdLineParameters.SourcePatchDiffDirectory = OutputPath;
}
}

ApplyEncryptionKeys(KeyChain);
}


// Start collecting files
TArray<FPakInputPair> FilesToAdd;
CollectFilesToAdd(FilesToAdd, Entries, OrderMap, CmdLineParameters);

if (CmdLineParameters.GeneratePatch)
{
// We need to get a list of files that were in the previous patch('s) Pak, but NOT in FilesToAdd
TArray<FPakInputPair> DeleteRecords = GetNewDeleteRecords(FilesToAdd, SourceFileHashes);

//if the patch is built using old source pak files, we need to handle the special case where a file has been moved between chunks but no delete record was created (this would cause a rogue delete record to be created in the latest pak), and also a case where the file was moved between chunks and back again without being changed (this would cause the file to not be included in this chunk because the file would be considered unchanged)
if (LowestSourcePakVersion < FPakInfo::PakFile_Version_DeleteRecords)
{
int32 CurrentPatchChunkIndex = GetPakChunkIndexFromFilename(PakFilename);

UE_LOG(LogPakFile, Display, TEXT("Some patch source paks were generated with an earlier version of UnrealPak that didn't support delete records. checking for historic assets that have moved between chunks to avoid creating invalid delete records"));
FString SourcePakFolder = FPaths::GetPath(CmdLineParameters.SourcePatchPakFilename);

//remove invalid items from DeleteRecords and set 'bForceInclude' on some SourceFileHashes
ProcessLegacyFileMoves(DeleteRecords, SourceFileHashes, SourcePakFolder, FilesToAdd, CurrentPatchChunkIndex);
}
FilesToAdd.Append(DeleteRecords);

// if we are generating a patch here we remove files which are already shipped...
RemoveIdenticalFiles(FilesToAdd, CmdLineParameters.SourcePatchDiffDirectory, SourceFileHashes, CmdLineParameters.SeekOptParams, CmdLineParameters.ChangedFilesOutputFilename);
}


bool bResult = CreatePakFile(*PakFilename, FilesToAdd, CmdLineParameters, KeyChain);

if (CmdLineParameters.GeneratePatch)
{
FString OutputPath = FPaths::GetPath(PakFilename) / FString(TEXT("TempFiles"));
// delete the temporary directory
IFileManager::Get().DeleteDirectory(*OutputPath, false, true);
}

GetDerivedDataCacheRef().WaitForQuiescence(true);

return bResult;
}

}

通过分析 UE 的 Patch 机制可以知道,它的版本比对比较粗暴,是直接得到 Pak 中文件的二进制信息计算 Hash 值与当前工程中 Cook 之后的 HASH 值来进行比对的,本质上就是基于二进制的比对,这就需要管理好 DDC 等 COOK 之后生成的文件,我觉得这样是不合理的,所以 HotPatcher 是基于原始资源的 GUID 的比对。