uefi-sct/SctPkg: Memory Allocation Failure on EDK2 stable202411

[r1chard-lyu]

While executing the UEFI SCT test (sct.efi) on the edk2-stable202411, the system encounters a memory allocation failure, preventing the sct application from starting properly. This issue results in an assertion failure in Pool.c at line 721, interrupting the test execution.

FSOpen: Open '\X64\sct.EFI' Success
[Security] 3rd party image[0] can be loaded after EndOfDxe: PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)/HD(1,MBR,0xBE1AFDFA,0x3F,0xFBFC1)/\X64\sct.EFI.
None of Tcg2Protocol/CcMeasurementProtocol is installed.
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 61E1040
Loading driver at 0x000060AE000 EntryPoint=0x000060CB81F SCT.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 6219998
ProtectUefiImageCommon - 0x61E1040
  - 0x00000000060AE000 - 0x0000000000036A00
InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 7E7E428
AllocatePoolPages: failed to allocate 64765 pages
AllocatePool: failed to allocate 265275784 bytes
ASSERT Pool.c(721): Head->Signature == ((('p') | ('h' << 8)) | ((('d') | ('0' << 8)) << 16)) || Head->Signature == ((('p') | ('h' << 8)) | ((('d') | ('1' << 8)) << 16))    

[edhay]

@r1chard-lyu
Thank you for raising this issue.
The issue itself may not be related to edk2-stable202411 and may in fact be a driver issue.

The next steps to narrow down the cause is to check the code after the
InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 7E7E428

@anncheng-arm to coordinate with @r1chard-lyu and update.

[r1chard-lyu]

@edhay Thank you for your help!

I have checked, and this failed with the GUID for gEfiShellParametersProtocolGuid when installing the Shell Parameters Protocol.

➜  sct_workspace grep -r "752F3136-4E16-4FDC-A22A-E5F46812F4CA" .
./edk2/Build/OvmfX64/DEBUG_GCC5/FV/Guid.xref:752F3136-4E16-4FDC-A22A-E5F46812F4CA gEfiShellParametersProtocolGuid

@anncheng-arm Do you have any idea how to trace the cause of this?

[sunnywang-arm]

Hi @r1chard-lyu
This looks not like an issue with installing the Shell Parameters Protocol. I think Edhaya suggested you to check what caused two lines below being printed out.

• AllocatePoolPages: failed to allocate 64765 pages
• AllocatePool: failed to allocate 265275784 bytes

The ASSERT message means a problem is with a FreePool call. It generally happens when a function is trying to free up an invalid memory pool. Probably, the function doesn't consider the failed case of allocating a memory pool.

I think the easiest way to debug this issue is just like Edhaya mentioned. Revert the changes to find out the last workable commit and problematic commit. Otherwise, I think you'll need to add additional prints from InitializeSct() to get more information.

Cc @edhay @anncheng-arm

[r1chard-lyu]

Hi @sunnywang-arm,

Thank you for your suggestion. I will work on it.

[r1chard-lyu]

Hi @edhay,
I have tested the following commits with the build option SctPkg/build.sh X64 GCC DEBUG, following the instructions in HowToBuildSct.txt, but running SCT.efi still hangs without printing any information to the QEMU console. According to debug.log, it always stops at the same position: InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 7FE7E638.

[Failed] 4b59759 UEFI-SCT Case spec conversion to markdown
[Failed] df9a877 EDK2-Test BUG4244 V2 - Print PCIe device path for SCT failures
[Failed] 0e2ced3 (tag: edk2-test-stable202406) Updated SCT Case Spec
[Failed] abb54c6 (tag: edk2-test-rc2_202405) uefi-sct/SctPkg: TCG2 Protocol: clean up type conversion warnings

However, the binary from the community release edk2-test-stable202406 runs normally on my QEMU console, and debug.log also prints out debug information as expected. In contrast, even when I use the source code with the build option SctPkg/build.sh X64 GCC DEBUG, I encounter the same issue as before. Therefore, I suspect the issue may not lie in the source code itself, but rather in the build options, environment, or toolchain. My current build option is SctPkg/build.sh X64 GCC DEBUG, and my local build environment is openSUSE Leap 15.6, with GCC version 12.3.0. May I ask what build options were used for the official release?

In addition, I used GDB to inspect the runtime behavior and confirmed that the SCT.efi built locally is actually running. However, it does not print any output to the QEMU console, and it seems that the debug information is not being recorded in debug.log. Here is the QEMU command I used qemu-system-x86_64 -bios ./bios.bin -m 2048 -drive file=fat:hda-contents,format=raw,snapshot=on -net none -debugcon file:debug.log -global isa-debugcon.iobase=0x402 -s. So the issue might really not be in the code, but rather in the build process.

Cc @sunnywang-arm @anncheng-arm

[anncheng-arm]

@sunnywang-arm @edhay

I am not that familiar with UEFI built, but ack to @r1chard-lyu's updated, I encountered exactly the same issue as Richard's just like I mentioned in the mail.
So, knowing the build environment, build flow, tool chain version when we released the tag version maybe helpful for us to narrow down or move forward.

[edhay]

Hi @r1chard-lyu,
Thank you for the debug.

The steps that were followed to generate the release binaries are as below (On Windows)
Build Instructions for UEFI SCTII X64 (Windows)

1. mkdir sct_workspace
2. cd sct_workspace
3. git clone https://github.com/tianocore/edk2-test.git
   Make sure to match tag hash
4. git clone https://github.com/tianocore/edk2
   Make sure to match tag hash
5. open "Developer Command Prompt for VS2015", cd edk2
   Use this command first to update submodules in edk2 clone: git submodule update --init
   This fixed file not found errors
   Install Nasm.exe and set NASM_PREFIX in dev cmd prompt as nasm is needed to build
6. edksetup.bat Rebuild
7. xcopy ..\edk2-test\uefi-sct\SctPkg\Tools\Source\GenBin BaseTools\Source\C\GenBin\
8. cd BaseTools\Source\C\GenBin
9. nmake
10. cd ........\
11. xcopy ..\edk2-test\uefi-sct\SctPkg /E .\SctPkg\
12. build -p SctPkg\UEFI\UEFI_SCT.dsc -a X64 -t VS2015x86
13. cd Build\UefiSct\DEBUG_VS2015x86
14. ......\SctPkg\CommonGenFramework.bat uefi_sct X64 InstallX64.efi
    This step is needed at the end to create the file structure that QA expects

The target subdirectory named as SctPackageX64 which includes test cases and UEFI SCT II applications will be generated in sct_workspace\edk2\ Build\UefiSct\DEBUG_VS2015x86

For the Linux Steps, please check, notes of this tag
https://github.com/tianocore/edk2-test/releases/tag/edk2-test-stable201910

We will make these steps upstream, after the verification is done.

[r1chard-lyu]

I reviewed the Linux build process, and I found that it uses an outdated gcc-4.9. However, in the edk2 commit f53f029122, the -mstack-protector-guard=global flag was introduced. This flag is not supported by older GCC versions. Since SCT.efi depends on edk2 libraries, building it with an outdated compiler results in a failure. Therefore, the Linux build process is outdated and needs to be revised.

I also tried building the X64 version using gcc-12. The build completed successfully, and the binary runs, but the original issue still persists. After further investigation, I traced the issue to ./Library/SctLib/Print.c:1204, where a call to PFLUSH(ps) leads to SctAllocatePool, which eventually calls Status = tBS->AllocatePool(PoolAllocationType, Size, (VOID **)&p); At this point, the debug log outputs the following messages because page allocation failed.

AllocatePoolPages: failed to allocate 1047805 pages
AllocatePool: failed to allocate 4291808680 bytes

Since the returned Status indicates failure, the following if (EFI_ERROR(Status)) branch is taken, resulting in p = NULL.

Later on, when SctFreePool is called and tries to free the pointer via tBS->FreePool(Buffer);, the debug log shows:

ASSERT Pool.c(721): Head->Signature == ((('p') | ('h' << 8)) | ((('d') | ('0' << 8)) << 16)) || Head->Signature == ((('p') | ('h' << 8)) | ((('d') | ('1' << 8)) << 16))

Since no further output appears on the QEMU console and the program hangs at this point, I suspect that some component related to console output was not properly initialized.

This is the failure point I've identified so far. I'll continue investigating and look for a potential fix.

[sunnywang-arm]

@r1chard-lyu Do you have any updates on this?
For the build instruction issue, I raised another issue for tracking it. #270

Cc @edhay

[r1chard-lyu]

@r1chard-lyu Do you have any updates on this? For the build instruction issue, I raised another issue for tracking it. #270

Cc @edhay

Hi @sunnywang-arm,
Regarding the issue I was originally tracking, it might be related to the console output not being initialized properly. Also, I think this bug is not solely caused by the SCT, but also related to recent updates in EDK2. Therefore, I haven't been able to identify the last workable commit for both SCT and EDK2 yet. I will continue to investigate this issue.

Also, thank you for helping create another issue to track the build instruction problem.

[sunnywang-arm]

Thanks for the update, @r1chard-lyu.

If I understand your comment correctly, the issue is specific to qemu-system-x86_64.

Also, we haven't seen this issue on the Arm platforms using our SystemReady ACS SCT built from edk2-stable202411 and edk2-test-stable202406. https://github.com/ARM-software/arm-systemready/blob/main/common/config/systemready-band-source.cfg

Cc @edhay

[r1chard-lyu]

Hi @sunnywang-arm,
Yes, it only happens on x86_64. I tested building the aarch64 SCT with edk2-stable202411. It built successfully and ran well.

[gapandya]

Hi @r1chard-lyu ,

I am not sure whether it will be relevant to what you are trying to build.
But some time back i was trying to build latest edk2 + latest Sct, and i was seeing run time ASSERT/HANG. Then i switch back to Sct edk2-test-stable202406 + edk2 edk2-stable202405. I was able to build and run but i was hitting the similar ASSERT/HANG.

I tried following change in uefi-sct/SctPkg/UEFI/UEFI_SCT.dsc

GCC:__IA32_CC_FLAGS = -D EFI32 $(GCC_VER_MACRO) -ffreestanding -nostdinc -nostdlib -Wno-error -mno-red-zone -Wno-address -mno-stack-arg-probe "-DEFIAPI=attribute((ms_abi))" -m32 -mabi=ms -D MDE_CPU_X32

• GCC:__IA32_CC_FLAGS = -D EFIX64 $(GCC_VER_MACRO) -Wno-error >>>>>>>> remove this

• GCC:__IA32_CC_FLAGS = -D EFIX64 $(GCC_VER_MACRO) -Wno-error -mabi=ms >>>>>>>>>>> Add this.

GCC:__IA32_VFRPP_FLAGS = -D EFI32 $(GCC_VER_MACRO)

GCC:__IA32_APP_FLAGS = -D EFI32 $(GCC_VER_MACRO)

GCC:__IA32_PP_FLAGS = -D EFI32 $(GCC_VER_MACRO)

GCC:__X64_CC_FLAGS = -D EFIX64 $(GCC_VER_MACRO) -ffreestanding -nostdinc -nostdlib -Wno-error -mno-red-zone -Wno-address -mno-stack-arg-probe "-DEFIAPI=attribute((ms_abi))" -m64 -mcmodel=large -mabi=ms -D MDE_CPU_X64

• GCC:__X64_CC_FLAGS = -D EFIX64 $(GCC_VER_MACRO) -Wno-error >>>>>>>>> remove this.

• GCC:__X64_CC_FLAGS = -D EFIX64 $(GCC_VER_MACRO) -Wno-error -mabi=ms >>>>>>>>> Add this.

GCC:__X64_VFRPP_FLAGS = -D EFIX64 $(GCC_VER_MACRO)

GCC:__X64_APP_FLAGS = -D EFIX64 $(GCC_VER_MACRO)

After above change i was able to run X64 builds with GCC properly.