8351487: [ubsan] jvmti.h runtime error: load of value which is not a valid value

[MBaesken]

When running with ubsan - enabled binaries, the following 2 issues are observed in jvmti.h (on macOS aarch64).

jtreg test vmTestbase/nsk/jvmti/SetVerboseFlag/setvrbflag002/TestDescription.java 

/priv/jenkins/client-home/workspace/openjdk-jdk-weekly-macos_aarch64-opt/build/support/modules_include/java.base/jvmti.h:2645:44: runtime error: load of value 4294967295, which is not a valid value for type 'jvmtiVerboseFlag'
    #0 0x104453540 in _jvmtiEnv::SetVerboseFlag(jvmtiVerboseFlag, unsigned char) jvmti.h:2645
    #1 0x10444ed5c in agentProc(_jvmtiEnv*, JNIEnv_*, void*) setvrbflag002.cpp:49
    #2 0x10444f1c8 in agentThreadWrapper(_jvmtiEnv*, JNIEnv_*, void*) agent_tools.cpp:151
    #3 0x10914895c in JvmtiAgentThread::call_start_function() jvmtiImpl.cpp:89
    #4 0x108ebd7f0 in JavaThread::thread_main_inner() javaThread.cpp:776
    #5 0x1096f6094 in Thread::call_run() thread.cpp:231
    #6 0x10941c37c in thread_native_entry(Thread*) os_bsd.cpp:601
    #7 0x1936fef90 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x6f90)
    #8 0x1936f9d30 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1d30)

vmTestbase/nsk/jvmti/GetErrorName/geterrname002/TestDescription.java

/priv/jenkins/client-home/workspace/openjdk-jdk-weekly-macos_aarch64-opt/build/support/modules_include/java.base/jvmti.h:2640:42: runtime error: load of value 4294967295, which is not a valid value for type 'jvmtiError'
    #0 0x1002b3504 in _jvmtiEnv::GetErrorName(jvmtiError, char**) jvmti.h:2640
    #1 0x1002aec9c in agentProc(_jvmtiEnv*, JNIEnv_*, void*) geterrname002.cpp:50
    #2 0x1002af198 in agentThreadWrapper(_jvmtiEnv*, JNIEnv_*, void*) agent_tools.cpp:151
    #3 0x104f4895c in JvmtiAgentThread::call_start_function() jvmtiImpl.cpp:89
    #4 0x104cbd7f0 in JavaThread::thread_main_inner() javaThread.cpp:776
    #5 0x1054f6094 in Thread::call_run() thread.cpp:231
    #6 0x10521c37c in thread_native_entry(Thread*) os_bsd.cpp:601
    #7 0x1936fef90 in _pthread_start+0x84 (libsystem_pthread.dylib:arm64e+0x6f90)
    #8 0x1936f9d30 in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1d30)

So the problem reported is SetVerboseFlag method (which is part of the VM) calls SetVerboseFlag function with that bad argument ; but the test code seems to intentionally use 'bad' arguments . But this does not work well with ubsan ; so better exclude those tests from ubsan checking .
Please note that jvmti.h is shipped so adding some macros/attributes to methods for avoiding ubsan is probably not the best thing to do in this case.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8351487: [ubsan] jvmti.h runtime error: load of value which is not a valid value (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/26159/head:pull/26159
$ git checkout pull/26159

Update a local copy of the PR:
$ git checkout pull/26159
$ git pull https://git.openjdk.org/jdk.git pull/26159/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 26159

View PR using the GUI difftool:
$ git pr show -t 26159

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/26159.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back mbaesken! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@MBaesken The following label will be automatically applied to this pull request:

• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (a0ca0ddc)
• 00: Full (a346557b)

[plummercj]

This is somewhat of an odd situation. SetVerboseFlag() says the argument type is an enum, but it treats it like an int and does error checking to make sure it one of the enum values. If ubsan dug a little deeper to see how the argument is handled, it would see that and not complain.