Add simple_tensor_and_data_graph sample #1394

arovir01 · 2025-08-13T14:13:48Z

Added initial tensor and data graph sample, common framework and documentation.

NOTE: This is essentially the same code as in the previously closed PR no. 1381, but the sample (and thus the branch) was renamed.

Description

This sample demonstrates the usage of the VK_ARM_tensors and VK_ARM_data_graph extensions for machine learning inference. These allow you to define a neural network with TOSA operations and then execute inferences using hardware acceleration.

This is the first in a series of samples which introduce the tensor and graph extensions.

General Checklist:

Please ensure the following points are checked:

Note: The Samples CI runs a number of checks including:

I have updated the header Copyright to reflect the current year (CI build will fail if Copyright is out of date)
My changes build on Windows, Linux, macOS and Android. Otherwise I have documented any exceptions

If this PR contains framework changes:

I did a full batch run using the batch command line argument to make sure all samples still work properly

Sample Checklist

If your PR contains a new or modified sample, these further checks must be carried out in addition to the General Checklist:

I have tested the sample on at least one compliant Vulkan implementation
If the sample is vendor-specific, I have tagged it appropriately
I have stated on what implementation the sample has been tested so that others can test on different implementations and platforms
Any dependent assets have been merged and published in downstream modules
For new samples, I have added a paragraph with a summary to the appropriate chapter in the readme of the folder that the sample belongs to e.g. api samples readme
For new samples, I have added a tutorial README.md file to guide users through what they need to know to implement code using this feature. For example, see conditional_rendering
For new samples, I have added a link to the Antora navigation so that the sample will be listed at the Vulkan documentation site

* Added initial tensor and data graph sample, common framework and documentation. Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

…_graph

asuessenbach

Some first comments...

framework/vulkan_type_mapping.h

...tensions/tensor_and_data_graph/simple_tensor_and_data_graph/simple_tensor_and_data_graph.cpp

samples/extensions/tensor_and_data_graph/tensor_and_data_graph_common.cpp

samples/extensions/tensor_and_data_graph/tensor_and_data_graph_common.h

asuessenbach

One more comment...

samples/extensions/tensor_and_data_graph/simple_tensor_and_data_graph/README.adoc

samples/tooling/profiles/vulkan_profiles.hpp

* Fixed issues highlighted by reviewers * Fixed copyright headers Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

arovir01 · 2025-09-03T16:10:32Z

@asuessenbach

It seems the script checking the copyright headers expects all copyright lines to be upgraded to the current year (2025), not just the one referring to the contributor making the latest changes (Arm). In consequence, the following header will result in failure:

/* Copyright (c) 2019-2025, Arm Limited and Contributors
 * Copyright (c) 2021-2024, NVIDIA CORPORATION. All rights reserved.

In effect the script is expecting us to update NVIDIA's copyright line to 2025 as well. I'm wondering if this makes sense, or whether it is a bug in the copyright checker script. Can you please confirm?

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

arovir01 · 2025-09-08T16:00:48Z

@asuessenbach

It seems the script checking the copyright headers expects all copyright lines to be upgraded to the current year (2025), not just the one referring to the contributor making the latest changes (Arm). In consequence, the following header will result in failure:
/* Copyright (c) 2019-2025, Arm Limited and Contributors
 * Copyright (c) 2021-2024, NVIDIA CORPORATION. All rights reserved.
In effect the script is expecting us to update NVIDIA's copyright line to 2025 as well. I'm wondering if this makes sense, or whether it is a bug in the copyright checker script. Can you please confirm?

Thanks for clarifying this, I have now updated the copyright headers according to what the CI script expects.

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

arovir01 · 2025-09-15T12:12:37Z

@asuessenbach Clang-format still fails in CI but passes locally. I suspect it is due to a mismatch in clang-format versions between my machine and CI, as you also suggested. If you're happy with what the code looks like, I would propose to ignore this check and get the PR merged nonetheless.

asuessenbach · 2025-09-15T14:22:42Z

Apparently, there's more than just the format check failing: on ios, the release build is failing as well.

SaschaWillems · 2025-09-15T14:47:00Z

The iOS issue is caused by the CI using an old version of the SDK where the shader compiler is lacking required extensions. I encountered the same issues with my Slang shaders PRs and had to disable shader compilation in CI on that platform.

That's something we need to fix separately.

arovir01 · 2025-09-16T10:02:49Z

Currently we do not support iOS and macOS – even though the build succeeds on macOS – as running the sample will not be possible via MoltenVK .

Since for now we have to rely on the ML Emulation Layer for Vulkan to execute this sample, we can only support the platforms that are able to run the Emulation Layer, namely Windows and Linux, with Android support being experimental.

All this is mentioned in the readme.

SaschaWillems · 2025-09-16T10:18:56Z

Then you could as well disable the sample on Mac/iOS. We do that for some of the samples, e.g. the OpenCL ARM sample that only works on Android. This is done at CMake level, see https://github.com/KhronosGroup/Vulkan-Samples/blob/main/samples/extensions/open_cl_interop_arm/CMakeLists.txt as an example.

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

matthewsloyanARM · 2025-09-30T15:25:50Z

Hi all,

Thank you for all your comments so far. We really appreciate it. I believe Aron has sorted the issues mentioned above and all checks are now passing. If possible could we get a few more reviewers to take a look please?

If you would like to try out this new sample locally, we provide an emulation layer which allows them to run. This can be found here: https://github.com/arm/ai-ml-emulation-layer-for-vulkan

There is also a pre-built Windows package available here: https://www.arm.com/-/media/Files/developer/MLEmulationLayerForVulkan or you can grab this through our tutorial: https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/vulkan-ml-sample/. This tutorial is actually for this first sample in this PR, so it shows you everything needed to try it out. If you do have any questions feel free to reach out.

The guide and pre-built is only for Windows though, it does work on Linux, but this will require a manual build from the repository (https://github.com/arm/ai-ml-emulation-layer-for-vulkan). See the build guide for this.

Lastly, we have also had to update the vulkan-headers and volk versions to 1.4.321.0, as support for these extensions was only added recently. Let me know your thoughts on this?

Thanks,

Matthew

SaschaWillems · 2025-10-01T16:22:15Z

I'll do a review before the next sample meeting.

SaschaWillems · 2025-10-01T19:02:47Z

samples/extensions/tensor_and_data_graph/README.adoc

+The samples in this folder are:
+
+* xref:simple_tensor_and_data_graph/README.adoc[Simple tensor and data graph]
+* xref:graph_constants/README.adoc[Graph constants]


None of these folders (except the first) exists (yet). Might be better to remove and add back once the (yet) missing samples land.

I agree, I have removed these and will add them back in follow up samples. Thank you!

SaschaWillems · 2025-10-01T19:03:50Z

samples/extensions/tensor_and_data_graph/README.adoc

+
+Alternatively, on Windows, the Vulkan Configurator tool — which is part of the Vulkan SDK — can also be used, as described below. If it is not already installed, it can be obtained from the link:https://vulkan.lunarg.com/[Vulkan SDK] website.
+
+Once installed, open Vulkan Configurator and navigate to the Vulkan Layers Location tab. Click the plus (+) symbol to append a user-defined layers path,


Should add a note that the validation layers need to be disabled, otherwise the sample will crash as the layers do not (yet) know about the new extension. At least that was the case for me.

We haven't encountered this so far, but it's definitely no harm adding it. Thank you for letting us know.

SaschaWillems · 2025-10-01T19:09:03Z

samples/extensions/tensor_and_data_graph/simple_tensor_and_data_graph/README.adoc

@@ -0,0 +1,589 @@
+:source-highlighter: coderay


I don't think this will work with the Antora site build and needs to be removed.

Removed, thank you!

SaschaWillems · 2025-10-03T16:28:04Z

samples/extensions/tensor_and_data_graph/README.adoc

+
+Note: It is recommended to disable the validation layer when running these samples as the new extensions are not supported yet.
+
+image::loading_layers.png[Loading Layers Completed]


Needs to be changed to

image::./loading_layers.png[Loading Layers Completed]

Done, thanks!

SaschaWillems · 2025-10-03T16:28:18Z

samples/extensions/tensor_and_data_graph/README.adoc

+
+This should add the layers to the Vulkan Loader Management tab. The Graph layer needs to sit on top of the Tensor layer.
+
+image::verify_layers.png[VK Layers]


Needs to be changed to

image::./verify_layers.png[VK Layers]

Done, thanks!

SaschaWillems · 2025-10-03T16:31:02Z

samples/extensions/tensor_and_data_graph/simple_tensor_and_data_graph/README.adoc

+
+== Conclusion
+
+In this sample we've introduced the VK_ARM_tensors and VK_ARM_data_graph extensions and shown how to use them to run a simple neural network. We've shown the steps needed to run a neural network - creating a pipeline layout, data graph pipeline, data graph pipeline session, tensor objects, descriptor sets and finally recording binding and dispatch commands in a command buffer. The following samples in xref:..\README.adoc[this series] build upon these concepts.


Link needs to be changed to

xref:samples/extensions/tensor_and_data_graph/README.adoc[this series]

Done, thank you!

SaschaWillems · 2025-10-03T16:43:39Z

You also should add the simple tensor sample as a child item to the tensor samples in the nav.adoc

SaschaWillems · 2025-10-03T16:44:11Z

Btw. the sample itself runs fine on my setup using the emulation layers. So once the documentation things are fixed, this is good to go (from my side).

matthewsloyanARM · 2025-10-06T14:49:46Z

That's great, thanks Sascha for reviewing this!

I had some trouble trying to build the docs site myself, is there is a guide or documentation on how to do this? I have created a patch anyway with the changes you suggested above, which hopefully will cover it.

SaschaWillems · 2025-10-06T14:58:30Z

That's great, thanks Sascha for reviewing this!

I had some trouble trying to build the docs site myself, is there is a guide or documentation on how to do this? I have created a patch anyway with the changes you suggested above, which hopefully will cover it.

Sadly not. I do have my own docs site testbed that's easier to build (due to not including the spec). Might be a good idea to offer something like that with an easy to use explanation to all maintainers of the samples.

SaschaWillems · 2025-10-06T15:13:44Z

Copyright header check fails in antora/CMakeLists.txt. If you can fix that I think this PR is good to go :)

CLAassistant · 2025-10-06T15:38:17Z

All committers have signed the CLA.

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

gary-sweet

I eventually got far enough that the sample just reports "[error] Required device extension VK_KHR_deferred_host_operations not available, cannot run", so that's as far as I can go I'm afraid.

arovir01 · 2025-10-07T13:34:22Z

I eventually got far enough that the sample just reports "[error] Required device extension VK_KHR_deferred_host_operations not available, cannot run", so that's as far as I can go I'm afraid.

Are you trying to run the sample on macOS? MoltenVK does not seem to fully support VK_KHR_deferred_host_operations, therefore the sample is not expected to work on macOS.

gary-sweet · 2025-10-07T13:40:16Z

I eventually got far enough that the sample just reports "[error] Required device extension VK_KHR_deferred_host_operations not available, cannot run", so that's as far as I can go I'm afraid.

Are you trying to run the sample on macOS? MoltenVK does not seem to fully support VK_KHR_deferred_host_operations, therefore the sample is not expected to work on macOS.

No, it's our own hardware, which doesn't support VK_KHR_deferred_host_operations. I'm really just saying that I can't test this.

arovir01 · 2025-10-07T13:49:10Z

I eventually got far enough that the sample just reports "[error] Required device extension VK_KHR_deferred_host_operations not available, cannot run", so that's as far as I can go I'm afraid.

Are you trying to run the sample on macOS? MoltenVK does not seem to fully support VK_KHR_deferred_host_operations, therefore the sample is not expected to work on macOS.

No, it's our own hardware, which doesn't support VK_KHR_deferred_host_operations. I'm really just saying that I can't test this.

VK_ARM_data_graph depends on VK_KHR_deferred_host_operations, so I'm afraid the sample will only run on hardware which supports the latter.

…_graph

Signed-off-by: Matthew Sloyan <matthew.sloyan@arm.com>

gary-sweet

I did a small amount of work to make VK_KHR_deferred_host_operations appear available on our platform. I then see an assert in our driver where the compiler is seeing 64-bit integers being used.

It looks like the emulation layer is using 64-bit integers in shaders. Those 64-bit extensions should probably be added to this sample so that it gracefully exits as unsupported if they aren't available.

Signed-off-by: Matthew Sloyan <matthew.sloyan@arm.com>

matthewsloyanARM · 2025-10-08T16:13:57Z

Thanks @gary-sweet for the suggestion and for trying this out, I have added a shaderInt64 check to the request_gpu_features function, hopefully that is what you meant.

matthewsloyanARM · 2025-10-14T11:33:51Z

Hi all,

Thanks for all the reviews!

@gary-sweet I am not sure if there is anything to do on my side to address the change requested, I pushed a patch to fix it, but, possibly it needs your input? Also would you be happy to approve based on your code review? However, I know you were blocked due to the VK_KHR_deferred_host_operations dependency. No worries if not.

@asuessenbach I believe you had said you wouldn't have time to try out the sample which is no problem at all, but would you also be happy to approve based on the code review you conducted? Again no worries if not.

Lastly, @SaschaWillems thanks again for trying this out and reviewing it. Your approval has been removed due to the changes above, would it be possible to add this again if you don't mind?

If anyone else does want to try it out please see my comment above: #1394 (comment)

Kind regards,

Matthew

gary-sweet

The best that I can do is to remove my "Request Changes" sadly. I can't run any more of this so I won't be able to approve it myself.

Changes made as requested

asuessenbach · 2025-10-14T13:54:29Z

I tried to run this sample using the emulation layer as described in https://github.com/arm/ai-ml-emulation-layer-for-vulkan, but failed.

I've copied the dll- and json-files from MLEmulationLayerForVulkan.zip into some directory, set the environment in VS to VK_ADD_LAYER_PATH="the\path\to\the\layers"
VK_INSTANCE_LAYERS="VK_LAYER_ML_Graph_Emulation;VK_LAYER_ML_Tensor_Emulation"

But there's still no "VK_ARM_tensors" extension supported.

Anything else I need to do, to get that?

matthewsloyanARM · 2025-10-15T08:28:53Z

Thanks @asuessenbach for trying this out. Hmm I have a feeling it's the Vulkan SDK version that's being used. Are you using 1.4.321 by any chance?
Also although the flags should work, we recommend to use the Vulkan Configurator on Windows in our guides, shown here for example: https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/vulkan-ml-sample/2-ml-ext-for-vulkan/

arovir01 added 2 commits August 13, 2025 15:09

Add simple_tensor_and_data_graph sample

98e67ff

* Added initial tensor and data graph sample, common framework and documentation. Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

Merge branch 'KhronosGroup:main' into arovir01/simple_tensor_and_data…

06ce09f

…_graph

asuessenbach reviewed Aug 14, 2025

View reviewed changes

asuessenbach reviewed Aug 18, 2025

View reviewed changes

samples/extensions/tensor_and_data_graph/tensor_and_data_graph_common.h Outdated Show resolved Hide resolved

asuessenbach reviewed Aug 19, 2025

View reviewed changes

samples/extensions/tensor_and_data_graph/simple_tensor_and_data_graph/README.adoc Outdated Show resolved Hide resolved

asuessenbach reviewed Aug 20, 2025

View reviewed changes

samples/tooling/profiles/vulkan_profiles.hpp Show resolved Hide resolved

arovir01 force-pushed the arovir01/simple_tensor_and_data_graph branch 2 times, most recently from 1095425 to 7631290 Compare August 20, 2025 16:27

Address 1st round of PR comments

4e46f43

* Fixed issues highlighted by reviewers * Fixed copyright headers Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

arovir01 force-pushed the arovir01/simple_tensor_and_data_graph branch from 7631290 to 4e46f43 Compare August 21, 2025 15:27

Fix copyright headers and formatting

ab6a0b3

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

Merge branch 'main' into arovir01/simple_tensor_and_data_graph

4db21e2

arovir01 force-pushed the arovir01/simple_tensor_and_data_graph branch from e2c3979 to 4db21e2 Compare September 8, 2025 16:09

Fix build after merging new PhysicalDevice API

4097cb0

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

arovir01 requested a review from asuessenbach September 15, 2025 12:09

arovir01 added 2 commits September 16, 2025 15:36

Disable building the sample on unsupported platforms

161edc3

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

Reformatted with clang-format-15

7f1c2e6

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

SaschaWillems self-requested a review October 1, 2025 16:21

SaschaWillems reviewed Oct 1, 2025

View reviewed changes

SaschaWillems reviewed Oct 3, 2025

View reviewed changes

SaschaWillems previously approved these changes Oct 6, 2025

View reviewed changes

matthewsloyanARM dismissed SaschaWillems’s stale review via 30fcd0a October 6, 2025 15:38

Address .adoc link issues

aad840f

Signed-off-by: Aron Virginas-Tar <aron.virginas-tar@arm.com>

arovir01 force-pushed the arovir01/simple_tensor_and_data_graph branch from 30fcd0a to aad840f Compare October 6, 2025 16:23

gary-sweet reviewed Oct 7, 2025

View reviewed changes

matthewsloyanARM and others added 2 commits October 7, 2025 14:55

Merge branch 'KhronosGroup:main' into arovir01/simple_tensor_and_data…

34e3e53

…_graph

Update simple_tensor_and_data_graph sample to latest main

9256e2c

Signed-off-by: Matthew Sloyan <matthew.sloyan@arm.com>

gary-sweet previously requested changes Oct 8, 2025

View reviewed changes

Fix clang-format issues and add Int64 check

620db84

Signed-off-by: Matthew Sloyan <matthew.sloyan@arm.com>

gary-sweet reviewed Oct 14, 2025

View reviewed changes

gary-sweet self-requested a review October 14, 2025 11:56

gary-sweet removed their request for review October 14, 2025 11:58

SaschaWillems approved these changes Oct 14, 2025

View reviewed changes


		Alternatively, on Windows, the Vulkan Configurator tool — which is part of the Vulkan SDK — can also be used, as described below. If it is not already installed, it can be obtained from the link:https://vulkan.lunarg.com/[Vulkan SDK] website.

		Once installed, open Vulkan Configurator and navigate to the Vulkan Layers Location tab. Click the plus (+) symbol to append a user-defined layers path,


		Note: It is recommended to disable the validation layer when running these samples as the new extensions are not supported yet.

		image::loading_layers.png[Loading Layers Completed]


		This should add the layers to the Vulkan Loader Management tab. The Graph layer needs to sit on top of the Tensor layer.

		image::verify_layers.png[VK Layers]


		== Conclusion

		In this sample we've introduced the VK_ARM_tensors and VK_ARM_data_graph extensions and shown how to use them to run a simple neural network. We've shown the steps needed to run a neural network - creating a pipeline layout, data graph pipeline, data graph pipeline session, tensor objects, descriptor sets and finally recording binding and dispatch commands in a command buffer. The following samples in xref:..\README.adoc[this series] build upon these concepts.

Add simple_tensor_and_data_graph sample #1394

Are you sure you want to change the base?

Add simple_tensor_and_data_graph sample #1394

Uh oh!

Conversation

arovir01 commented Aug 13, 2025

Description

General Checklist:

Sample Checklist

Uh oh!

asuessenbach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asuessenbach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

arovir01 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arovir01 commented Sep 8, 2025

Uh oh!

arovir01 commented Sep 15, 2025

Uh oh!

asuessenbach commented Sep 15, 2025

Uh oh!

SaschaWillems commented Sep 15, 2025

Uh oh!

arovir01 commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SaschaWillems commented Sep 16, 2025

Uh oh!

matthewsloyanARM commented Sep 30, 2025

Uh oh!

SaschaWillems commented Oct 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SaschaWillems Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SaschaWillems commented Oct 3, 2025

Uh oh!

SaschaWillems commented Oct 3, 2025

Uh oh!

matthewsloyanARM commented Oct 6, 2025

arovir01 commented Sep 3, 2025 •

edited

Loading

arovir01 commented Sep 16, 2025 •

edited

Loading

SaschaWillems Oct 3, 2025 •

edited

Loading

CLAassistant commented Oct 6, 2025 •

edited

Loading

arovir01 commented Oct 7, 2025 •

edited

Loading

arovir01 commented Oct 7, 2025 •

edited

Loading

matthewsloyanARM commented Oct 8, 2025 •

edited

Loading