Skip to content

Update kernel porting documentation #3144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update kernel porting documentation #3144

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 76 additions & 23 deletions tensorflow/lite/micro/docs/porting_reference_ops.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,17 @@ https://github.com/ekalinin/github-markdown-toc#auto-insert-and-update-toc
* [Notes](#notes)
* [Frequently Asked Questions](#frequently-asked-questions)
* [Can I use malloc/free or new/delete in my operator code?](#can-i-use-mallocfree-or-newdelete-in-my-operator-code)
* [Can I use static variable allocation in my operator code?](#can-i-use-static-variable-allocation-in-my-operator-code)
* [Can I use global/static variable constructors in my operator code](#can-i-use-globalstatic-variable-constructors-in-my-operator-code)
* [How do I allocate persistent memory?](#how-do-i-allocate-persistent-memory)
* [When am I allowed to allocate persistent memory?](#when-am-i-allowed-to-allocate-persistent-memory)
* [How do I allocate/use temporary memory?](#how-do-i-allocateuse-temporary-memory)
* [When can I allocate/use temporary memory?](#when-can-i-allocateuse-temporary-memory)
* [Can I resize my input/output tensors?](#can-i-resize-my-inputoutput-tensors)
* [Can I resize the tensors of my model?](#can-i-resize-the-tensors-of-my-model)
* [Can I change the shape of tensors in my operator code?](#can-i-change-the-shape-of-tensors-in-my-operator-code)
* [When can I change the shape of tensors in my operator code?](#when-can-i-change-the-shape-of-tensors-in-my-operator-code)
* [Can I modify a TfLiteTensor or TfLiteEvalTensor?](#can-i-modify-a-tflitetensor-or-tfliteevaltensor)
* [When can I inspect tensor data?](#when-can-i-inspect-tensor-data)
* [How do I fix optimized kernel unit test failures?](#how-do-i-fix-optimized-kernel-unit-test-failures)

<!-- Added by: advaitjain, at: Thu 16 Sep 2021 11:49:51 AM PDT -->

Expand Down Expand Up @@ -278,11 +280,17 @@ differences are actually significant or just stylistic.
## Can I use malloc/free or new/delete in my operator code?
No. All memory allocation in TensorFlow Lite Micro (TFLM) is done using C++
stack based automatic allocation, or through specialized TFLM persistent
and temporary allocation methods.
and temporary allocation methods. The only use of `new` in TFLM is through the
use of [placement new](https://en.cppreference.com/w/cpp/language/new.html#Placement_new).
## Can I use static variable allocation in my operator code?
No. This is due to the call ordering of C++ static constructors being
platform/compiler dependent.
## Can I use global/static variable constructors in my operator code?
Global constructors are not allowed. This is due to the call ordering of C++
global constructors being platform/compiler dependent. Some platforms do not
support global constructors at all, and will silently fail.
Static variables that use constructors are safe to declare within methods. This
is because the static variable executes its call-once sequence only when the
encapsulating method is called.
## How do I allocate persistent memory?
Use `TfLiteContext::AllocatePersistentBuffer` to allocate persistent memory.
Expand All @@ -306,9 +314,9 @@ Use the `TfLiteContext::RequestScratchBufferInArena` and
`TfLiteContext::GetScratchBuffer` methods. The temporary memory is shared
between all operators, and is only valid for your operator within the scope
of your operator's `Invoke` method. Do not attempt to use temporary memory
to share data between operator invocations. Temporary memory is to be used
only as pre-allocated storage during the execution scope of your operator's
`Invoke` method.
to share data between operator invocations. Scratch buffer temporary memory is
to be used only as pre-allocated storage during the execution scope of your
operator's `Invoke` method.

An example code snippet looks like ([add_n.cc](../kernels/add_n.cc)):
```C++
Expand All @@ -321,7 +329,7 @@ if (output->type == kTfLiteFloat32) {
context, scratch_size, &scratch_index));
node->user_data =
reinterpret_cast<decltype(node->user_data)>(scratch_index);
}
}
```
And to use the buffer:
```C++
Expand All @@ -330,24 +338,59 @@ int scratch_index =
void* scratch_buffer = context->GetScratchBuffer(context, scratch_index);
```
If temporary memory is needed for a calculation during the operator's `Prepare`
method, use `MicroContext::AllocateTempBuffer`. A matching call to
`MicroContext::DeallocateTempBuffer` must be made prior to returning from your
operator's `Prepare` method.
An example code snippet ([depthwise_conv_vision.cc](../kernels/xtensa/depthwise_conv_vision.cc)):
```C++
TfLiteTensor filter_int8;
if (filter->type == kTfLiteInt4) {
const size_t bytes_unpacked = filter->bytes * 2;
filter_int8.data.data = micro_context->AllocateTempBuffer(
bytes_unpacked, tflite::MicroArenaBufferAlignment());
filter_int8.dims = filter->dims;
filter_int8.type = kTfLiteInt8;
tflite::tensor_utils::UnpackDenseInt4IntoInt8(
GetTensorData<int8_t>(filter), GetTensorShape(filter).FlatSize(),
GetTensorData<int8_t>(&filter_int8));
}
...
if (filter->type == kTfLiteInt4) {
micro_context->DeallocateTempBuffer(GetTensorData<uint8_t>(&filter_int8));
}
```
## When can I allocate/use temporary memory?
The `TfLiteContext::RequestScratchBufferInArena` method is available only within
the scope of your operator's `Prepare` method.
The `TfLiteContext::GetScratchBuffer` method is available only within
the scope of your operator's `Invoke` method.
## Can I resize my input/output tensors?
No. The storage space for each input/output tensor is a fixed, calculated value
determined at the time the TensorFlow Lite (TfLite) model converter is executed.
During the `Init` phase of the `tflite::MicroInterpreter` all tensor storage is
allocated by the `tflite::MicroInterpreter` instance, using the calculated values
of the model converter.
The `MicroContext::AllocateTempBuffer` and `MicroContext::DeallocateTempBuffer`
methods are only available within the scope of your operator's `Prepare` method.
## Can I resize the tensors of my model?
No. The storage space for each `input`/`output`/`const` tensor is a fixed,
calculated value determined at the time the TensorFlow Lite (TfLite) model
converter is executed.
All tensor storage is allocated by the `tflite::MicroInterpreter` instance,
using the tensor shape values of the TfLite model converter.
TFLM does not support dynamic tensors. A dynamic tensor is one whose storage
(and shape) can change during inference.
For more information see: [Memory Allocation Overview](online_memory_allocation_overview.md)
## Can I change the shape of tensors in my operator code?
Yes. The new shape must not exceed the storage space indicated by the old shape.
Because tensor shape values may live in memory that is not directly writable
(ex. Flash, EEPROM, ROM), a special method must be called before modification
Yes. The new shape must be exactly equal to the storage space indicated by the
old shape.
Because tensor shape values may live in non-volatile memory that is not directly
writable (ex. Flash, ROM), a special method must be called before modification
is attempted. The `tflite::micro::CreateWritableTensorDimsWithCopy` method will
move the tensor shape values to guaranteed persistent writable memory.
Expand Down Expand Up @@ -378,8 +421,18 @@ structures. Your code should not modify these data structures. The only
directly allowed modification of tensors is to change their data values, or
their shape values.
## When can I inspect tensor data?
Tensor data can always be inspected within the scope of your operator's `Invoke`
method.
Within the scope of your operator's `Prepare` method, use
`tflite::IsConstantTensor` before inspecting the tensor data.
Only when `tflite::IsConstantTensor` returns `true`, does the tensor have valid
data.
## How do I fix optimized kernel unit test failures?
Kernel unit tests for all optimizated kernels should pass. By default kernel unit
tests for the newly added op may fail for optimized kernels as they may not have the
correct references. In this case, we should let the optimized kernels fall back
to the newly added reference kernels. For example, refer to this [this commit](https://github.com/tensorflow/tflite-micro/pull/1274/commits/d36c9dd598dcbf352f2c60463fd0d4153703a1cd).
Kernel unit tests for all optimized kernels should pass.
It should be noted that optimized kernels may not handle all tensor types or
operator parameters.
In this case, the optimized kernel should fallback on calling the reference
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe providing example commit will be helpful?

kernel methods.
Loading