Universal Inference API #183

Abstract
Individual frameworks such as uTensor and TFLM have their own sets of on-device APIs. In some cases, significant boilerplate code and framework-specific knowledge are required to implement an inference task at its simplest form. A developer-friendly universal high-level inference API will be valuable for on-device ML.

On-device inferencing is generalized into these steps:

• Configuration (optional)
• Setting up the input
• Evaluating the model
• Reading the output

The code snippets below aim to illustrate the current API designs for uTensor and Tensorflow. The newly proposed API will likely utilize the code-generation technology to create an adaptor layer between the universal interface and the underlying framework-specific APIs.

Examples:

uTensor:

  Context ctx;  //creating the context class, the stage where inferences take place 
  //wrapping the input data in a tensor class
  Tensor* input_x = new WrappedRamTensor<float>({1, 784}, (float*) input_data);
  get_deep_mlp_ctx(ctx, input_x);  // pass the tensor to the context
  S_TENSOR pred_tensor = ctx.get("y_pred:0");  // getting a reference to the output tensor
  ctx.eval(); //trigger the inference

TFLM:
Please refer to this hello-world example

Requirements

The newly proposed API should have high-level abstraction aims to accelerate, simplify application development, and, helps to streamline the edge-ML deployment flow, especially for resource-constrained devices.

The new API should:

• Framework/tool and platform-independent
• Encapsulate/abstract framework-specific boilerplate code
• Provide a clear interface that enables collaboration between data scientists and embedded engineers
• Prioritize developer-experience and simplicity

Proposals

1. Single-function-call inferencing, by @janjongboom ,

uint8_t utensor_mem_pool[4096]; // <-- CLI should tell me how much I need

utensor_something_autogenerated_init(utensor_mem_pool);

float input[33] = { 1,2,3,4 ... }
float output[5];

utensor_run_something_autogenerated(input, 33, output, 5);

2. Model object, discussion with @sandeepmistry, @mbartling and @neil-tan

char input_buffer[512];
int result[1];

MyModel model; //generated
model.setArenaSize(1024);
model.bind_input0(input_buffer, input_buffer_size);
model.bind_prediction0(result, 1);
model.run();

printf(“The inference result is: %d”, result[0]);

This is at its most minimal. The generated bind method names corresponding to the tensor names in the graph. The method’s signatures reflect their respective tensor-data-types. Additional methods can be implemented to support advanced configurations.

What’s Next

This issue serves as a starting point for this discussion. It will be reviewed by uTensor core-devs, Arduino, ISG data scientists, IPG engineers, and Google. We are be particular interested in reviewing use-cases which the current proposed API cannot cover. We are looking to reiterate and converge on a design in the next weeks.