How to deploy multimodal models on edge devices with ONNX Runtime and CoreML?

Export the model to ONNX, apply static quantization, then run it with ONNX Runtime on Android or convert it to CoreML for iOS deployment.

1. Export: Use torch.onnx.export with explicit input_names, output_names, and dynamic_axes for image and text tensors.
2. Validate: Load the ONNX file with onnx.checker.check_model to catch unsupported ops early.
3. Optimize: Run onnxruntime-tools to fuse nodes and eliminate dead code.
4. Quantize: Apply static 8‑bit integer quantization (per‑channel weights) to meet edge latency and memory budgets.
5. Android path:
- Add implementation "com.microsoft.onnxruntime:onnxruntime-android:1.17.0" to Gradle.
- Load the .ort model with OrtEnvironment and OrtSession using SessionOptions → setOptimizationLevel(ORT_ENABLE_ALL).
6. iOS path:
- Convert the quantized ONNX to CoreML with coremltools.convert(..., compute_units=ct.ComputeUnit.ALL).
- Compile to .mlmodelc using Xcode's xcrun coremlcompiler compile.
7. Inference: Feed a pre‑processed image (float32, 224x224, RGB) and a padded token sequence (fixed length, e.g., 16) to the runtime API.

Feature comparison
| Feature | ONNX Runtime (Android) | CoreML (iOS) |
|---|---|---|
| Supported ops | ~95 % of PyTorch ops (v1.17) | ~90 % of ONNX ops (v7) |
| Quantization | 8‑bit int static/dynamic | 8‑bit int, float16 |
| Typical latency (ResNet‑50) | ~12 ms @ Snapdragon 8 Gen 2 | ~9 ms @ A16 Bionic |
| Tooling | onnxruntime‑tools, ort‑mobile | coremltools, mlmodelc |

Checklist
- [ ] Export with opset_version>=17.
- [ ] Verify that all custom ops are replaced by ONNX equivalents.
- [ ] Quantize with --per_channel True for weights.
- [ ] For iOS, ensure text input is padded to a constant length.
- [ ] Profile on‑device memory; aim for <150 MB total model footprint.

import torch, onnx
model = torch.load("multimodal.pt")
dummy = {"image": torch.randn(1,3,224,224), "text": torch.randint(0,30522,(1,16))}
torch.onnx.export(model, (dummy["image"], dummy["text"]), "model.onnx",
                input_names=["image","text"], output_names=["logits"],
                opset_version=17, dynamic_axes={"image":{0:"batch"}, "text":{0:"batch"}})

python -m onnxruntime.tools.convert_onnx_models_to_ort \
    --model_path model.onnx --output_path model_quant.onnx \
    --optimization_level 99 --quantization_mode static \
    --per_channel True --weight_type QInt8

import coremltools as ct
mlmodel = ct.convert("model_quant.onnx", source="onnx",
                     inputs=[ct.ImageType(name="image", shape=(1,3,224,224)),
                             ct.TensorType(name="text", shape=(1,16), dtype="int32")],
                     compute_units=ct.ComputeUnit.ALL)
mlmodel.save("Multimodal.mlmodel")

Gotcha: CoreML does not support variable‑length text tensors; you must pad or truncate the token sequence to a fixed length before conversion, otherwise the app will crash at launch.