- Measure the full camera-to-result pipeline before optimising the model.
- Use representative data and evaluate errors by operating condition.
- Validate precision, compatibility, thermals and recovery on the target device.
For mL and embedded engineers deploying vision models on edge devices.
A model that runs quickly on a workstation can miss its deadline on a small device. Camera decoding, resizing, memory copies and post-processing may consume more time than the neural network. Lower precision can help, but it can also change the errors that matter to the product.
This guide follows an illustrative inspection camera that identifies damaged packaging. The task is to preserve useful detection while meeting a device budget. It does not assume that INT8 gives a fixed speedup or that a smaller model automatically uses less power in every deployment.
1. Measure the complete inference path
Break the pipeline into capture, decode, resize, normalisation, inference and post-processing. Measure each stage on the target hardware using representative image sizes and batch settings. Record queueing separately from processing, because a camera can accumulate old frames while individual inference calls appear fast.
Define the product requirement. An inspection system may need a result before an item reaches a physical point on the line. That is an end-to-end deadline, not just a model benchmark. Decide how the application handles a missed deadline or an uncertain prediction.
Establish an accuracy baseline with the original model and a held-out dataset. Record false positives and false negatives by relevant class and condition. A single average score can hide a damaging increase in missed defects for a rare packaging type. Keep the baseline and test setup reproducible.
2. Prepare representative calibration and evaluation data
Collect examples that reflect deployment conditions: lighting, camera position, motion blur, backgrounds and the range of expected objects. Keep calibration data separate from the final evaluation set so tuning decisions do not consume the evidence used to judge the result.
Check preprocessing parity before quantization. A different channel order, resize method or normalisation constant can cause an accuracy change that is mistakenly blamed on precision. Store the preprocessing configuration with the model artefact and use the same path in testing and deployment.
Review rare but consequential cases explicitly. For packaging inspection, reflective wrappers or partially occluded damage may be more important than easy examples that dominate the dataset. Document where coverage is limited. A calibration set is a representation of expected inputs, not a guarantee that unseen operating conditions will behave well.
3. Choose a precision workflow supported by the target stack
TensorRT’s quantization documentation describes supported low-precision types and explicit quantization workflows. Supported operations and build behaviour depend on the TensorRT version, model format and hardware. Check those together instead of copying a build command from an unrelated release.
Start with a supported floating-point baseline, then evaluate a lower-precision candidate. Inspect which layers use the intended precision and where conversion or fallback occurs. A nominally INT8 model may still spend substantial time in other operations.
Use post-training quantization when it meets the accuracy requirement with the available data. Consider quantization-aware training when important errors remain and the training workflow is available. Preserve the original model and every conversion configuration so you can attribute a regression to a specific change.
4. Reduce data movement and stale-frame work
Profile memory transfers and preprocessing after the first model optimisation. Repeated copies between camera buffers, CPU arrays and accelerator memory can erase inference gains. Reuse buffers where safe, keep ownership clear and validate that asynchronous processing does not read a buffer after it has been replaced.
Choose a queue policy appropriate to the task. For a live preview, dropping an old frame may be preferable to showing delayed results. For an inspection record, every item may require a retained result or an explicit failure. The application must not silently apply a latest-frame policy to a process that needs complete accounting.
Measure throughput and per-item latency together. Larger batches may improve throughput while increasing waiting time. Use the batch size and concurrency that satisfy the actual deadline, and make overload visible to the operator.
5. Test accuracy, thermal behaviour and sustained operation
Run the device long enough to expose thermal throttling, memory growth and camera recovery problems. Record ambient conditions, power mode, input rate and software versions. A short burst benchmark can overstate the performance available during continuous operation.
| Check | Evidence to collect |
|---|---|
| Accuracy | Per-class errors on held-out examples |
| Latency | End-to-end percentiles at expected input rate |
| Thermal behaviour | Sustained timing and temperature observations |
| Memory | Stable use across repeated processing |
| Recovery | Camera reconnect and process restart results |
Compare the baseline and candidate with the same conditions. If lower precision increases a consequential error class, decide whether to revise calibration, retain higher precision for sensitive operations or reject the candidate. Optimisation is a constrained decision, not simply a race for a smaller file.
6. Package a reversible deployment
Version the model, preprocessing, label map, runtime and device configuration as a compatible unit. Verify the artefact before activation and retain a known-good version. A model file on its own is not enough to reproduce the application’s predictions.
Roll out to a limited device group and observe error indicators, deadline misses and recovery behaviour. Where ground truth is delayed, plan a review sample and define how unexpected conditions are escalated. Avoid collecting unnecessary raw camera data merely because storage is available.
Keep a rollback procedure that the device can execute under the connectivity conditions it actually has. For broader model evaluation decisions, see the reasoning-model evaluation guide; the same discipline of baselines and held-out evidence applies even though the workloads differ.
Common questions
Will INT8 always be faster?
No. Hardware support, unsupported operations, conversion overhead and the rest of the pipeline affect the result. Benchmark the complete application on the target device.
How much calibration data is enough?
Coverage matters more than a universal count. Include representative operating conditions and evaluate on a separate held-out set, with attention to consequential rare cases.
Can a workstation benchmark predict edge performance?
It can help compare candidates, but it does not replace device testing. Memory bandwidth, power limits, thermal behaviour and camera processing can change the bottleneck.
Sources & further reading
Primary documentation and research used for the technical context in this guide. Examples and checklists are illustrative; validate them against your system and the versions you deploy.
