On-Device Large Language Model Inference at the Network Edge

Research question

How can one model artifact execute across heterogeneous consumer devices while respecting memory, thermal, latency and privacy constraints?

Architecture examined

The report describes a cross-platform runtime spanning iOS, Android and web environments, with hardware capability profiling, execution-provider selection and a hybrid retrieval design.

Optimization position

Quantization, runtime selection and memory planning are treated as system concerns rather than isolated model-compression tasks. The paper describes 4-bit post-training quantization and device-aware calibration.

Limits and open questions

This is an architectural technical report, not evidence of a client deployment or commercial result. Production fitness still depends on device populations, sustained thermal behavior, workload distributions, security review and field measurement.

Decision relevance

The work informs whether a private or latency-sensitive AI workload belongs on-device, in a hybrid topology or in centralized infrastructure.