Espresso Observability: OTel on a Microcontroller

Beyond the Brew: OTel's Microcontroller Leap

This article is a masterclass in applying sophisticated observability tooling to a constrained embedded system. The author's ingenuity in adapting OpenTelemetry (OTel) to an ESP32 microcontroller is truly remarkable. Key innovations include the pragmatic approach to OTLP protobuf schema reduction using nanopb and meticulously preserving upstream field numbers, ensuring compatibility with standard collectors. The dual-core architecture for separating the real-time control loop from network I/O, employing a snapshot pattern for safe data sharing, is a textbook example of robust embedded system design. Furthermore, the on-device computation of derived metrics like puck resistance variance and the clever use of concatenated protobuf messages for high-resolution data streaming are highly noteworthy. The article effectively demonstrates that even resource-constrained devices can benefit from rich telemetry, provided the instrumentation is carefully engineered.

The primary limitation, inherent to the domain, is the complexity and the significant engineering effort required. While the author makes it look accessible, replicating this on other microcontrollers or for different applications would demand a deep understanding of embedded systems, real-time operating systems (RTOS), network protocols, and the OTel specification. The 'clocks lie' problem, while elegantly handled, is a reminder of the foundational challenges in embedded telemetry. The potential for increased resource consumption (RAM, flash, battery life) is a trade-off that needs careful consideration for production deployments, especially in battery-powered devices. However, the article's focus on minimizing impact through on-device aggregation and schema optimization mitigates this significantly. The article implicitly highlights the barrier to entry for those not deeply familiar with embedded development or OTel's intricacies.

Key Points

• OpenTelemetry (OTel) can be effectively instrumented on resource-constrained microcontrollers like the ESP32.
• Pragmatic adaptation of OTLP protobuf schema using nanopb by flattening and preserving upstream field numbers is crucial for embedded use.
• Dual-core architecture and snapshot pattern are essential for separating real-time control loops from blocking network I/O, ensuring system stability.
• On-device computation of derived metrics (e.g., puck resistance variance) reduces telemetry volume and complexity.
• Concatenating protobuf messages of the same type allows for high-resolution data streaming without exceeding message size limits.
• Careful management of shared state, task stack sizes, and clock synchronization is vital for reliable embedded telemetry.
• ClickStack (OTel Collector, ClickHouse, HyperDX) provides an OTel-native, schema-agnostic platform suitable for diverse telemetry sources, including microcontrollers.

---

📖 Source: Instrumenting my espresso machine with OpenTelemetry