OpenTelemetry/Research/AI/25/07/OTel Logs in FastAPI Context

OpenTelemetry/Logs in FastAPI - json Logging Strategy
- Part 1: OTel-Compatible JSON Logging Strategy
  - OpenTelemetry/Logs API Maturity
    - OpenTelemetry’s Python logging support is still not fully mature as of 2025. The logs API/SDK for Python is in a “development” stage (not yet stable) [1]. In practice, this means the native OTel logging pipeline is still evolving – it’s not as battle-tested or feature-rich as the tracing API. While you can use OTel’s logging SDK to emit logs (even exporting them via OTLP), it may require extra setup and currently lacks some conveniences (like built-in JSON formatting or a rich logger interface). In short, OpenTelemetry’s native logging is not yet a drop-in replacement for traditional logging. You would need to configure it to output JSON in a Splunk-friendly way (or send logs to a collector), which can be non-trivial. Given this, many teams continue to rely on established Python logging libraries to produce structured JSON logs, even while using OTel for tracing.
  - Using structlog vs. Native OTel Logging
    - Libraries like structlog remain a best-practice choice for structured JSON logging in Python – even with OpenTelemetry in the mix. Structlog is purpose-built for generating structured log events (e.g. JSON) and integrates easily with existing logging frameworks [2] [3]. By contrast, OpenTelemetry’s logging instrumentation doesn’t itself format logs as JSON (its default formatter produces a text line with trace/span IDs). Achieving “Splunk-ingestible” JSON via pure OTel would mean either using a custom logging formatter or exporting logs to a collector that serializes them. In practice, it’s often simpler to use structlog (or the standard logging module with a JSON formatter) to emit JSON to stdout, and just enrich those logs with OTel trace context. Structlog can output clean JSON to stdout with minimal fuss, which aligns well with ECS+Splunk setups (ECS will collect the JSON logs from stdout). Its design is lightweight (just one Python package) and it supports asynchronous frameworks, making it suitable for a FastAPI service [3]. Until OTel’s logging API becomes more mature, structlog (or similar structured logging libraries) is a pragmatic choice for production-ready JSON logs.
  - Combining OpenTelemetry and structlog
    - It’s important to understand how OpenTelemetry and structlog intersect so you don’t double-instrument or conflict. OpenTelemetry’s logging instrumentation works by injecting trace IDs into standard Python log records automatically [4]. It does this by registering a custom LogRecord factory that adds fields like otelTraceID and otelSpanID to each record. If you were to use the built-in logging module, you could enable this via LoggingInstrumentor (setting OTEL_PYTHON_LOG_CORRELATION=true), then format logs to include those fields.
    - When using structlog, however, you typically bypass direct use of LogRecord formatting – instead, structlog lets you build log entries as dictionaries and render them (as JSON, etc.). In this case, you wouldn’t rely on OTel’s LogRecord injection; instead you’d pull the OTel trace context into structlog yourself. Structlog provides hooks (processors) to add context to each log event. For example, you can write a processor that grabs the current OTel span and inserts its trace_id and span_id into the event dict [5]. This way, every JSON log produced by structlog will contain the OTel trace identifiers for correlation. This does not create duplicate instrumentation – it’s simply using OTel’s trace API to enrich logs. You should avoid also enabling OTel’s logging injection in this scenario (don’t double-inject the same IDs), and let structlog handle the output format.
  - Injecting Trace IDs into Log Output
    - Injecting OTel trace IDs cleanly into your logs is crucial for correlating logs with traces (e.g. in Splunk or any analysis tool). With structlog, it’s straightforward: you can add a processor function that uses opentelemetry.trace.get_current_span() to get the active span, then read its context. For example:
      - python
        from opentelemetry import trace def add_trace_info(_, __, event_dict): span = trace.get_current_span() if span: ctx = span.get_span_context() event_dict["trace_id"] = format(ctx.trace_id, "032x") event_dict["span_id"] = format(ctx.span_id, "016x") return event_dict structlog.configure(processors=[add_trace_info, structlog.processors.JSONRenderer()])
    - This sample (adapted from structlog’s docs) binds the current span’s IDs into the log event [5]. If no span is active, it can simply skip or set a null. The result is that each log line (as JSON) will include a "trace_id" and "span_id" field with the OTel IDs, which Splunk can use for correlation. There’s no heavy “double instrumentation” here – you’re not creating spans or logs twice, just propagating context. Alternatively, if you were using standard logging, you could rely on OTel’s instrumentation which injects otelTraceID/otelSpanID fields into the LogRecord automatically [6] [7]. You’d then use a logging formatter (or python-json-logger) to output those fields in JSON. Both approaches achieve the same goal: trace IDs in your logs. Structlog just gives you more direct control over formatting and is very convenient for JSON output.
  - Pitfalls of Using structlog with OTel
    - When combining structlog with OpenTelemetry, watch out for a few pitfalls:
      - Double Logging or Handlers: If you wrap structlog over the standard library logging (via structlog.stdlib), be careful with OTel’s LoggingHandler or any root handlers. For instance, the OTel SDK offers a LoggingHandler that can capture logs and send them via OTLP [8] [9]. If you attach that to the root logger and also have structlog outputting to stdout, you might end up sending the same log to two places. Decide if you want to export logs via OTel or just stdout – a minimal ECS setup usually just needs stdout JSON logs (no double export).
      - Context Propagation: Ensure that the OTel trace context is properly available where you log. In FastAPI (async), the OTel span context will typically be propagated in contextvars, and structlog’s processor will pick it up as shown. This is generally reliable, but if you create new threads or tasks outside request context, you might lose the span context (then trace_id will be empty). Using OTel’s AsyncInstrumentor (if needed) and structlog’s contextvars integration can help maintain context in async code.
      - Performance: There is a slight overhead to injecting trace IDs on every log call (calling get_current_span() each time). In practice this is minor, but you should use structlog’s async-friendly contextvars (which it does by default in new versions) rather than thread-local context to avoid issues in concurrency. Structlog is designed to be efficient and non-blocking [10], so it works well in high-throughput async services. Just avoid overly complex log processors or heavy string formatting in hot code paths.
      - Formatting Consistency: If you use structlog for JSON and perhaps use OTel’s logging auto-instrumentation for any third-party libraries, make sure the outputs align. For example, if some library is using stdlib logging with OTel injection, those logs might appear in a different format (not JSON) unless you configure a JSON formatter for them too. One way to unify this is to configure structlog as the default logger for everything (structlog can integrate with the stdlib logging so that logging.getLogger() uses structlog under the hood). This ensures all logs go through the same processors and end up in JSON. The key is to avoid mixed formatting in your output.
    - Overall, these pitfalls are manageable. Many teams successfully use structlog alongside OTel tracing – the main caveat is to configure things in a clear way (and document that developers should log via the structlog logger, not via raw logging calls, to get JSON + trace context).
  - Recommended Logging Approach
    - For a FastAPI service template on ECS, the preferred approach is to use structured logging with context, rather than relying solely on OTel’s experimental log API. Concretely, we recommend using structlog to produce JSON-formatted logs to stdout, enriched with OTel trace IDs. This approach meets all your criteria:
      - Minimal Dependencies: Structlog is a small library with no heavy requirements. It avoids pulling in a full logging backend. The alternative (OTel logging SDK + an exporter) isn’t really lighter – you’d still need the OTel SDK and likely a JSON formatter. Since you’ll likely use structlog for structured logging anyway, it’s an acceptable dependency for the template.
      - Async-Safe: Structlog supports asynchronous frameworks and even provides contextvars-based context storage [3]. This means in an async FastAPI app, each request’s context (e.g. the OTel span) can be bound and logged without conflicts between concurrent requests. The example processor above uses trace.get_current_span(), which internally uses contextvars to get the span – so it works properly in async contexts. Many FastAPI projects use structlog for exactly this reason (to preserve request IDs or trace IDs in logs concurrently).
      - ECS stdout → Splunk: By logging JSON to stdout, you align with ECS Fargate’s default logging (which sends container stdout to CloudWatch or directly to a forwarder). Splunk can ingest JSON logs easily – either via a Fluent Bit/FireLens sidecar or via CloudWatch subscription – and parse the fields. As long as your JSON is line-delimited and not too nested, Splunk will happily index fields like trace_id. The recommended format is to include all relevant metadata (service name, environment, request IDs, etc.) as fields in the JSON. Structlog makes this easy: you can bind persistent context (e.g. service name) and it will appear in all logs. The result is a clean JSON log line per event, which is ideal for ECS and Splunk.
      - Trace Correlation: Including trace_id (and span ID if needed) in logs means that in Splunk you can search for all logs from a given trace. This is crucial for debugging. If you use the OTel Trace ID (a 32-character hex), ensure you’re consistent in formatting (e.g. hex string with no dashes). The example above uses format(ctx.trace_id, "032x") which produces the 32-char hex string. That will match the trace IDs you see in trace systems or can be used to join with trace data if needed. In the future, OpenTelemetry might offer a more unified way to handle logs, but until it’s stable, this approach is simple, robust, and proven.
    - To summarize, structlog + OpenTelemetry is the recommended combo: structlog for output and structure, OpenTelemetry for tracing and context. This gives you production-ready JSON logs that Splunk can ingest, with minimal overhead and clear correlation between logs and traces.
- Part 2: Local Log Viewing Tools for Devs
  - Developers often need a way to view and search logs locally (during development) without relying on Splunk’s cloud environment. Below is a shortlist of lightweight tools and approaches that can ingest OpenTelemetry-flavored logs (structured JSON or OTLP) and provide search/filtering and visualization in a local setup. All of these can be integrated into your Docker Compose dev stack. We’ll outline each option with pros, cons, and setup notes:
  - 1. SigNoz (OpenSource Observability Platform)
    - SigNoz is an open-source APM platform that includes log management. It uses OpenTelemetry under the hood and provides a unified UI for logs, metrics, and traces. In a dev setup, you can run SigNoz via a one-command Docker Compose deployment [11]. It spins up an OTel Collector, database (ClickHouse), and a web UI. Developers can push their logs to SigNoz either by using the OTel Collector exporter or simply stdout (SigNoz’s collector can tail Docker stdout). Once set up, you get an “intuitive logs explorer” in the browser for searching and filtering logs in real time [12].
    - Pros: SigNoz offers a unified observability dashboard – you can see your service’s traces and logs correlated in one place [11]. The log search is powerful, with filtering and a query builder similar to Splunk’s search (it supports full-text search and field-based queries) [13]. It’s built on open standards (OpenTelemetry and ClickHouse) so it works smoothly with your OTel instrumentation (no custom agents needed). Setup is straightforward (pull the Docker images and go), and it’s free/self-hosted. This makes it great for simulating a production-like observability experience locally.
    - Cons: SigNoz’s all-in-one nature means it’s heavier than a dedicated log-only tool. It will run multiple containers (collector, Kafka/Zookeeper, ClickHouse, frontend, etc.), which can consume significant resources – something to keep in mind for each developer’s laptop. While setup is one-command, it might be overkill if you only need log viewing (and not traces/metrics) in dev. Also, the UI, while good, has a learning curve if devs are not familiar with observability dashboards (though it’s quite user-friendly for anyone used to Splunk or Kibana).
    - Setup Difficulty: Moderate. SigNoz provides Docker Compose files that you can include in your dev environment. You’d configure your app’s OTel exporter/endpoint to point to the SigNoz OTel Collector (or just let the collector scrape stdout). Following SigNoz’s docs, most get it running in a few minutes. The main “difficulty” is resource usage; you’ll want to ensure the Docker dev environment has enough memory for it. If your team is okay with that, SigNoz is a very convenient one-stop solution for local log analysis.
  - 2. Grafana Loki (with Grafana UI)
    - Grafana Loki is a popular lightweight log aggregation system, often used as an alternative to the ELK stack. Loki is designed to be efficient by indexing only metadata (labels) and not the full log text [14]. In a dev context, you can run a Loki container plus a Grafana container (for the UI) to search your logs. Logs can be sent to Loki via the OTel Collector (there’s an exporter for Loki) or via Grafana’s own agent/Promtail. Grafana will then allow you to query logs using LogQL, which is a querying language similar to PromQL (you filter by labels and can grep within log lines) [15].
    - Pros: The stack is fairly lightweight – Loki is a single Go binary (small memory footprint compared to ElasticSearch), and Grafana is also quite efficient. This is a good choice if you already use Grafana for metrics, as you can extend it to logs in the same UI. It provides real-time log tailing as well as searching, and you can even set up basic dashboards or alerts on log data. Integration is straightforward: the OTel Collector can treat stdout logs as input and push to Loki, or you run a Promtail sidecar to ship logs. Loki’s approach of not indexing full text keeps it fast and resource-friendly for dev use.
    - Cons: Loki’s search paradigm is a bit different from Splunk’s. You typically must define labels (metadata tags) for your logs (e.g. app name, environment, etc.) and query primarily by those. Full-text search is possible but only after filtering by labels, which can be a slight adjustment for developers. In a small dev setup, you might label all logs with just the service name, etc., and then use text search on messages. The Grafana UI, while powerful, might not be as specialized for log analysis as, say, Kibana or Splunk (though Grafana’s Explore view for Loki is quite good). Another consideration: you need to manage two containers and configuration. It’s not “one-click” like SigNoz; you’ll have to configure Grafana to use Loki as a data source (which is usually automatic in Grafana’s Docker if you provide GF_LOGGING env configs or a config file).
    - Setup Difficulty: Moderate-Easy. There are readily available Docker Compose snippets for Loki+Grafana. You’d include those services in your compose and configure either the OTel Collector or Promtail. If using the OTel Collector in your dev stack, add a Loki exporter and point it at your Loki container (Loki exposes an HTTP endpoint for logs). Grafana then needs to know about the Loki datasource – Grafana’s provisioning can automate this. Overall, within an hour you can have this running. It’s a bit more manual than SigNoz, but also more modular (you can turn it off when not needed, etc.). This solution is ideal if your team is comfortable with Grafana or wants a lighter-weight log viewer that still supports decent searching.
  - 3. Vector + Local Log Storage (e.g. Quickwit or ZincObserve)
    - Vector is a high-performance log collector/forwarder (by Datadog, open source) that can be very useful in a dev environment. On its own, Vector is not a UI – it’s a pipeline tool – but it can aggregate logs from all your services and then send them to a local storage/search tool. This approach gives you flexibility to mix and match components. For example, you can run Vector to capture stdout from your FastAPI (and other services in Compose) and feed those logs into an open-source storage with a UI, such as ZincObserve or Quickwit:
      - ZincObserve (by ZincSearch) is a lightweight log index and web UI you can run locally. One StackOverflow answer notes that ZincObserve is “light weight that you run on your laptop” and you just need to forward the logs using Fluent Bit or Vector [16]. Essentially, ZincObserve aims to be a minimal Splunk/ELK alternative – you run its server (single binary) and send logs via Elasticsearch API or Vector’s HTTP sink. It provides a web interface to search and filter logs. Setup would involve running the ZincObserve container and configuring Vector with an http sink to send logs there.
      - Quickwit is another modern open-source log management tool. It natively understands OpenTelemetry log data and even comes with a built-in UI for search. For instance, Quickwit can create an index for OTel logs, and you can send logs to it via HTTP. In a demo, Vector was used to transform logs and POST them to Quickwit’s OTLP ingest endpoint [17] [18]. After that, you can open Quickwit’s web UI at http://localhost:7280/ui/search to query the logs (by fields, text, etc.) [19]. This means Quickwit + Vector can function as a mini Splunk: Vector collects and normalizes logs, Quickwit indexes them and provides search with a web interface.
    - Pros: Using Vector in your dev stack gives you a powerful log pipeline. You can easily add filtering, parsing, or enrichment in Vector (it has a rich remap language) if needed before storing the logs. It’s also very lightweight (written in Rust, optimized for performance). Pairing it with a focused log store like Quickwit or ZincObserve can be more resource-efficient than running a full ELK stack. You also get to choose your storage: Quickwit, for example, can store indices on disk and is optimized for search speed. This modular approach can be tailored to your needs – e.g., if you only care about recent logs, you don’t even need long-term storage, you could have Vector just pretty-print to console with filtering. In fact, Vector even has a console sink that can format JSON logs in a human-friendly way (with coloring, etc.), which could be used for real-time debugging if full-text search is not needed at every moment.
    - Cons: The trade-off for flexibility is complexity. You’ll be managing multiple moving parts: e.g., Vector + a storage/UI component (and possibly configuring schemas or indices). If something goes wrong (no logs showing up), there are more places to check (is Vector output configured correctly? Is the log UI up and indexing?). The UIs of tools like Quickwit or ZincObserve, while improving, might not be as polished as Grafana or SigNoz. Quickwit’s UI, for instance, is basic but functional – it may not support fancy visualizations, mostly search queries. ZincObserve is newer and might not be as stable as more established projects. Another con: documentation and community for these newer tools are smaller, so solving issues might require more effort.
    - Setup Difficulty: Moderate. Getting Vector running is easy (single binary or container) – you’ll configure a source (e.g. Docker logs, or read from stdout/socket) and a sink (HTTP to Quickwit, or Elasticsearch-compatible output to Zinc). Quickwit or ZincObserve themselves are relatively easy to run (docker pull and go). The key is configuring them to talk to each other. Quickwit, for example, has an OTLP ingest endpoint by default for logs, so you’d point Vector’s sink there and it works out-of-the-box for OTel JSON logs [20]. ZincObserve might accept Elasticsearch Bulk API or its own API – you’d have to configure that accordingly. Overall, you might spend a bit of time writing a vector.toml config, but this is a one-time setup you can commit to your repo. Once done, developers just run docker-compose up and get the pipeline. This option is good if your team enjoys having fine-grained control or already uses Vector/FluentBit in prod and wants to mimic that pipeline in dev.
  - 4. Seq (Seq Logs Viewer) – Bonus Option
    - If you’re open to non-open-source but free-for-dev tools, Seq by Datalust is worth mentioning. Seq is a log aggregation server with a very user-friendly web interface for viewing and querying logs in real-time. It’s especially popular in the .NET community but works with any JSON logs. You can run Seq locally as a single Docker container, and it has a generous free tier (e.g. it can handle a certain number of events per day which is usually plenty for development). Developers can send logs to Seq via many methods: it supports ingestion over HTTP, GELF, and it also has an OTLP endpoint. In fact, the OpenTelemetry Python SDK can export logs to Seq via OTLP – Seq provides an endpoint at /ingest/otlp/v1/logs for this [8] [9]. Alternatively, you could configure a simple HTTP sink in Vector or use Seq’s own Python client (seqlog) to push logs.
    - Pros: Seq probably has one of the smoothest UI experiences for developers. The log search is powerful (supports filtering by fields, full-text, time ranges) and the interface is real-time – you can watch logs tail live and click on fields/values to filter down. It’s designed for debugging scenarios. Setting it up is extremely simple (one container, and you visit localhost:5341 in your browser). It also can correlate with traces if you send OTLP spans to it (though it’s primarily a log tool). Since Seq can ingest structured data, your JSON log fields (including trace_ids) become first-class searchable fields. The free version is usually sufficient for personal or small-team dev use.
    - Cons: Seq is proprietary (though free for limited use), so not everyone is comfortable adding it to a open source stack. It’s log-only (no metrics/traces in the free self-hosted version, aside from basic trace support via OTLP). Also, if you have an entirely open-source policy, this might not fit. Another minor con is that you’ll likely use a slightly different pipeline to send logs to Seq (like the OTel LoggingHandler or seqlog library), which is extra config, though not too difficult.
    - Setup Difficulty: Easy. It’s arguably the easiest of all options – run the container and configure a single environment variable or two for your app’s logging. For example, you could install opentelemetry-exporter-otlp in your app and set the endpoint to the Seq container (as shown in Seq’s docs) [21] [8]. Or use Vector to forward logs to Seq’s HTTP ingestor. Seq’s web UI will show incoming logs immediately with zero additional config. It’s very much plug-and-play, which is a big plus for quick troubleshooting.
  - In summary, for local log viewing, you have a spectrum of choices. If you want a full “Splunk-like” experience locally, SigNoz is a great all-in-one solution (and brings the added benefit of trace correlation). If you prefer a lighterweight approach, Grafana Loki with Grafana UI is a solid, modular choice, especially if you’re already using Grafana. For those who like to tinker and customize, using Vector with a backend like Quickwit or ZincObserve can be rewarding and efficient. And as a convenience, tools like Seq can provide a polished log viewing experience with minimal effort (at the cost of not being OSS). All these can integrate with your Docker Compose setup – it’s a matter of balancing convenience vs. resource footprint. You might even use one tool during active development (e.g. Seq or Grafana for quick feedback) and another to simulate prod observability (e.g. SigNoz) when doing more comprehensive testing. The good news is that OpenTelemetry emits standard data, so you can mix and match these solutions as needed without changing your application code – just swap out where the logs go. Each option above has been used successfully by development teams to make debugging easier without relying on the production log infrastructure.
- Footnotes

2 Linked References

Hierarchy

OpenTelemetryResearchAI2507

OpenTelemetry/Research/AI/25/07/OTel Logs in FastAPI Context

OpenTelemetry/Logs in FastAPI - json Logging Strategy

Part 1: OTel-Compatible JSON Logging Strategy

OpenTelemetry/Logs API Maturity

Using structlog vs. Native OTel Logging

Combining OpenTelemetry and structlog

Injecting Trace IDs into Log Output

Pitfalls of Using structlog with OTel

Recommended Logging Approach

Part 2: Local Log Viewing Tools for Devs

1. SigNoz (OpenSource Observability Platform)

2. Grafana Loki (with Grafana UI)

3. Vector + Local Log Storage (e.g. Quickwit or ZincObserve)

4. Seq (Seq Logs Viewer) – Bonus Option

Footnotes

2 Linked References

Hierarchy

Unlinked References