Serilog & OpenTelemetry in .NET: Build Business Advantage

This post is a continuation of the Serilog series (previous part: Serilog: Enrichers for Better Context).

In this part, we will:

• Explore the extended application workflow introduced in the previous part
• View logs and telemetry in the Grafana Drilldown dashboard
• Talk about the upsides of using OpenTelemetry for observability

Let's get started! 🚀

Prerequisites

• This episode starts from branch 006-opentelemetry. You will find it here.
• This post introduces an enormous amount of new code. Try to focus on the business workflow and architecture, not on every implementation detail. Feel free to skim parts of the code when needed.

Note: The goal of this post is to showcase the benefits of using logging (Serilog) and observability (OpenTelemetry) and not to be a step-by-step tutorial. Also note that the code is not production-ready and is only meant for demonstration purposes. It is not fully optimized for performance, security, or maintainability (it was written mostly by GPT-5.4 with my review and guidance). If you have any questions about the code, concepts, or implementation details, feel free to ask in the comments or reach out via LinkedIn or GitHub.

Quick recap: SerilogDemo in one minute

• Scalable .NET SerilogDemo API behind NGINX
• Single-instance supporting APIs: PaymentGateway API, Notification API, Fulfillment API
• Shared database (PostgreSQL)
• Each API streams its logs and telemetry to the OpenTelemetry Collector, which forwards them to Grafana Loki, Prometheus, and Tempo
• Grafana Drilldown dashboard for logs and telemetry visualization
• A load-test scenario is prepared with k6. It simulates realistic user behavior and generates load on the API, which allows us to see how the system performs under stress and how logs and telemetry help us understand its behavior.

Business workflow at a glance

Before diving into observability details, here is the end-to-end business flow from cart and checkout through payment outcomes, fulfillment, order tracking, and the Grafana view of the whole journey.

Add products to the cart

The shopper browses the store and builds an order with the items they want to buy.

Choose delivery and payment

At checkout, the shopper confirms the basket, picks a delivery option, selects a payment method, and chooses a demo payment scenario (one of: Success, Decline, SlowSuccess, Timeout).

Place the order

The shopper confirms the purchase and the order request is sent to the checkout flow. This is the moment the order leaves the cart and enters checkout processing.

Payment is processed

The payment step decides whether the order can continue immediately, continue after a delay, or stop before fulfillment begins.

• Success: the order continues right away.
• SlowSuccess: adds a noticeable wait but still finishes successfully.
• Decline: stops the order before fulfillment starts.
• Timeout: stops the flow because the payment takes too long.

Successful payment

When payment succeeds, the shopper gets a confirmation. The warehouse receives a new order to prepare. Notification is sent (email or/and SMS) to the shopper.

• The demo has fake notification sender - it logs the notification instead of sending it.

Order shipment

Warehouse staff collect the items, pack the order, and move it toward shipment.

Order lookup

The shopper can look up the order to see its current status and the details of what was purchased.

Observability

Grafana brings the whole business journey together so you can inspect what happened through logs, traces, and metrics in one place.

Run load test

The load test is designed to simulate realistic user behavior and exercise the full business workflow. It is implemented with k6 and can be found in the load-test folder. To run it, execute the following command in the terminal:

It is divided into three files:

1. load-test.js - ecommerce users hitting the main API through Nginx with browse, basket, checkout, and order-status polling behavior.
2. load-test-warehouse.js - warehouse workers hitting the Fulfillment API directly to collect, pack, ship, and inspect fulfillment backlog.
3. load-test-restock.js - a replenishment worker that watches warehouse availability and tops stock back up through the existing warehouse API when items fall below a configurable low-water mark. Without this script load test would eventually grind to a halt as inventory is depleted.

Logs drilldown in Grafana

Open Grafana in the browser and navigate to the Drilldown dashboard (Menu > Drilldown > Logs). You should see the following APIs:

• SerilogDemo API
• PaymentGateway API
• Notification API
• Fulfillment API

Each of them has its own panel with logs and telemetry. In the filters section, you can select a specific API for further details. Here is what it looks like for the SerilogDemo API:

Log details

Any log entry contains trace and span IDs, which allow us to correlate logs with traces and metrics. This is crucial for understanding the context of each log entry and how it fits into the overall workflow.

For a specific log entry, you can click on it to see more details, including the log line, structured metadata, and indexed metadata. This can help you understand what happened at that point in time and why.

Now copy traceId, OrderId, or any other relevant metadata and use it to filter logs in the Logs panel. This allows you to see all logs related to a specific business process (orderId/orderNumber) or trace, which can be incredibly useful for debugging and understanding the flow of a request through the system.

Traces

In the Drilldown > Traces panel, you can also see traces related to specific traceId. This allows you to see the entire flow of a request through the system, including all spans and their metadata. This is incredibly useful for understanding the performance and behavior of the system under load.

Let me give you an example. A client calls your service desk with the following problem: "I cannot check out my order; the payment fails." You can ask for the order number and then use it to find all logs and traces related to that order. This way you can see exactly what happened during the checkout process, where it failed, and why. You can see whether the payment gateway was called, whether any errors were returned, and what the response looked like. This can help you quickly identify the root cause of the problem and provide a solution to the customer (let's say, "It seems that the payment gateway is currently experiencing issues. We are working on it and will notify you once it's resolved. In the meantime, you can try a different payment method or place the order later.").

Here is what the trace looks like in Grafana:

Benefits of using OpenTelemetry for observability

Logs empower you to understand what lines of code were executed and what was the context of those executions. Modern web applications are mostly distributed and often involve multiple services, APIs, and databases. Logs alone can be overwhelming and insufficient to understand the full picture of what is happening in the system. This is where OpenTelemetry comes in.

OpenTelemetry provides a standardized way to collect and correlate logs, metrics, and traces. This allows you to have a holistic view of your system's behavior and performance. With OpenTelemetry, you can easily correlate logs with traces and metrics, which can help you understand the context of each log entry and how it fits into the overall workflow. For example, if you see an error log, you can quickly find the corresponding trace and see the full flow of the request that led to that error, including all the spans and their metadata. This can help you quickly identify the root cause of the problem and understand how it affects the overall system.

Conclusion

You have probably noticed in daily life that visualizing data is often more effective than reading raw data. For example, when you look at a map, you can quickly understand geography and the relationships between different locations, which would be much harder if you were staring at a list of coordinates. Similarly, in software development, having a visual representation of your logs, metrics, and traces can help you quickly understand the behavior and performance of your system, identify patterns and anomalies, and make informed decisions about how to improve it.

By using OpenTelemetry and a visualization tool like Grafana, you can gain deeper insights into your system's behavior and performance, which can help you deliver better software and provide a better experience for your users.

Talking about visualization: in the next episode of the Serilog series, we will introduce a user interface for our demo project, which will allow us to interact with the system in a more intuitive way. Stay tuned!

In the meantime, happy coding! 🚀