This is a 2 part-series post.
- Part 1: Key concepts that you should know when using OpenTelemetry Metrics with .NET. If you want to read it, click here.
- Part 2: A practical example about how to add OpenTelemetry Metrics on a real life .NET app and how to visualize those metrics using Prometheus and Grafana.
Just show me the code!
As always, if you don’t care about the post I have uploaded the source code on my Github.
In part 1 we talked about some OpenTelemetry Metrics key concepts, now it’s time to focus on instrumenting a real application.
This is what we are going to build.

- The BookStore WebAPI will generate some business metrics and use the OTLP exporter package (
OpenTelemetry.Exporter.OpenTelemetryProtocol
) to send the metric data to the OpenTelemetry Collector. - Prometheus will obtain the metric data from the OpenTelemetry Collector.
- We will have a Grafana dashboard to visualize the metrics emitted by the BookStore WebApi.
Application
The application we’re going to instrument using OpenTelemetry Metrics is a BookStore API built using .NET6.
The API can do the following actions:
- Get, add, update and delete book categories.
- Get, add, update and delete books.
- Get, add, update and delete inventory.
- Get, add and update orders.
For a better understanding, here’s how the database diagram looks like:

OpenTelemetry Metrics
The first step before writing any code is to decide what we want to measure and what kind of instruments are we going to use.
BookStore API custom metrics
The following business metrics will be instrumented directly on the application using the Metrics API.
BooksAddedCounter
is aCounter
that counts books added to the store.BooksDeletedCounter
is aCounter
that counts books deleted from the store.BooksUpdatedCounter
is aCounter
that counts books updated.TotalBooksGauge
is anObservableGauge
that contains the total number of books that the store has at any given time.CategoriesAddedCounter
is aCounter
that counts book categories added to the store.CategoriesDeletedCounter
is aCounter
that counts book categories deleted from the store.CategoriesUpdatedCounter
is aCounter
that counts book categories updated.TotalCategoriesGauge
is anObservableGauge
that contains the total number of book categories that the store has at any given timeOrdersPriceHistogram
is aHistogram
that records the price distribution of the orders.NumberOfBooksPerOrderHistogram
is aHistogram
that records the number of books distribution per order.OrdersCanceledCounter
is anObservableCounter
that counts the total number of orders cancelled.TotalOrdersCounter
is aCounter
that counts the total number of orders that the store has received.
Http requests metrics
Those metrics are generated by the OpenTelemetry.Instrumentation.AspNetCore
NuGet package.
This is an instrumentation library, which instruments .NET and creates metrics about incoming web requests.
There is no need to instrument anything on the application. To start using the OpenTelemetry.Instrumentation.AspNetCore
package you only need to add the AddAspNetCoreInstrumentation()
extension method when setting up the .NET OpenTelemetry MeterProvider
component. Here’s an example:
using Microsoft.Extensions.DependencyInjection;
using OpenTelemetry.Trace;
public void ConfigureServices(IServiceCollection services)
{
services.AddOpenTelemetryTracing((builder) => builder
.AddAspNetCoreInstrumentation()
);
}
System.Runtime performance metrics
Those metrics are generated by the OpenTelemetry.Instrumentation.Runtime
NuGet package. This is an instrumentation library, which instruments .NET Runtime and collects runtime performance metrics.
There is no need to instrument anything on the application. To start using the OpenTelemetry.Instrumentation.Runtime
package you only need to add the AddRuntimeInstrumentation()
extension method when setting up the .NET OpenTelemetry MeterProvider
component. Here’s an example:
using Microsoft.Extensions.DependencyInjection;
using OpenTelemetry.Trace;
public void ConfigureServices(IServiceCollection services)
{
services.AddOpenTelemetryTracing((builder) => builder
.AddRuntimeInstrumentation()
);
}
The OpenTelemetry.Instrumentation.Runtime
package collects telemetry about the following System.Runtime
counters:
process.runtime.dotnet.gc.collections.count
: Number of garbage collections that have occurred since process start.process.runtime.dotnet.gc.allocations.size
: Count of bytes allocated on the managed GC heap since the process startprocess.runtime.dotnet.gc.committed_memory.size
: The amount of committed virtual memory for the managed GC heap, as observed during the latest garbage collection.process.runtime.dotnet.gc.heap.size
: The heap size (including fragmentation), as observed during the latest garbage collection.process.runtime.dotnet.gc.heap.fragmentation.size
: The heap fragmentation, as observed during the latest garbage collection.process.runtime.dotnet.jit.il_compiled.size
: Count of bytes of intermediate language that have been compiled since the process start.process.runtime.dotnet.jit.methods_compiled.count
: The number of times the JIT compiler compiled a method since the process start.process.runtime.dotnet.jit.compilation_time
: The amount of time the JIT compiler has spent compiling methods since the process start.process.runtime.dotnet.monitor.lock_contention.count
: The number of times there was contention when trying to acquire a monitor lock since the process start.process.runtime.dotnet.thread_pool.threads.count
: The number of thread pool threads that currently exist.process.runtime.dotnet.thread_pool.completed_items.count
: The number of work items that have been processed by the thread pool since the process start.process.runtime.dotnet.thread_pool.queue.length
: The number of work items that are currently queued to be processed by the thread pool.process.runtime.dotnet.timer.count
: The number of timer instances that are currently active.process.runtime.dotnet.assemblies.count
: The number of .NET assemblies that are currently loaded.process.runtime.dotnet.exceptions.count
: Count of exceptions that have been thrown in managed code, since the observation started.
Some of the GC related metrics will be unavailable until at least one garbage collection has occurred.
OpenTelemetry .NET Client
To get started with OpenTelemetry Metrics we’re going to need the following packages.
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9.6" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.6" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.0.0" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.3.1" />
- The OpenTelemetry.Extensions.Hosting package contains some extensions that allows us to configure the MeterProvider.
- The OpenTelemetry.Instrumentation.* packages are instrumentation libraries. These packages are instrumenting common libraries/functionalities/classes so we don’t have to do all the heavy lifting by ourselves. In our application we’re using the following ones:
- The OpenTelemetry.Instrumentation.AspNetCore package generats and collects metrics about incoming web requests.
- The OpenTelemetry.Instrumentation.Runtime package collects runtime performance metrics.
- The OpenTelemetry.Exporter.OpenTelemetryProtocol package allows us to export the metrics to the OpenTelemetry Collector using the OTLP protocol.
Add OpenTelemetry Metrics on the BookStore app
1 – Setup the MeterProvider
var meters = new OtelMetrics();
builder.Services.AddOpenTelemetryMetrics(opts => opts
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("BookStore.WebApi"))
.AddMeter(meters.MetricName)
.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.AddOtlpExporter(opts =>
{
opts.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]);
}));
Let’s review what we’re doing line by line.
AddOpenTelemetryMetrics()
OpenTelemetry Metrics works by using the MeterProvider
to create a Meter
and associating it with one or more Instruments
, each of which is used to create a series of Measurements
.
The MeterProvider
holds the configuration for metrics like Meter names, Readers or Views.
The MeterProvider
must be configured using the AddOpenTelemetryMetrics()
extension method.
SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("BookStore.WebApi"))
A Resource
is the immutable representation of the entity producing the telemetry. With the SetResourceBuilder
method we’re configuring the Resource
for the application.
The SetResourceBuilder
gives us the possibility to configure attributes like the service name or the application name amongst others.
Using the AddService("BookStore.Webapi")
method we can set the service name as an attribute of the metric data. For example, if we take a look at the OrdersCanceledCounter
instrument on Prometheus, we will see the following information:

AddMeter(meters.MetricName)
Any Instrument
that we create in our application needs to be associated with a Meter
. The AddMeter()
extension method configures OpenTelemetry to transmit all the metrics collected by this concrete Meter
.
As you’ll see later, in the BookStore app I have a single Meter
with multiple Instruments
on it, but you can also have multiple Meters
in a single application, in that case you’ll need to add multiple calls to the AddMeter()
method.
AddAspNetCoreInstrumentation()
This method comes from the OpenTelemetry.Instrumentation.AspNetCore
NuGet package, it instruments .NET and collects metrics and traces about incoming web requests.
AddRuntimeInstrumentation()
This method comes from the OpenTelemetry.Instrumentation.Runtime
NuGet package, it instruments .NET and collects metrics and collects runtime performance metrics.
AddOtlpExporter(opts =>
{
opts.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]);
}));
The AddOtlpExporter
method is used to configure the exporter that sends all the metric data to the OpenTelemetry Collector.
2 – Create the Meter and Instruments
After setting up the MeterProvider
, it’s time to create a Meter
and use it to create Instruments
.
There are a few ways to do that, but I’m going to show you how I tend to do it.
First of all, I’m going to create a Singleton
class which will contain:
- A meter.
- The instruments associated with the meter.
- A series of helper methods to record measurements with those instruments.
Here’s how the class looks like:
using System.Collections.Generic;
using System.Diagnostics.Metrics;
using System.Threading;
namespace BookStore.Infrastructure.Metrics
{
public class OtelMetrics
{
//Books meters
private Counter<int> BooksAddedCounter { get; }
private Counter<int> BooksDeletedCounter { get; }
private Counter<int> BooksUpdatedCounter { get; }
private ObservableGauge<int> TotalBooksGauge { get; }
private int _totalBooks = 0;
//Categories meters
private Counter<int> CategoriesAddedCounter { get; }
private Counter<int> CategoriesDeletedCounter { get; }
private Counter<int> CategoriesUpdatedCounter { get; }
private ObservableGauge<int> TotalCategoriesGauge { get; }
private int _totalCategories = 0;
//Order meters
private Histogram<double> OrdersPriceHistogram { get; }
private Histogram<int> NumberOfBooksPerOrderHistogram { get; }
private ObservableCounter<int> OrdersCanceledCounter { get; }
private int _ordersCanceled = 0;
private Counter<int> TotalOrdersCounter { get; }
public string MetricName { get; }
public OtelMetrics(string meterName = "BookStore")
{
var meter = new Meter(meterName);
MetricName = meterName;
BooksAddedCounter = meter.CreateCounter<int>("books-added", "Book");
BooksDeletedCounter = meter.CreateCounter<int>("books-deleted", "Book");
BooksUpdatedCounter = meter.CreateCounter<int>("books-updated", "Book");
TotalBooksGauge = meter.CreateObservableGauge<int>("total-books", () => new[] { new Measurement<int>(_totalBooks) });
CategoriesAddedCounter = meter.CreateCounter<int>("categories-added", "Category");
CategoriesDeletedCounter = meter.CreateCounter<int>("categories-deleted", "Category");
CategoriesUpdatedCounter = meter.CreateCounter<int>("categories-updated", "Category");
TotalCategoriesGauge = meter.CreateObservableGauge<int>("total-categories", () => _totalCategories);
OrdersPriceHistogram = meter.CreateHistogram<double>("orders-price", "Euros", "Price distribution of book orders");
NumberOfBooksPerOrderHistogram = meter.CreateHistogram<int>("orders-number-of-books", "Books", "Number of books per order");
OrdersCanceledCounter = meter.CreateObservableCounter<int>("orders-canceled", () => _ordersCanceled);
TotalOrdersCounter = meter.CreateCounter<int>("total-orders", "Orders");
}
//Books meters
public void AddBook() => BooksAddedCounter.Add(1);
public void DeleteBook() => BooksDeletedCounter.Add(1);
public void UpdateBook() => BooksUpdatedCounter.Add(1);
public void IncreaseTotalBooks() => _totalBooks++;
public void DecreaseTotalBooks() => _totalBooks--;
//Categories meters
public void AddCategory() => CategoriesAddedCounter.Add(1);
public void DeleteCategory() => CategoriesDeletedCounter.Add(1);
public void UpdateCategory() => CategoriesUpdatedCounter.Add(1);
public void IncreaseTotalCategories() => _totalCategories++;
public void DecreaseTotalCategories() => _totalCategories--;
//Orders meters
public void RecordOrderTotalPrice(double price) => OrdersPriceHistogram.Record(price);
public void RecordNumberOfBooks(int amount) => NumberOfBooksPerOrderHistogram.Record(amount);
public void IncreaseOrdersCanceled() => _ordersCanceled++;
public void IncreaseTotalOrders(string city) => TotalOrdersCounter.Add(1, KeyValuePair.Create<string, object>("City", city));
}
}
In the class constructor, we create the meter and then use it to create every necessary instrument. Also we create a series of public helper methods to record measurements.
- Why create all those helper methods?
To improve code readability, it is easier to understand what this line of code _meters.AddBook()
is doing, rather than this other one BooksAddedCounter.Add(1)
.
3 – Record measurements using the instruments
Now it’s time to use the instruments we have created in the previous section to start recording measurements.
You just need to inject the OtelMetrics
instance whenever we want to record a measurement and use any of the helper methods exposed on the OtelMetrics
class.
Record book metrics
- Every time a new book gets added into the database.
- Increase +1 the
BooksAddedCounter
instrument and increase +1 theTotalBooksGauge
instrument.
- Increase +1 the
- Every time a new book gets updated.
- Increase +1 the
BooksUpdatedCounter
instrument.
- Increase +1 the
- Every time a new book gets deleted from the database.
- Increase +1 the
BooksDeletedCounter
instrument and decrease -1 theTotalBooksGauge
instrument.
- Increase +1 the
The next snippet of code shows how to record those measurements every time a book gets added, updated or deleted from the database.
public class BookRepository : Repository<Book>, IBookRepository
{
private readonly OtelMetrics _meters;
public BookRepository(
BookStoreDbContext context,
OtelMetrics meters) : base(context)
{
_meters = meters;
}
public override async Task<List<Book>> GetAll()
{
return await Db.Books.Include(b => b.Category)
.OrderBy(b => b.Name)
.ToListAsync();
}
public override async Task<Book> GetById(int id)
{
return await Db.Books.Include(b => b.Category)
.Where(b => b.Id == id)
.FirstOrDefaultAsync();
}
public async Task<IEnumerable<Book>> GetBooksByCategory(int categoryId)
{
return await Search(b => b.CategoryId == categoryId);
}
public async Task<IEnumerable<Book>> SearchBookWithCategory(string searchedValue)
{
return await Db.Books.AsNoTracking()
.Include(b => b.Category)
.Where(b => b.Name.Contains(searchedValue) ||
b.Author.Contains(searchedValue) ||
b.Description.Contains(searchedValue) ||
b.Category.Name.Contains(searchedValue))
.ToListAsync();
}
public override async Task Add(Book entity)
{
await base.Add(entity);
_meters.AddBook();
_meters.IncreaseTotalBooks();
}
public override async Task Update(Book entity)
{
await base.Update(entity);
_meters.UpdateBook();
}
public override async Task Remove(Book entity)
{
await base.Remove(entity);
_meters.DeleteBook();
_meters.DecreaseTotalBooks();
}
}
Record book categories metrics
- Every time a new book category gets added into the database.
- Increase +1 the
CategoriesAddedCounter
instrument and increase +1 theTotalCategoriesGauge
instrument.
- Increase +1 the
- Every time a new book category gets updated.
- Increase +1 the
CategoriesUpdatedCounter
instrument.
- Increase +1 the
- Every time a new book category gets deleted from the database.
- Increase +1 the
CategoriesDeletedCounter
instrument and decrease -1 theTotalCategoriesGauge
instrument.
- Increase +1 the
The next snippet of code shows how to record those measurements every time a book category gets added, updated or deleted from the database.
public class CategoryRepository : Repository<Category>, ICategoryRepository
{
private readonly OtelMetrics _meters;
public CategoryRepository(BookStoreDbContext context,
OtelMetrics meters) : base(context)
{
_meters = meters;
}
public override async Task Add(Category entity)
{
await base.Add(entity);
_meters.AddCategory();
_meters.IncreaseTotalCategories();
}
public override async Task Update(Category entity)
{
await base.Update(entity);
_meters.UpdateCategory();
}
public override async Task Remove(Category entity)
{
await base.Remove(entity);
_meters.DeleteCategory();
_meters.DecreaseTotalCategories();
}
}
Record orders metrics
- Every time a new order gets added into the database.
- Increase +1 the
TotalOrdersCounter
instrument. - Record the order total price using the
OrdersPriceHistogram
instrument. - Record the amount of books in the order using the
NumberOfBooksPerOrderHistogram
instrument.
- Increase +1 the
- Every time a new order gets updated.
- Increase +1 the
OrdersCanceledCounter
instrument.
- Increase +1 the
The next snippet of code shows how to record those measurements every time an order gets added or updated.
public class OrderRepository : Repository<Order>, IOrderRepository
{
private readonly OtelMetrics _meters;
public OrderRepository(BookStoreDbContext context, OtelMetrics meters)
: base(context)
{
_meters = meters;
}
public override async Task<Order> GetById(int id)
{
return await Db.Orders
.Include(b => b.Books)
.FirstOrDefaultAsync(x => x.Id == id);
}
public override async Task<List<Order>> GetAll()
{
return await Db.Orders
.Include(b => b.Books)
.ToListAsync();
}
public override async Task Add(Order entity)
{
DbSet.Add(entity);
await base.SaveChanges();
_meters.RecordOrderTotalPrice(entity.TotalAmount);
_meters.RecordNumberOfBooks(entity.Books.Count);
_meters.IncreaseTotalOrders(entity.City);
}
public override async Task Update(Order entity)
{
await base.Update(entity);
_meters.IncreaseOrdersCanceled();
}
public async Task<List<Order>> GetOrdersByBookId(int bookId)
{
return await Db.Orders.AsNoTracking()
.Include(b => b.Books)
.Where(x => x.Books.Any(y => y.Id == bookId))
.ToListAsync();
}
}
As you can see from the snippets of code from this section recording a measurement it’s a really simple task, just invoke the instrument function wherever you want to record a measurement, and that’s it!
4 – Setup the Histogram bucket aggregation accordingly
A Histogram
is a graphical representation of the distribution of numerical data. It groups values into buckets and then counts how many values fall into each bucket.
When using a Histogram
instrument, it’s important to make sure the buckets are also configured properly. The bucket histogram aggregation default values are [ 0, 5, 10, 25, 50, 75, 100, 250, 500, 1000 ], and that’s not always ideal.
In the BookStore app we are using 2 Histograms
:
OrdersPriceHistogram
: Shows the price distribution of the orders.NumberOfBooksPerOrderHistogram
: Shows the number of books per order distribution.
For the NumberOfBooksPerOrderHistogram
makes no sense using the bucket aggregation default values because no one is going to make an order that contains 250, 500 or 1000 books. And the same could be said for the OrdersPriceHistogram
.
To customize the bucket aggregation values accordingly to every Histogram
we need to use a View
.
A View
in OpenTelemetry defines an aggregation, which takes a series of measurements and expresses them as a single metric value at that point in time.
To create a View
we can use the AddView
extension method from the MeterProvider
. Like this:
builder.Services.AddOpenTelemetryMetrics(opts => opts
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("BookStore.WebApi"))
.AddMeter(meters.MetricName)
.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.AddView(
instrumentName: "orders-price",
new ExplicitBucketHistogramConfiguration { Boundaries = new double[] { 15, 30, 45, 60, 75 } })
.AddView(
instrumentName: "orders-number-of-books",
new ExplicitBucketHistogramConfiguration { Boundaries = new double[] { 1, 2, 5 } })
.AddOtlpExporter(opts =>
{
opts.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]);
}));
OpenTelemetry Collector
The OpenTelemetry Collector consists of three components:
Receivers
: Can be push or pull based, is how data gets into the Collector.Processors
: Run on data between being received and being exported.Exporters
: Can be push or pull based, is how you send data to one or more backends/destinations.
In this case, the OpenTelemetry Collector receives metrics from the BookStore API via gRPC and exports them into Prometheus.
Here’s how the OTEL Collector config file looks like:
receivers:
otlp:
protocols:
grpc:
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
processors:
batch:
extensions:
health_check:
service:
extensions: [health_check]
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
Prometheus
Prometheus must be configured to scrape the OpenTelemetry Collector metrics endpoints.
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 5s
static_configs:
- targets: ['otel-collector:8889']
- targets: ['otel-collector:8888']
After setting up Prometheus, we are going to generate traffic on the BookStore API and afterwards access Prometheus to start analyzing the metrics that the app is sending us.
The business metrics from the BookStore API are all available in Prometheus.

If we take a peek at the “orders-price” Histogram
we can see that the bucket aggregation values are the correct ones that we defined in the MeterProvider
using the AddView
extension method.

The metrics about incoming requests generated by the OpenTelemetry.Instrumentation.AspNetCore
NuGet package are also available on Prometheus.

And the performance metrics generated by the OpenTelemetry.Instrumentation.Runtime
NuGet package are also being ingested by Prometheus.

Grafana
Having the metric data from the BookStore API in Prometheus is great at all, but we need to visualize them in a friendly manner, so let’s build a few Grafana dashboards.
Not going to explain how to build a Grafana dashboard, that’s out of the scope for this post.
I ended up building 3 dashboards.
Custom metrics dashboard
This dashboard uses the business metrics instrumented directly on the application using the Metrics API.

Here’s a closer look of how the “Orders” panel from the dashboard look like:

Http Requests dashboard
This dashboard uses the metrics generated by the OpenTelemetry.Instrumentation.AspNetCore
NuGet package.

Performance counters dashboard
This dashboard uses the metrics generated by the OpenTelemetry.Instrumentation.Runtime
NuGet package.

How to test the BookStore Api
If you want to take a look at the source code, you can go to my GitHub repository.
If you want to execute the BookStore API for yourselves, I have uploaded a docker-compose
file that starts up the app and also the external dependencies.
The external dependencies (Prometheus, MSSQL Server, Grafana and OpenTelemetry Collector) are already preconfigured so you don’t need to do any extra setup. Just run docker-compose
up and you’re good to go!
Here’s how the docker-compose
file looks like:
version: '3.8'
networks:
metrics:
name: bookstore-network
services:
mssql:
build:
context: ./scripts/sql
ports:
- "1433:1433"
environment:
SA_PASSWORD: "P@ssw0rd?"
ACCEPT_EULA: "Y"
networks:
- metrics
prometheus:
build:
context: ./scripts/prometheus
depends_on:
- app
ports:
- 9090:9090
networks:
- metrics
grafana:
build:
context: ./scripts/grafana
depends_on:
- prometheus
ports:
- 3000:3000
networks:
- metrics
otel-collector:
image: otel/opentelemetry-collector:0.60.0
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./scripts/otel-collector/otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "8888:8888"
- "8889:8889"
- "13133:13133"
- "4317:4317"
networks:
- metrics
app:
build:
context: ./
dockerfile: ./src/BookStore.WebApi/Dockerfile
depends_on:
- mssql
- otel-collector
ports:
- 5001:8080
environment:
ConnectionStrings__DbConnection: Server=mssql;Database=BookStore;User Id=SA;Password=P@ssw0rd?
Otlp__Endpoint: http://otel-collector:4317
networks:
- metrics
How to generate metrics to test the Grafana dashboards
In my GitHub repository you’ll also find a seed-data.sh
Shell script, this script will invoke some endpoints of the BookStore API via cURL.
To execute the seed-data.sh
you need to have cURL installed on your local machine.
The seed-data.sh
script runs the following actions:
- Add 8 book categories.
- Update 3 book categories.
- Delete 2 book categories.
- Add 17 books into the store.
- Update 4 existing books.
- Delete 2 existing books.
- Add inventory for every book on the store.
- Create 10 orders.
- Cancel 3 existing orders.
The purpose behind this script is to generate a decent amount of business metrics that later can be visualized in Grafana and Prometheus.