Enhancing Telemetry in Aspire

In a previous blog post, we covered the basic concepts of OpenTelemetry (OTEL). While it’s easy to add support for OTEL to any .NET project, if you’re working with an Aspire project that includes the ServiceDefaults template, then support for OpenTelemetry for all of your .NET code is available to you out of the box. And to top it off, the Aspire dashboard provides a lot of great ways to delve into those logs for your local development efforts.

You can view raw console output:

You can see it in a structured format:

You can view traces to follow the code paths:

You can see those traces in a graphical waterfall format:

And finally you have insights into a great many metrics of your code and the processes running it:

That’s a great place to start, and it provides a great deal of insight into your code locally while it’s being developed. It also comes with some problems. For example, it generally isn’t recommended that you deploy the Aspire dashboard to your deployment servers. The Aspire dashboard is meant primarily for local development support. Deploying it to production could open up a host of security concerns, as anyone with the link will get access to all of your debugging logs and information. It’s better for you to make use of a dedicated, and secure, observability platform for your deployed servers.

Non .NET Application Support

One great advantage with Aspire is that it provides support for non-.NET applications, like JavaScript SPA applications such as Angular and React. With Aspire v13, support for pulling in data for OTEL from JavaScript, NodeJs, Python, and more is supported out of the box. For basic console information, it’s built in to the default templates. For anything more, it does take a little bit of configuration and setup. You can get more direction on how to do that on the Aspire documents website.

Exporting Telemetry To A Dedicated Observability Platform

Once you deploy your application to a server, you need to send that data somewhere. There are a great many options available to you, from open source tools you can host yourself, such as Jaeger, to commercial hosted services such as DataDog, Grafana, or AppInsights. Some solutions, like Elastic Stack, offer both options.

Each has its own advantages and disadvantages, but it basically comes down to this: With a self-hosted platform, you are responsible to maintain and update the platform and your costs are server resources and labor. Commercial products remove most of that, but generally charge you for the amount of data you ship to their platform. There is also the consideration of longevity with commercial platforms. If they suddenly go out of business, you’re unexpectedly without an observability resource. And while OpenTelemetry makes it easy to redirect your logging to a different platform with nothing more than a configuration update in most cases, your historical data is all gone. That might or might not be a major concern for you, but it’s something to consider when laying out your application infrastructure and architecture design.

There is no one right answer for everyone or every situation. You’ll need to carefully consider the full costs and impacts of each option. And there is, of course, a third option: You can roll your own observability platform. That is, if you want to take the time, resources, and investment to do so. Like security, however, I wouldn’t recommend it. Leave it to the dedicated experts. It’ll be a lot cheaper in the long run.

Whatever the option you decide to pursue, however, for an OpenTelemetry compatible platform, the main thing you need to do on the Aspire application side is to configure the exporter for that service. For example, in the ServiceDefaults template project, they include the following line of commented out code:

// Uncomment the following lines to enable the Azure Monitor exporter (requires the Azure.Monitor.OpenTelemetry.AspNetCore package)
//if (!string.IsNullOrEmpty(builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
//{
//    builder.Services.AddOpenTelemetry()
//       .UseAzureMonitor();
//}

To send your logs to AppInsights, just add the listed package to the project, uncomment the code, and add your connection string. Poof! Support for AppInsights is in place. You’ll probably want to enhance the code a little bit so that it doesn’t send the data to AppInsights from your local development device. Depending on how you’ve set up your deployment environments, you could do something like the following:

if (!builder.Environment.IsDevelopment())
{
    if (!string.IsNullOrEmpty(builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
    {
        builder.Services.AddOpenTelemetry()
            .UseAzureMonitor();
    }
}

For most other providers, it will probably consist of a change to the defined OTEL_EXPORTER_OTLP_ENDPOINT environment variable. That’s even easier since all you need to do is update the variable in the deployed servers. It just depends on the provider you want to use. Consult the documentation for your chosen provider to get the details.

Custom Metrics

We touched in the previous post on adding custom metrics to a .NET application. Let’s look a little deeper on how to set that up in partnership with our ServiceDefaults shared project. The MeterProviderBuilder fluent syntax of the .WithMetrics() code allows you to pass in a string array of meter names. Let’s say, for example, that we have two endpoints that we want to track the activity of more closely: GetAllTeachers and AddTeacher. Our code defines them as follows:

// Get all teachers endpoint
app.MapGet("/teachers", async (AspireDemoEfContext context) =>
{
    var teachers = await context.Teachers.ToListAsync();
    return Results.Ok(teachers);
})
.WithName("GetAllTeachers")
.WithOpenApi();

// Add teacher endpoint
app.MapPost("/teachers", async (Teacher teacher, AspireDemoEfContext context) =>
{
    context.Teachers.Add(teacher);
    await context.SaveChangesAsync();
    return Results.Created($"/teachers/{teacher.TeacherId}", teacher);
})
.WithName("AddTeacher")
.WithOpenApi();

We want to create a metric for each endpoint to determine how many times each is being called. Let’s start by creating an extension method in our ServiceDefaults Extensions.cs class that adds the two new metric names to our telemetry configuration.

private static void AddCustomMetrics(this MeterProviderBuilder builder)
{
    builder.AddMeter(["Teachers.AddTeacher", "Teachers.GetAllTeachers"]);
}

Once we have that in place, we’ll add a call to that extension method in our .WithMetrics block of the ConfigureOpenTelemetry function:

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics.AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation()
            .AddCustomMetrics();  // <== Calling our extension method
    })
    .WithTracing(tracing =>

Now that we have that in place, we can go to the code in our API project and add calls to those metrics. First, we’ll set up the metric definitions:

var serviceName = "OtelDemo.TracingService";
builder.Services.AddSingleton(TracerProvider.Default.GetTracer(serviceName));
var teacherCreatedMeter = new Meter("Teachers.AddTeacher");
var teacherCreatedCounter = teacherCreatedMeter.CreateCounter<long>("call_to_create_teacher",
    description: "The number of calls to create a teacher");
var teachersListRetrievedMeter = new Meter("Teachers.GetAllTeachers");
var teachersListRetrievedCounter = teachersListRetrievedMeter.CreateCounter<long>("call_to_get_teachers_list",
    description: "The number of calls to get the teachers list");

Next, we’ll want to add the counter triggers to our endpoints.

// Get all teachers endpoint
app.MapGet("/teachers", async (AspireDemoEfContext context) =>
    {
        teachersListRetrievedCounter.Add(1);  // Trigger a Teacher.GetAllTeachers counter
        var teachers = await context.Teachers.ToListAsync();
        return Results.Ok(teachers);
    })
    .WithName("GetAllTeachers")
    .WithOpenApi();

// Add teacher endpoint
app.MapPost("/teachers", async (Teacher teacher, AspireDemoEfContext context) =>
    {
        teacherCreatedCounter.Add(1);  // Trigger a Teacher.AddTeacher counter
        context.Teachers.Add(teacher);
        await context.SaveChangesAsync();
        return Results.Created($"/teachers/{teacher.TeacherId}", teacher);
    })
    .WithName("AddTeacher")
    .WithOpenApi();

Now, we’ll launch our application and trigger each of the endpoints once or twice. Once we do, we’ll look at our Aspire dashboard Metrics section and we’ll see the metrics showing up on our dashboard.

Reusability

For a simple application, this would do. For a full application, however, you would probably want to create a full factory approach for your metrics to make implementation across a large app more efficient. The ServiceDefaults project makes a perfect place to create reusable metrics and spans that you can integrate across the multiple projects that typically make up an Aspire application.

Let’s say we want to create a generic function that can be used across all of our applications to trace certain portions of saving data to a database. We might create a generic function in our ServiceDefaults project that looks like this:

public static Activity? GetDatabaseSpan<T>(this ActivitySource activitySource, string name, T databaseObject)
{
    var activity = activitySource.StartActivity(name, ActivityKind.Internal);
    
    if (activity != null)
    {
        Type type = databaseObject.GetType();
        FieldInfo[] fields = type.GetFields(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
        foreach (var fieldInfo in fields)
        {
            object? value = fieldInfo.GetValue(databaseObject);
            activity.SetTag(fieldInfo.Name, value?.ToString() ?? "null");
        }
    }

    return activity;
}

Then, in our POST endpoint, we can update it as follows:

teacherCreatedCounter.Add(1);

using (var activity = activitySource.GetDatabaseSpan("AddTeacher", teacher))
{
    activity?.SetTag("operation.type", "create");
    
    context.Teachers.Add(teacher);
    await context.SaveChangesAsync();
}
return Results.Created($"/teachers/{teacher.TeacherId}", teacher);

Doing so, we can now see the custom span show up in our traces as part of the AddTeacher flow.

Conclusion

Aspire makes adding custom re-usable telemetry extremely easy. Its built in support for OpenTelemetry and its array of exporters are there to help you monitor your production applications just as easily as the local dashboard makes it to develop on your local machine.

Resources

Telemetry in Aspire Sending Aspire telemetry to Application Insights OpenTelemetry Exporters