A Step by Step Guide to Benchmarking in .NET

In this article I will show, step by step, how to set up a .NET benchmarking project. It is a cleaned up transcript from a video I made.

We’re gonna start with a blank slate and create our project. We’re gonna be writing two methods that we’re going to benchmark. And those methods are going to be different ways of concatenate strings. As you may know, in .NET, depending on how you do that, it can have a performance impact.

We’re going to be using a package called BenchmarkDotNet. I’m gonna show you how to use some of its attributes, and APIs that are available to us with that library.

We’re going to actually run the benchmarks. I’ll explain the results and what some of those statistics mean. And also how to get an actual report output that you can use and share and store somewhere else if you want to.

Alright, so this should be fun. Let’s dive right in.

We’re going to be doing this with the dotnet CLI and Visual Studio Code, but you can use any other IDE that you want.

The first thing we want to do is install the BenchmarkDotNet template.

dotnet new -i BenchmarkDotNet.Templates

Now we can create our actual project and I’m just going to create a project without a solution. But obviously what you would want to do in a real practical situation where you want to benchmark methods from an application or a library you’d probably just create that new benchmarking project inside of your existing solution.

dotnet new benchmark --console-app -f net6.0 -o MyBenchmark

There are a few flags here that we can add. --console-app is just a flag that tells the template that we want a console application. -f net6.0 is a flag that tells the template that we want to use .NET 6.0. -o StringBenchmarks is a flag that tells the template that we want to call our project StringBenchmarks and create a StringBenchmarks folder for it.

We can then open our project in our IDE.

cd StringBenchmarks
code .

Inside the project folder, the template has created a Benchmark class and file for us and this is where we will be spending the rest of our time.

namespace StringBenchmarks;

public class Benchmarks
{
  [Benchmark]
  public void Scenario1()
  {
    // Implement your benchmark here
  }

  [Benchmark]
  public void Scenario2()
  {
    // Implement your benchmark here
  }
}

Notice the Benchmark attribute here above those two methods? That’s what’s telling the BenchmarkDotNet package that these methods are special and they need to be benchmarked.

One thing I will do here, if you’re following along in VS Code, is to install and enable the C# extension.

Before we’ll start writing our benchmark methods, we’re going to add a MemoryDiagnoser attribute to our Benchmark class.

namespace StringBenchmarks;

[MemoryDiagnoser]
public class Benchmarks
{
  [Benchmark]
  public void Scenario1()
  {
    // Implement your benchmark here
  }

  [Benchmark]
  public void Scenario2()
  {
    // Implement your benchmark here
  }
}

This will just add some information about memory to the benchmark output; how much memory is allocated and how much garbage collection is taking place.

We’re now ready to implement the two methods that we will be benchmarking. We’re going to benchmark two different ways of creating strings of incrementing integers like this "0, 1, 2, 3, 4".

namespace StringBenchmarks;

[MemoryDiagnoser]
public class Benchmarks
{
  [Benchmark]
  public string StringJoin()
  {
    return string.Join(", ", Enumerable.Range(0, N).Select(i => i.ToString()));
  }

  [Benchmark]
  public string StringBuilder()
  {
    var sb = new StringBuilder();
    for (int i = 0; i < N; i++)
    {
        sb.Append(i);
        sb.Append(", ");
    }

    return sb.ToString();
  }
}

We’ve renamed our methods to StringJoin and StringBuilder so that we can easily distinguish them in the output. So here we have two methods that do the same thing, but in different ways. Let’s run our benchmarks to see which one is faster.

You might have noticed that StringBuilder outputs and extra ”, ” at the end of the string, compared to StringJoin. That is a mistake on my part. Thankfully, for this particular benchmark, that’s not a big deal and won’t affect the results in any meaningful way. But this is a good reminder that you might want to include some kind of test along your benchmarks to make sure that your benchmarks are actually doing what you think they are doing. Check out my follow up article on Validating Benchmarks for more information.

dotnet run -c Release

We are using a release configuration here because we want to use and optimized build for our benchmarks. Also, one thing to note is that ideally, if you want your benchmarks to be as accurate as possible, you should exit every running application on your system and run the command from the terminal.

The program will run for a little while and you will see a bunch of output to the terminal until it’s done.

Near the bottom of the output you’ll see a summary section that looks like this:

// * Summary *

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.22000
12th Gen Intel Core i9-12900K, 1 CPU, 24 logical and 16 physical cores
.NET Core SDK=6.0.303
  [Host]     : .NET Core 6.0.8 (CoreCLR 6.0.822.36306, CoreFX 6.0.822.36306), X64 RyuJIT
  DefaultJob : .NET Core 6.0.8 (CoreCLR 6.0.822.36306, CoreFX 6.0.822.36306), X64 RyuJIT


|        Method |     Mean |    Error |   StdDev |  Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------------- |---------:|---------:|---------:|-------:|------:|------:|----------:|
|    StringJoin | 69.71 ns | 1.374 ns | 1.582 ns | 0.0086 |     - |     - |     136 B |
| StringBuilder | 41.19 ns | 0.868 ns | 0.929 ns | 0.0102 |     - |     - |     160 B |

So we’ve got both of our scenarios here and it looks like, when considering the Mean column, that the method using a StringBuilder was significantly faster. But we can also see that it Allocated a bit more memory that the other method. Interesting. I wonder if we would get similar results if we increased the number of integers added to our strings. Let’s do that.

namespace StringBenchmarks;

[MemoryDiagnoser]
public class Benchmarks
{
  [Params(5, 50, 500)] //<-- This is a parameter attribute
  public int N { get; set; }

  [Benchmark(Baseline = true)] // <-- this is the baseline
  public string StringJoin()
  {
    return string.Join(", ", Enumerable.Range(0, N).Select(i => i.ToString()));
  }

  [Benchmark]
  public string StringBuilder()
  {
    var sb = new StringBuilder();
    for (int i = 0; i < N; i++)
    {
        sb.Append(i);
        sb.Append(", ");
    }

    return sb.ToString();
  }
}

Here we’ve added a N property that we can use to change the number of integers that we add to our strings. Notice that we’ve added a Params attribute to the N property. This is a parameter attribute that tells the benchmark that we want to use this property as a parameter for our benchmarks.

We’ve also added a Baseline parameter to our Benchmark attribute on the StringJoin method. This is a parameter that tells the benchmark that we want to use this method as the baseline for our benchmarks. It will will add just one or two columns to our report that give us basically a percentage difference between our baseline and the other test.

All right, So let’s run this again and and see what we get this time.

dotnet run -c Release

// * Summary *

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.22000
12th Gen Intel Core i9-12900K, 1 CPU, 24 logical and 16 physical cores
.NET Core SDK=6.0.303
  [Host]     : .NET Core 6.0.8 (CoreCLR 6.0.822.36306, CoreFX 6.0.822.36306), X64 RyuJIT
  DefaultJob : .NET Core 6.0.8 (CoreCLR 6.0.822.36306, CoreFX 6.0.822.36306), X64 RyuJIT


|        Method |   N |        Mean |      Error |     StdDev | Ratio |  Gen 0 |  Gen 1 | Gen 2 | Allocated |
|-------------- |---- |------------:|-----------:|-----------:|------:|-------:|-------:|------:|----------:|
|    StringJoin |   5 |    72.34 ns |   0.493 ns |   0.461 ns |  1.00 | 0.0086 |      - |     - |     136 B |
| StringBuilder |   5 |    41.29 ns |   0.833 ns |   0.780 ns |  0.57 | 0.0102 |      - |     - |     160 B |
|               |     |             |            |            |       |        |        |       |           |
|    StringJoin |  50 |   657.84 ns |   9.012 ns |   8.430 ns |  1.00 | 0.1125 |      - |     - |    1768 B |
| StringBuilder |  50 |   408.76 ns |   4.335 ns |   3.843 ns |  0.62 | 0.0815 |      - |     - |    1280 B |
|               |     |             |            |            |       |        |        |       |           |
|    StringJoin | 500 | 7,482.13 ns | 126.355 ns | 118.192 ns |  1.00 | 1.3046 | 0.0229 |     - |   20568 B |
| StringBuilder | 500 | 3,827.67 ns |  70.136 ns |  65.605 ns |  0.51 | 0.8698 | 0.0381 |     - |   13680 B |

Notice how the output is a little different. We’ve added a column for the N parameter. This is the number of integers that we add to our strings. We can now see that the StringBuilder method is significantly faster in all scenarios. We can also see that, except for when N is 5, the StringBuilder method ends up allocating less memory than the StringJoin method.

So there you have it a simple way to benchmark anything you want.

The last thing I will show you is that the BenchmarkDotNet package will output the results in a BenchmarkDotNet.Artifacts folder. This folder will contain generated report files in different formats; text, html, csv and markdown.

Here is what the generated markdown table looks like:

Pretty cool, right? This can actually be really useful for adding to a PR or issue comment on Github or other similar platforms.

We’re all done. Go forth and benchmark!