A day ago, reddit user zigs asked on the csharp subreddit:

What is the lowest effort, highest impact helper method you’ve ever written?

This proved a popular post, and one of the more popular answers from user _mattmc3_ was:

I’ve written a lot of SQL in my years as a developer, so foo IN(1, 2, 3) is a more intuitive way to express the concept to me than foo == 1

foo == 2

foo == 3 or even new int[] {1,2,3}.Contains(foo). Having foo being first just makes more sense, so I have a handy IsIn() extension method so I can write foo.IsIn(1, 2, 3):

    public static bool IsIn<T>(this T obj, params T[] values) {
        foreach (T val in values) {
            if (val.Equals(obj)) return true;
        }
        return false;
    }

    public static bool IsIn<T>(this T obj, IComparer comparer, params T[] values) {
        foreach (T val in values) {
            if (comparer.Compare(obj, val) == 0) return true;
        }
        return false;
    }

There was some discussion about this, pointing out that newer versions of C# supports params ReadOnlySpan<T> to avoid allocations.

I decided to benchmark a simple scenario where you might use this.

Benchmark

Let’s say we have a list of 10,000 numbers which randomly range from 0 to 63 inclusive. We want to count how many of these are 1, 3 or 7. That sounds abstract, but it’s a fairly common real-world pattern. Although in the real world, this might actually be counting a list of fruits for bananas, strawberries and mangoes. In general this scenario comes up when you have a set of items whose properties are generally known at compile time, but whose domain is open to future expansion so it doesn’t fit into a flags enum.

Our setup is therefore straightforward, we create the list of numbers.

    [GlobalSetup]
    public void Setup()
    {
        for (int i = 0; i < 10_000; i++)
        {
            keys[i] = Random.Shared.Next(0, 63);
        }
    }

As our baseline, we’ll compare approaches to perhaps the most natural thing to do: create a list and use Contains:

    [Benchmark(Baseline = true)]
    public int Contains()
    {
        return keys.Count(key => new int[] { 1, 3, 7 }.Contains(key));
    }

We want to compare this to the helper method suggested, as well as the suggested ReadOnlySpan version:

    [Benchmark]
    public int IsIn()
    {
        return keys.Count(key => key.IsIn(1, 3, 7));
    }

    [Benchmark]
    public int IsInReadOnly()
    {
        return keys.Count(key => key.IsInReadOnly(1, 3, 7));
    }

We should also compare it to the simplest approach, if or switch case:

    [Benchmark]
    public int WithIf()
    {
        return keys.Count(key => key == 1 || key == 3 || key == 7);
    }

    [Benchmark]
    public int WithSwitch()
    {
        return keys.Count(key =>
        {
            switch (key)
            {
                case 1:
                case 3:
                case 7:
                    return true;
            }
            return false;
        });
    }

Results

.NET Framework

I also ran a comparison on .NET Framework:

Without native ReadOnlySpan<T> support, that version is much slower than params T[], despite fewer allocations.

With the lack of LINQ optimisation in .NET Framework, Contains is by far the worst performing here.

Conclusion

The biggest surprise for me here was actually the performance difference between .NET 8 and .NET 10 for the naive if statement method. A dramatic improvement between .NET 8 and .NET 10 for a simple set of if statements suggests JIT improvements have made a big difference.

The ReadOnlySpan<T> approach avoided some allocations, but it still allocated more than the Contains approach, which was still more than the switch / case version.

For such a popular and tempting helper method, this is a real performance trap. Overall the worst performing version was 28x slower than hard-coded if statements.

As with any micro-benchmark, it’s important to stress that this is only important if profiling your real-world use case demonstrates it.

Always profile your real-world application and let that guide any optimisation you do. I have seen this pattern impact performance in a real world application, so keep it in mind.

Always profile within the context of your application. Blindly removing the helper method and replacing with Contains would be a disaster if you were still on .NET Framework.

Source Code

The source code to generate these results is available at https://github.com/richardcocks/IsInList .

Pull requests always welcome.

I’m preparing a blog post on remote procedure calls (RPC) and Interprocess Communication between .NET Framework and dotnet 8, but while doing so I ran into an issue with a service getting stuck consuming CPU after a client has closed.

It’s not a good fit for asking Claude, etc., because there’s too much confusion on whether you’re talking about CoreWCF, WCF client on .Net Core (System.ServiceModel nuget package version ), or WCF on .NET Framework ( System.ServiceModel from the framework ). This is confusing enough for humans, let alone a machine that will happily reproduce calls and configuration from something that is almost identical, yet subtly different and incompatible.

I didn’t want to raise a github issue, because I doubt I’ve found a bug in WCF, especially since it feels like what I’m trying to do is probably a misunderstanding of how WCF streams are supposed to work. I’m also quickly hitting maximum message lengths, and that feels like a red flag that I’m not using it as intended.

I took my question to StackOverflow (SO). I’m not a stranger to SO, although it’s been a while since I have contributed anything there. This turned out to be a mistake, because I rushed asking my question. I extracted the classes I thought were pertinent to the question and that I thought could reproduce the issue. I skipped over some boilerplate starting Kestrel, but I also included a link to my repo in case answerers wanted to see the full context of the servicescalled. I had forgotten that any external links are a big no-no in SO land, so my question immediately attracted 2 close votes.

I went back, rewrote the entire question, now with a complete minimal reproduction of the issue. Client and Server fully complete in 3 source files. It now contained all the code needed to reproduce the issue, and nothing but that code.

Two days later my question got it’s third vote for closure, and remains unanswered and now closed forever.

In my frustration I’m writing this blog post, to briefly introduce CoreWCF and hope someone will be able to answer my question about what I’m doing wrong.

Scenario

For the purpose of testing RPC throughput, I want to stream random numbers from one process to another. We can test requesting numbers call-by-call or streamed. Ideally with sequence numbers so we can also examine the reliabilty for certain transport types.

In gRPC, the proto file would look something like this:

syntax = "proto3";

option csharp_namespace = "RandomNumberGrpc";

package randomService;

service RandomProvider {
  rpc NextInt (NextIntRequest) returns (ValueWithSequence);
  rpc Stream (NextIntStreamRequest) returns (stream ValueWithSequence);
}

message NextIntRequest {}
message NextIntStreamRequest{}

message ValueWithSequence {
  int32 sequenceNumber = 1;
  int32 Value = 2;
}

In WCF, we define services through ServiceContract attributes. For this post I’ll just focus on the streaming service and ignore the per-call implementation. In WCF, as I understand it, we can’t strongly type our stream. For now therefore, to get streams working, I just made a contract that returns a raw byte Stream, and will worry about casting or marshalling that to the right structure later.

Let’s look at our service class.

namespace RandomNumberCore;

[ServiceContract]
public interface IStreamingService
{
    [OperationContract]
    Stream GetRandomStream();
}
    

public class StreamingService : IStreamingService
{
    public Stream GetRandomStream()
    {
        return new RandomStream(Random.Shared);
    }
}

Where RandomStream is my own class that exposes Random.Shared as a stream, without worrying about returning the sequence number for now:

namespace RandomNumberCore;

public class RandomStream : Stream
{
    public RandomStream(Random random)
    {
        this._random = random;
    }
    
    private int _sequence;
    private readonly Random _random;
    public override bool CanRead => true;

    public override bool CanSeek => false;

    public override bool CanWrite => false;

    public override long Length => throw new NotSupportedException();

    // ReSharper disable once ValueParameterNotUsed
    public override long Position { get => _sequence; set => throw new NotSupportedException(); }

    public override void Flush()
    {}
    
    public override int Read(byte[] buffer, int offset, int count)
    {
        var internalBuffer = new Span<byte>(buffer, offset, count);
        _random.NextBytes(internalBuffer);
        _sequence+=count;
        return count;
    }

    public override int Read(Span<byte> buffer)
    {
        _random.NextBytes(buffer);
        _sequence+=buffer.Length;
        return buffer.Length;
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        throw new NotSupportedException();
    }

    public override void SetLength(long value)
    {
        throw new NotSupportedException();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        throw new NotSupportedException();
    }
}

A read-only, non-seekable stream that returns random numbers. I’ve not extensively tested this stream, but it appears to work fine.

Let’s write a simple client, including the contract. In a real world application you’d likely define a contract assembly to share the contracts.

using System.ServiceModel;

namespace RandomNumberConsumerNet8
{
    [ServiceContract]
    public interface IStreamingService
    {
        [OperationContract]
        Stream GetRandomStream();
    }
    public interface IStreamingServiceChannel : IStreamingService, IClientChannel;

    internal class Program
    {
        public static async Task Main(string[] args)
        {
            var cts = new CancellationTokenSource();
            using var channelFactory = new ChannelFactory<IStreamingServiceChannel>(new BasicHttpBinding(BasicHttpSecurityMode.Transport){TransferMode = TransferMode.Streamed, MaxReceivedMessageSize = 1_000_000_000 }, new EndpointAddress("https://localhost:7151/StreamingService.svc"));
            using var service = channelFactory.CreateChannel();
            service.Open();
            using var randomStream = service.GetRandomStream();
            byte[] buffer = new byte[4];
            await randomStream.ReadExactlyAsync(buffer, cts.Token);
            
            Console.WriteLine($"Received bytes {buffer[0]} , {buffer[1]}, {buffer[2]}, {buffer[3]} ");
            service.Close();
            channelFactory.Close();
        }
    }
}

Results

Now when I run the server and client, it appears to work:

Received bytes 101 , 18, 99, 251

Great, we opened the stream, streamed 4 bytes and then closed service.

But now when I look at my CPU, it’s still chugging along. Profiling the server shows it’s still trying to write bytes to the stream, well after the client has long ago disconnected.

What is it doing? It’s trying to write to the stream, with no hint of back-pressure or sense it shouldn’t be doing so. The sending stream doesn’t have any sense that it’s not being read from.

I’m coming to the conclusion that WCF streaming is not suitable for this, and is only suitable for single bounded streams, not for streams of unknown length or a stream of messages.

But it’s not therefore clear what to do in this scenario, of wanting to transfer an unknown quantity of random numbers. Do I go back to requesting numbers via single messages? That has limited throughput. My initial testing showed ~5k messages / sec that way. I could manually increase the buffer so that each message sends a greater quantity of random numbers, but that loses the fidelity, and rather misses the point, this isn’t really about random numbers, it’s about how quickly we can pass messages between applications.

Do I need to use session mode and coordinate the stream externally to the RPC?

I’m not sure if I’ve missed the point of WCF streams or something else about WCF tuning entirely, but what I really wish is that this could have been answered on StackOverflow, so others trying something similarly misguided could have learned from my mistakes.

Footnote

Comments? Questions? Want to point out how much of an idiot I’ve been? Please reach out at @eterm.bsky.social.

Code behind this post is available at https://github.com/richardcocks/randomNumberStackOverflow .

In this post I look at improvements in .NET and using Span for performance and portability.

I was examining portability issues in a code base that I wanted to migrate from .NET framework 4.8.1 to .NET8. I discovered use of msvcrt.dll. I quickly established it is a popular stackoverflow answer for a fast way to compare byte arrays in .NET

The answer as provided, and faithfully copied into codebases, is this:

[DllImport("msvcrt.dll", CallingConvention=CallingConvention.Cdecl)]
static extern int memcmp(byte[] b1, byte[] b2, long count);

static bool ByteArrayCompare(byte[] b1, byte[] b2)
{
    // Validate buffers are the same length.
    // This also ensures that the count does not exceed the length of either buffer.  
    return b1.Length == b2.Length && memcmp(b1, b2, b1.Length) == 0;
}

A big performance improvement in modern .NET is the Span<T> type. The documentation describes it as:

Provides a type-safe and memory-safe representation of a contiguous region of arbitrary memory.

That’s not a super helpful description, but the summary is that it’s stack-allocated rather than heap allocated.

Span<T> has an extension method SequenceEqual<T>(this ReadOnlySpan<T> span, ReadOnlySpan<T> other), and we’ll see how it fares.

    public static bool EqualsSpan(ReadOnlySpan<byte> b1, ReadOnlySpan<byte> b2)
    {
        return b1.SequenceEqual(b2);
    }

Let’s see how it stacks up against a couple of naive implementations, a for loop and using Enumerable.SequenceEquals:

    public static bool EqualsLoop(byte[] b1, byte[] b2)
    {

        if (b1.Length != b2.Length) return false;
        for (int i = 0; i < b1.Length; i++)
        {
            if (b1[i] != b2[i]) return false;
        }

        return true;
    }

    public static bool EqualsSequenceEqual(byte[] b1, byte[] b2)
    {
        return b1.SequenceEqual(b2);
    }

We compare using two identical arrays since this is typically the worst-case for equality checking, and we’re going to benchmark on a range of array sizes: 10 bytes, 1KB, 1MB and 1GB.

    [Params(10, 1_024, 1_048_576, 1073741824)]
    public int Length { get; set; }

    byte[] first;
    byte[] second;

    [GlobalSetup]
    public void Setup()
    {
        var r = new Random(0);

        first = new byte[Length];
        second = new byte[Length];

        r.NextBytes(first);
        Array.Copy(first, second, Length);
    }

The setup is straightforward, we fill the first array with random data, and copy the data to the second array.

Results

BenchmarkDotNet v0.14.0, Windows 10 (10.0.19045.5679/22H2/2022Update)
AMD Ryzen 7 3800X, 1 CPU, 16 logical and 8 physical cores
.NET SDK 10.0.100-preview.2.25164.34
  [Host]               : .NET 9.0.3 (9.0.325.11113), X64 RyuJIT AVX2
  .NET 8.0             : .NET 8.0.14 (8.0.1425.11118), X64 RyuJIT AVX2
  .NET 9.0             : .NET 9.0.3 (9.0.325.11113), X64 RyuJIT AVX2
  .NET Framework 4.8.1 : .NET Framework 4.8.1 (4.8.9290.0), X64 RyuJIT VectorSize=256

| Method | Job | Runtime | Length | Mean | Ratio | RatioSD | Allocated | |—————|———————-|———————-|—————-|———————–:|———:|———:|———-:| | MemCmp | .NET 8.0 | .NET 8.0 | 10 | 7.957 ns | 0.65 | 0.01 | - | | MemCmp | .NET 9.0 | .NET 9.0 | 10 | 7.877 ns | 0.64 | 0.01 | - | | MemCmp | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 10 | 12.239 ns | 1.00 | 0.02 | - | | | | | | | | | | | Loop | .NET 8.0 | .NET 8.0 | 10 | 4.390 ns | 0.88 | 0.03 | - | | Loop | .NET 9.0 | .NET 9.0 | 10 | 6.439 ns | 1.29 | 0.05 | - | | Loop | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 10 | 4.995 ns | 1.00 | 0.03 | - | | | | | | | | | | | SequenceEqual | .NET 8.0 | .NET 8.0 | 10 | 21.341 ns | 0.15 | 0.00 | - | | SequenceEqual | .NET 9.0 | .NET 9.0 | 10 | 7.611 ns | 0.05 | 0.00 | - | | SequenceEqual | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 10 | 139.476 ns | 1.00 | 0.02 | 64 B | | | | | | | | | | | Span | .NET 8.0 | .NET 8.0 | 10 | 2.394 ns | 0.21 | 0.00 | - | | Span | .NET 9.0 | .NET 9.0 | 10 | 1.624 ns | 0.14 | 0.00 | - | | Span | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 10 | 11.523 ns | 1.00 | 0.01 | - | | | | | | | | | | | MemCmp | .NET 8.0 | .NET 8.0 | 1024 | 36.745 ns | 0.89 | 0.01 | - | | MemCmp | .NET 9.0 | .NET 9.0 | 1024 | 36.317 ns | 0.88 | 0.01 | - | | MemCmp | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1024 | 41.452 ns | 1.00 | 0.01 | - | | | | | | | | | | | Loop | .NET 8.0 | .NET 8.0 | 1024 | 247.326 ns | 0.66 | 0.01 | - | | Loop | .NET 9.0 | .NET 9.0 | 1024 | 246.738 ns | 0.66 | 0.01 | - | | Loop | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1024 | 372.410 ns | 1.00 | 0.01 | - | | | | | | | | | | | SequenceEqual | .NET 8.0 | .NET 8.0 | 1024 | 32.069 ns | 0.003 | 0.00 | - | | SequenceEqual | .NET 9.0 | .NET 9.0 | 1024 | 21.439 ns | 0.002 | 0.00 | - | | SequenceEqual | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1024 | 9,542.293 ns | 1.000 | 0.02 | 64 B | | | | | | | | | | | Span | .NET 8.0 | .NET 8.0 | 1024 | 12.408 ns | 0.51 | 0.01 | - | | Span | .NET 9.0 | .NET 9.0 | 1024 | 12.310 ns | 0.51 | 0.01 | - | | Span | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1024 | 24.117 ns | 1.00 | 0.01 | - | | | | | | | | | | | MemCmp | .NET 8.0 | .NET 8.0 | 1048576 | 31,477.776 ns | 0.99 | 0.02 | - | | MemCmp | .NET 9.0 | .NET 9.0 | 1048576 | 31,790.009 ns | 1.00 | 0.02 | - | | MemCmp | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1048576 | 31,693.469 ns | 1.00 | 0.02 | - | | | | | | | | | | | Loop | .NET 8.0 | .NET 8.0 | 1048576 | 247,350.116 ns | 0.65 | 0.01 | - | | Loop | .NET 9.0 | .NET 9.0 | 1048576 | 251,317.223 ns | 0.66 | 0.02 | - | | Loop | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1048576 | 379,628.993 ns | 1.00 | 0.02 | - | | | | | | | | | | | SequenceEqual | .NET 8.0 | .NET 8.0 | 1048576 | 20,974.963 ns | 0.002 | 0.00 | - | | SequenceEqual | .NET 9.0 | .NET 9.0 | 1048576 | 20,615.505 ns | 0.002 | 0.00 | - | | SequenceEqual | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1048576 | 9,744,674.688 ns | 1.000 | 0.01 | - | | | | | | | | | | | Span | .NET 8.0 | .NET 8.0 | 1048576 | 20,955.331 ns | 0.99 | 0.02 | - | | Span | .NET 9.0 | .NET 9.0 | 1048576 | 20,598.672 ns | 0.97 | 0.01 | - | | Span | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1048576 | 21,176.643 ns | 1.00 | 0.02 | - | | | | | | | | | | | MemCmp | .NET 8.0 | .NET 8.0 | 1073741824 | 111,762,734.375 ns | 1.04 | 0.03 | 80 B | | MemCmp | .NET 9.0 | .NET 9.0 | 1073741824 | 110,374,794.400 ns | 1.03 | 0.03 | 80 B | | MemCmp | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1073741824 | 107,072,063.077 ns | 1.00 | 0.01 | - | | | | | | | | | | | Loop | .NET 8.0 | .NET 8.0 | 1073741824 | 280,450,679.167 ns | 0.69 | 0.02 | 200 B | | Loop | .NET 9.0 | .NET 9.0 | 1073741824 | 523,091,792.857 ns | 1.29 | 0.02 | 400 B | | Loop | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1073741824 | 404,927,735.714 ns | 1.00 | 0.01 | - | | | | | | | | | | | SequenceEqual | .NET 8.0 | .NET 8.0 | 1073741824 | 95,954,794.298 ns | 0.010 | 0.00 | 67 B | | SequenceEqual | .NET 9.0 | .NET 9.0 | 1073741824 | 94,486,122.500 ns | 0.010 | 0.00 | 80 B | | SequenceEqual | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1073741824 | 9,944,911,760.000 ns | 1.000 | 0.00 | - | | | | | | | | | | | Span | .NET 8.0 | .NET 8.0 | 1073741824 | 92,945,091.026 ns | 0.97 | 0.01 | 67 B | | Span | .NET 9.0 | .NET 9.0 | 1073741824 | 94,375,230.882 ns | 0.99 | 0.03 | 67 B | | Span | .NET Framework 4.8.1 | .NET Framework 4.8.1 | 1073741824 | 95,817,247.619 ns | 1.00 | 0.02 | - |

The first notable result is that for very small arrays, the overhead of calling memcmp is a waste compared to a naive loop. In .NET Framework, the loop is overall fastest for 10 element arrays. This is not unexpected, but it’s important to note; don’t try to optimise if you actually have small arrays. The loop does not scale well at all however, and that advantage has completely disappeared by just 1000 elements.

The more biggest difference is between .NET framework and .NET 8. Even the loop is notably faster in .NET 8. There is a bizarre performance regression from .NET 8 to .NET 9 for 1GB arrays, which I will investigate separately to try to confirm that result, it may have been a glitch in the benchmark, given twice the memory allocations and twice the time taken.

When we look at IEnumerable<T>.SequenceEqual, it is 500 times faster for our 1MB array in .NET 8 than in .NET framework. In .NET8 and .NET9, IEnumerable<T>.SequenceEqual is faster than the version of memcmp I have on my machine.

There wasn’t a significant difference between ReadOnlySpan<T>.SequenceEqual and IEnumerable<T>.SequenceEqual, with the difference around the margin of error.

memcmp is still a little slower than SequenceEqual across the board. It’s still very fast, much faster than naive methods, but it’s clearly no longer necessary for achieving high performance for array comparisons. When the original stackoverflow answer was written, there was nothing available in .NET that could come close to that performance, as it was before Span<T> was added to the runtime.

A benefit of the ReadOnlySpan<T> implementation over using IEnumerable<T>.SequenceEqual is that we can also trust that it still perform acceptably when we target .NET Framework.

Conclusion

If you’re using .NET 8 and don’t need to run in the .NET Framework runtime, don’t write your own utility function and just use IEnumerable<T>.SequenceEqual, it’s incredibly fast and doesn’t need any external dependencies to just work.

If you’re on .NET Framework, then bring in System.Memory and use Span<T>.SequenceEquals instead of relying on external C libraries. Make sure any calls to IEnumerable<T>.SequenceEquals are checked to make sure they aren’t operating on large arrays.

Other considerations

If you regularly need to compare very large arrays and they are append-only and/or are compared more often than constructed, then it may make sense to avoid the comparison entirely by using a data structure that includes and maintains an order-sensitive hash of its contents. Most negative cases can be discounted through a hash comparison before doing the more expensive array comparison. Rebuilding this hash could be expensive for arrays that shuffle or remove items however.

Source Code

The source code to generate these results is available at https://github.com/richardcocks/memcomparison/ .

Pull requests always welcome.

I’ve been nerd-sniped by co-pilot. I don’t normally have it enabled, but was working on another machine which did. I was implementing a feature when it suggested autocompleting GeneratePassword.

It took me on a journey of benchmarking with BenchmarkDotNet and discovering what did and did not affect performance in microbenchmarking.

Co-pilot output

The original suggestion, straight from co-pilot was:

public string GeneratePassword(int length)
{
    string password = "";
    string characters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*()_+";
    System.Random random = new System.Random();

    for (int i = 0; i < length; i++)
    {
        password += characters[random.Next(characters.Length)];
    }
    return password;
}

It surprised me, because it’s in the perfect spot of being just about good enough to sneak through some reviews. It would certainly get a few remarks if reviewed as a standalone feature, but it could get through if bundled as part of 20 files in a wider feature. This is a benefit of small commits, but that’s a post for another day.

Yet it’s also bad code. There are to me, three main concerns:

• Performance - The use of new System.Random() rather than System.Random.Shared, and string concatenation rather than StringBuilder, are two things in particular that stand out as quick wins for performance.
• Security - Using System.Random instead of a cryptographically secure generator such as System.Security.Cryptography.RandomNumberGenerator is a concern for any password generator.
• Lack of validating output for symbols - It’s desirable for passwords to always have at least one symbol to pass most validation sets. With this password generator, you’d be fine most of the time given the input set, but then occassionally be frustrated when it generates a password without symbols. This would happen around 5% of the time for a 16 character password. Just often enough to get missed in testing then hit you later.

Improving Security

Let’s address security first, it’s no good having a password generator be fast if you can’t trust the output.

public string SecureRandom(int length)
{
    string password = "";
    string characters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*()_+";

    for (int i = 0; i < length; i++)
    {
        password += characters[RandomNumberGenerator.GetInt32(characters.Length)];
    }
    return password;
}

I’ve kept the same naive implementation, but swapped random.Next() for the static (and thread safe) RandomNumberGenerator.GetInt32 which is a more secure random number generator.

Benchmarking and performance optimisation

Let’s get this under the benchmark microscope to see how we can address performance.

Measuring baseline performance is easy with BenchmarkDotNet, we just create a class and then annotate it.

Since we’re creating a class, let’s move length and characters to properties in the class.

This will also let us easily parameterise the length across our benchmarks. I’ve gone with 3 different lengths of passwords, so that we can see the effect of increasing password length on generation, and also used two separate categories so we can compare our optimisation efforts on both the secure and vulnerable versions of the password generator.

[Benchmark()]
public string GeneratePasswordSharedRandom()
{
    string password = "";

    for (int i = 0; i < Length; i++)
    {
        password += characters[Random.Shared.Next(characters.Length)];
    }
    return password;
}

Let’s immediately look at the impact of removing System.Random random = new System.Random(); and using the static System.Random.Shared for the vulnerable example.

For this example I’ll post the whole class, for future examples I’ll just post individual methods.

We can now see that avoiding new System.Random() increased performance, roughly 35% faster for the 24 character example.

We can also see that using RandomNumberGenerator.GetInt32 destroyed our performance, taking us into microseconds territory and taking around 8 times as long to do the same work.

To make comparisons in the table easier, we can add [BenchmarkCategory("Secure")] and [BenchmarkCategory("Vulnerable")] attributes to our benchmarks to mark GeneratePassword and SecureRandom as two separate baselines, so we can more easily examine the performane impact on changes made to each. We also need to mark the class with [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] and [CategoriesColumn] to get a category column in the output table.

String Building

Okay, let’s do the other straightforward and obvious improvement, to use StringBuilder, so that our non-secure version now looks like this:

[BenchmarkCategory("Vulnerable"), Benchmark()]
public string StringBuilder()
{

    StringBuilder password = new(Length);
    for (int i = 0; i < Length; i++)
    {
        password.Append(characters[Random.Shared.Next(characters.Length)]);
    }

    return password.ToString();
}

// With an equivalent Secure version not shown here.

Since we know the size of the string, we are able to intialize our string builder with that capacity. However, given this knowledge, we can also allocate a char[] and fill it. Let’s also try that at the same time:

[BenchmarkCategory("Secure"), Benchmark()]
public string CharArraySecure()
{
    char[] buffer = new char[Length];

    for (int i = 0; i < Length; i++)
    {
        buffer[i] = characters[RandomNumberGenerator.GetInt32(characters.Length)];
    }

    return new string(buffer);
}

// With an equivalent Vulnerable version not shown here.

Okay, that’s another modest improvement. It’s hard to see with the secure version, but with the vulnerable version we’ve confirmed the char[] approach is better than StringBuilderfor building short fixed-length strings up from characters.

The time in the secure version is dominated by the random number generation, so let’s fix that.

Faster random generators

We need to address time spent in our secure version which is completely dominated by GetInt32. We can improve performance by getting all the bytes we need at once and then encoding them into characters.

To allow us to do this, we will need to make an important compromise, so that we don’t introduce bias.

Our character set is 74 characters. If we were to generate a byte and then do value % 74, we would be introducing a bias toward characters a through H. I won’t go into the mathematics, but this can be seen by running this code:

byte[] foo = new byte[1024*1024];
string characters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*()_+";

Random.Shared.NextBytes(foo);

var frequencies = foo.Select(f => f % 74)
.GroupBy(f => f)
.OrderBy(f => f.Key)
.Select(f => new { Value = characters[f.Key], Frequency = f.Count() });

foreach (var c in frequencies) {
 	Console.WriteLine($"'{c.Value}': {c.Frequency}");
}

'a': 16452
'b': 16693
...
'G': 16527
'H': 16543
'I': 12233
'J': 12248
'K': 12242
...
'_': 12279
'+': 12203

There’s a heavy bias with a through H having approximately ~16,300 appearances with I through + having roughly 12,200.

To eliminate this bias, we need a character set that divides evenly into 256. We can either try to pad to 128 characters, which will significantly increase the entropy of our password for a given length, or we can cut down to 64 characters, which will have the unfortunate side-effect of also reducing entropy.

We’ll cut down to 64. It’s difficult to think of 50 more recognisable characters and we can also take the opportunity to cut out characters that can be confused for each other in some fonts, such as I and l.

The reduction in entropy can be calculated. For a 16 character password, we are going from 74^16 ~= 100 bits, to 64^16 = 96 bits. So we’ve lost around 4 bits from our password. If this is a concern, then we can increase our password length by one character to accomodate.

The actual entropy lost will be log_2(64/74) = -0.2094534 bits per character in the password.

It is a weakening, but if it’s properly documented, should not be a concern.

Let’s define our new character set:

private const string charactersShortSet = "abcdefghjkmnpqrstuwxyzABCDEFGHJKLMNPQRSTVWXYZ0123456789@#$%&()_+";

Now we have a character set that won’t introduce bias, let’s add a function that gets all the bytes at once from our Random sources:

public string Buffer()
{
    byte[] bytebuffer = new byte[Length];
    RandomNumberGenerator.Fill(bytebuffer);

    char[] buffer = new char[Length];

    for (int i = 0; i < Length; i++)
    {
        buffer[i] = charactersShortSet[bytebuffer[i] % 64];
    }

    return new(buffer);
}

We’ll also produce an equivalent “vulnerable” version using Random.Shared.NextBytes, although I’d recommend not using it.

Now there’s the improvement we were hoping for! Our vulnerable version is now up to 8 times faster than the original co-pilot output.

More importantly, our secure version is actually faster than the original vulnerable co-pilot version.

Using the full symbol set

We’ve sacrificed some symbols and characters from the output to achieve this. Can we maintain a good speed while also using our full character set? It might be possible: while looking up RandomNumberGenerator.Fill I spotted RandomNumberGenerator.GetItems<T>, which:

Creates an array populated with items chosen at random from choices

That sounds exactly like what we’re after. There’s also an equivalent Random.Shared.GetItems<T>, let’s implement them, going back to our original 74 character set to do so. This leaves our methods as:

[BenchmarkCategory("Vulnerable"), Benchmark()]
public string GetItems()
{
    char[] buffer = Random.Shared.GetItems<char>(characters, Length);
    return new(buffer);
}

[BenchmarkCategory("Secure"), Benchmark()]
public string GetItemsSecure()
{
    char[] buffer = RandomNumberGenerator.GetItems<char>(characters, Length);
    return new(buffer);
}

That’s definitely a lot neater code than the original, let’s see how it performs:

Okay, well that’s a bit of a performance regression. We’d need to understand the trade-offs of using the full character set vs the reduced character set with better performance.

At this point we can drop benchmarking the “weak” version entirely. We now have a secure version that has feature parity with the original while also being faster than the original code. I’ll be using GetItemsSecure as the baseline for benchmarks going forward.

There’s a couple of final performance tweaks we can try, but before we do, we should address the third bullet point on the list of problems with the original co-pilot code.

Fixing the functionality

It’s important we don’t lose sight of our goal, a working password generator that won’t frustrate us by sometimes returning a password that contains no symbols.

There are a few approaches to fixing this issue, some of which are fraught with the danger of re-introducing bias. One naive way is to generate a character from the symbols set (or k characters) and then generate n-k characters from the full set, then shuffle both sets together. This would however reintroduce a subtle bias.

The easiest way to demonstrate this bias is with a set of 3 fair coins.

We have 8 possible random outcomes:

HHH
HHT
HTH
HTT
THH
THT
TTH
TTT

If we demand a passcode that “must have at least 1 T”, then we have 7 possible equally likely outcomes, that is all outcomes except HHH.

Let’s ee what happens if we try to generate this by picking a T, then randomly picking two more characters and then shuffling.

We have 4 intial outcomes from our two remaining coin tosses:

T + HH
T + HT
T + TH
T + TT

For each initial outcome, we have 6 ways to shuffle the 3 characters.

So when shuffled, these become 24 equally likely outcomes:

( From T + HH )

THH
THH
HTH
HHT
HTH
HHT

( From T + HT )
THT
TTH
HTT
HTT
THT
TTH

( From T + TH )

THT
TTH
HTT
HTT
THT
TTH

( From T + TT )
TTT
TTT
TTT
TTT
TTT
TTT

Yes, 6 in 24, a full quarter of all outcomes are TTT, yet with a fair coin there should be just 1 in 7 cases of TTT given there is at least one T.

So that approach is out. The fairer way to do this is to generate a password, then if it doesn’t meet the criteria, throw it out entirely and start again.

We’ll also parameterise the minimum number of special characters in our code, to look at requiring 0, 1 or 2 special characters.

[Params(0, 1, 2)]
public int MinmumSpecialCharacters { get; set; }

When we’re using RandomNumberGenerator.Fill and looking up the result in our character set, our character set is fortunately arranged so that we can cheaply determine if there is a special character. We just have to look at the value of the byte to see if it is 55 (0x37) or higher, since indexes 55 to 63 in our string are all special characters.

For the case where we are using GetItems, we have to count char.IsAsciiLetterOrDigit and take that from our password length, then compare that count against the requested minimum symbols.

[BenchmarkCategory("Secure"), Benchmark()]
public string RejectionSampleSecure()
{
    byte[] bytebuffer = new byte[Length];
    char[] buffer = new char[Length];

    while (true)
    {
        RandomNumberGenerator.Fill(bytebuffer);
        int specialChars = 0;
        bool metMinimum = false;

        for (int i = 0; i < Length; i++)
        {
            if (!metMinimum && (bytebuffer[i] > 54) && (++specialChars >= MinmumSpecialCharacters))
            {
                metMinimum = true;
            }

            buffer[i] = charactersShortSet[bytebuffer[i] % 64];
        }

        if (metMinimum)
        {

            return new(buffer);
        }
    }
}

[BenchmarkCategory("Secure"), Benchmark()]
public string GetItemsWithRejectionSecure()
{
    while (true)
    {
        char[] buffer = RandomNumberGenerator.GetItems<char>(characters, Length);

        if ((buffer.Length - buffer.Count(char.IsAsciiLetterOrDigit)) >= MinmumSpecialCharacters)
        {
            return new(buffer);
        }
    }
}

For this solution we have had to enumerate the array again for the GetItems approach. We expect this to further worsen its performance relative to the 64 character set approach, so let’s see the results:

As expected, this has added overhead, it leaves us with a choice between RejectionSampleSecure with it’s slightly reduced entropy per output character but preferable performance characteristics and GetItemsWithRejectionSecure, which uses the full symbol set for feature parity with the original code.

Further performance improvements

We can try to speed up the GetItems based methods by using a loop to count the special characters, so we can exit early when we’ve met our target rather than counting all special characters.

Also, we can try to avoid a heap allocation by using stackalloc to allocate the span on the stack.

This leaves our code like the following:

[Benchmark()]
public string StackAllocSecure()
{
    Span<char> buffer = stackalloc char[Length];

    while (true)
    {
        RandomNumberGenerator.GetItems<char>(characters, buffer);

        int specialChars = 0;

        for (int i = 0; i < Length; i++)
        {
            if (!char.IsAsciiLetterOrDigit(buffer[i]) && (++specialChars >= MinmumSpecialCharacters))
            {
                return new(buffer);
            }
        }

    }
}

Checking in a loop has significantly reduced the overhead of counting special characters.

Stack allocation has barely changed running time, but has halved allocated bytes, as can be seen in these graphs by the purple allocation line.

Double-rejection

We’ve seen how rejection sampling avoids bias in the outcome when dealing with passwords that don’t meet minimum characters.

We can try a similar approach to avoid the bias in trying to sample from 74 characters.

If we first create 4 copies of each character, we pad our set to 222 characters. We also take the opportunity to sort the character map so the symbols are the first 48 characters in our expanded set.

If we then sample bytes, skipping any results outside of 0-221, and checking for values <=48 at the same time. Once we’ve generated a password, if we have the minimum, we return the password, else we keep going through the generated bytes.

We’ll need to take enough bytes into our buffer so that we can reasonably expect to not exhaust the buffer.

The chance of fewer than K special bytes in N characters is calculatable using the Binomial distribution, but to get a rough idea we’ll suggest people might want up to 1/4 their characters be symbols. For our worst case here, that is 8 symbols in 32. The probability of achieving this is only 0.135157, or roughly 1 in 7.

We also have only 222 in 256 chance of being able to use each byte, so we’ll need another 15% overhead to accomodate bytes rejected initially.

Together, we’ll need to ask for roughly 8.5 times as much random data as previous. For 32 length passwords, that is ~273 bytes, but we’ll be generous and round up to the next power of 2, and take a 512 byte buffer.

Alternatively, we can have a strategy in place for getting a new buffer if we run out, and therefore we can be more conservative and target a 256 byte buffer, which could have performance benefits given it can be addressed by a single byte.

We can parameterise the buffer length to find an acceptable trade-off between requesting too much data, and having to re-fill the buffer.

[Benchmark]
public string DoubleRejection()
{
    Span<byte> bytebuffer = stackalloc byte[RandomBufferLength];
    Span<char> buffer = stackalloc char[Length];


    while (true)
    {
        int i = 0;
        RandomNumberGenerator.Fill(bytebuffer);
        int charIndex = 0;
        bool metMinimum = false;

        int specialChars = 0;

        while (i < RandomBufferLength)
        {
            byte value = bytebuffer[i];
            if (value >= 222)
            {
                i++;
                continue;
            }

            buffer[charIndex] = charactersLongSet[bytebuffer[i]];

            if (!metMinimum && (value < 36) && (++specialChars >= MinmumSpecialCharacters))
            {
                metMinimum = true;
            }

            if (++charIndex == Length)
            {
                if (metMinimum)
                {
                    return new(buffer);
                }

                // reset charIndex
                charIndex = 0;
                specialChars = 0;
                i++;
                continue;

            }
            else
            {
                i++;
                continue;
            }
        }
    }
}

Here we can see that the fastest time was indeed a 512 byte buffer. This will be overkill for shorter password lengths and fewer required minimum special characters, but we can see from the shortest buffer lengths that the cost of having to refill the buffer is great.

That’s a great speedup with a 64 byte buffer, a modest saving with a 128 byte buffer, and a regression generating 14 character passwords with a 256 byte buffer but a modest saving generating 32 character passwords.

Conclusion

We’ve gone from the initial co-pilot generated code with its naive use of new Random() and lack of features, to a couple of performant and secure alternatives. We can now configure minimum symbol guarantees in our generator and we can alter settings to fine-tune the performance for the length of password we want to generate.

If we want to minimise runtime, we can use a reduced character set to avoid having to reject bytes. I would recommend this approach.

Notes

Future improvements

Investigate alternative Crypto-secure Random Number Generators (CSPRNG) or Key Derivation Functions (KDF) for faster generation.

Motivation

You may be thinking:

Password generation shouldn’t happen often enough that even 1ms, let alone 1µs should matter. All this optimization is a waste.

You’d be right if the goal of this was to optimise password generation in production rather than a personal exercise and demonstration of the tooling with an easy to understand example.

I also feel that code quality, including performance, is still important. Two things happen if you let in poor quality code:

1. The code gets copied around, and used outside of its original context.
2. Your team gets used to checking in bad code, and a “Looks good to me” culture.

Password generation almost certainly isn’t a hot path, but who knows when next time someone might need some random data. If this code is hanging around your code-base, it can easily get copied and used in a different context where the performance concerns are more valid.

The second is more subjective, but it’s a matter of pride to work on a team where the first code that co-pilot generated would get picked apart and not accepted. A culture of rubber-stamping PRs can quickly set in if standards aren’t held up and assumptions aren’t checked.

False Optimisations

It’s worth noting here some approaches taken that did not improve performance. I originally wanted to include them in the bulk of the post but I felt it was already getting too long, so they got cut from the final version.

### Replacing the character lookup string with char[]

This actually decreased performance. string in c# are immutable and as such can be treated as a ReadOnlySpan<char> wherever needed, and are fast to index against. There was no improvement to using char[], just a neglible downside.

### Avoiding Modulo with a longer lookup

This one was flat. I had the idea to avoid the modulo by using a repeated string, i.e. instead of:

string characters = "abcdefghjkmnpqrstuwxyzABCDEFGHJKLMNPQRSTVWXYZ0123456789@#$%&()_+";
char randomChar = characters[value % 64];

You would do:

string characters = "abcdefghjkmnpqrstuwxyzABCDEFGHJKLMNPQRSTVWXYZ0123456789@#$%&()_+abcdefghjkmnpqrstuwxyzABCDEFGHJKLMNPQRSTVWXYZ0123456789@#$%&()_+abcdefghjkmnpqrstuwxyzABCDEFGHJKLMNPQRSTVWXYZ0123456789@#$%&()_+abcdefghjkmnpqrstuwxyzABCDEFGHJKLMNPQRSTVWXYZ0123456789@#$%&()_+";
char randomChar = characters[value];

I was hopeful for this one, but it had no appreciable effect on performance.

Avoiding modulo with bittricks / bitshifiting

Other approaches to avoiding modulo such as bitwise & or bitshifting were the same performance as modulo. I didn’t check whether they compiled to the same IL but I’d imagine they did. I’m not smarter than a compiler.

String.Create

I’d heard that rather than new string(buffer), there were faster methods to allocate strings with String.Create. I experimented with what is an awkward method and I couldn’t get it close. That may be down to the very small string values I’m dealing with.

RandomNumberGenerator.GetBytes

Instead of RandomNumberGenerator.Fill you can use RandomNumberGenerator.GetBytes to return the array. This is cleaner syntax but it has no effect on performance. I left RandomNumberGenerator.Fill to make the example have syntax that matched Random.Shared.NextBytes more closely to make it clear they were otherwise the same.

Matching special characters with SearchValues

Instead of !char.IsAsciiLetterOrDigit, I tried System.Buffer.SearchValues with a span of special characters. This added ~20ns to the runtime. ( See Searching.cs for code. )

Addendum

Credits

Graphs generated with https://chartbenchmark.net/

Thank you to members on the csharp discord for reviewing a draft and finding mistakes, posting improvements and suggesting alternative approaches.

Code - Pull Requests Welcome

The code to generate all the results in this post is available at https://github.com/richardcocks/passwordgen

Pull requests are welcomed. If you have improvements or corrections, please send me a PR with the change.

Full comparison table

Here’s a benchmark run with most of the functions, with a single baseline of the original co-pilot output. This run was done before the very final versions of some of the functions, so some minor differences in naming or results may be observed. ( This full run takes well over an hour to generate. )