Property Based Testing By Example

Steve Love

@IAmSteveLove@mastodon.social
@stevelove.bsky.social
https://moderncsharp.blogspot.com/
@IAmSteveLove

https://www.arventech.com/ — steve@arventech.com

OOP 2025

Please leave feedback! :-)

Testing—Augmented

Property-based testing for .NET programmers

Example-based testing

Property-based testing

Generalization

Generating data

Triangulation

A bit of the other

Guess a word

STARE

The game replies…

STARE

Breaking it down

SHORT

STARE

0 → 0 : S
1 → 4 : T
2 → x : A
3 → 3 : R
4 → x : E

[ 0, 4, -1, 3, -1 ]

A naive attempt

public static int[] Match(string word, string guess) 
{
    return guess.Select((ch, idx) 
            => ch == word[idx] ? idx    // prefer exact match 
                : word.IndexOf(ch))     // returns -1 if not found
        .ToArray();        
}

We can test for that!

[Test]
public void Simple_eg_test()
{
    var (word, guess) = ( "SHORT", "STARE" );
    
    var result = Match(word, guess);
    int[] expected = [0, 4, -1, 3, -1];
    
    Assert.That(result, Is.EqualTo(expected));
}

A more general idea

Matching a string with itself
should always produce the same output

a.k.a. Reflexivity

SHORT

[ 0, 1, 2, 3, 4 ]

We can test for that, too!

[Test]
public void A_string_matches_itself_exactly()
{
    string word = "ANYOLDSTRING";
    
    var result = Match(word, word);
    var expected = Enumerable.Range(0, word.Length);
    
    Assert.That(result, Is.EqualTo(expected));
}

Another general test

Strings with no matches have another predictable result:

SHORT → GUNGE

[ -1, -1, -1, -1, -1 ]

Test Parameters

[TestCase("SHORT", "GUNGE")]
[TestCase("GUNGE", "SHORT")]
public void Disjoint_strings_have_no_matches(string word, string guess)
{
    var result = Match(word, guess);
    var expected = Enumerable.Repeat(-1, word.Length);   // every element
                                                         // is -1
    Assert.That(result, Is.EqualTo(expected));
}

Ask Copilot…

[TestCase("apple", "apple", new[] { 0, 1, 2, 3, 4 })]
[TestCase("apple", "apric", new[] { 0, 1, 2, -1, -1 })]
[TestCase("apple", "ppale", new[] { 1, 0, 2, 3, 4 })]
[TestCase("apple", "leapp", new[] { 3, 4, 0, 1, 2 })]
[TestCase("apple", "xxxxx", new[] { -1, -1, -1, -1, -1 })]
public void Match_ShouldReturnExpectedResults(string word, string guess, int[] expected)
{
    var result = Match(word, guess);
    Assert.AreEqual(expected, result);
}

Not bad

…although it doesn’t compile

Tests are about confidence…

[TestCase("apple", "apple", new[] { 0, 1, 2, 3, 4 })]
[TestCase("apple", "apric", new[] { 0, 1, 2, -1, -1 })]
[TestCase("apple", "ppale", new[] { 1, 0, 2, 3, 4 })]
[TestCase("apple", "leapp", new[] { 3, 4, 0, 1, 2 })]
[TestCase("apple", "xxxxx", new[] { -1, -1, -1, -1, -1 })]
public void Match_ShouldReturnExpectedResults(string word, string guess, int[] expected)
{
    var result = Match(word, guess);
    Assert.That(result, Is.EqualTo(expected));
}

Includes our 3 test cases
but 2 of these fail:

apric and ppale

A more specific test

Each matched letter is matched exactly once

SHORT → STARE

[ 0, 4, -1, 3, -1 ]

Another test example

[TestCase("SHORT", "STARE")]
[TestCase("STARE", "SHORT")]
public void Matched_indices_are_unique(string word, string guess)
{
    var result = Match(word, guess);
    
    var matches = result.Where(i => i != -1).ToArray();
    
    Assert.That(matches, Is.Unique);
}

So far, so good

Reflexivity
no matches
uniqueness

Hypothesis: These are properties that should apply to any arbitrary string with respect to the Match method

Adding new cases

[TestCase("GRASP", "STASH")]
[TestCase("SHORT", "STARE")]
[TestCase("STARE", "SHORT")]
public void Matched_indices_are_unique(string word, string guess)
{
    var result = Match(word, guess);
    
    var matches = result.Where(i => i != -1).ToArray();
    
    Assert.That(matches, Is.Unique);
}

Unexpected failure?

    Failed Matched_indices_are_unique("GRASP","STASH") [103 ms]
    Error Message:
        Assert.That(matches, Is.Unique)
        Expected: all items unique
        But was:  < 3, 2, 3 >
        Not unique items: < 3 >

Breaking it down again

GRASP

STASH

Should have been:

[ -1, -1, 2, 3, -1 ]

But was:

[ 3, -1, 2, 3, -1 ]

Revisit the code

public static int[] Match(string word, string guess) 
{
    return guess.Select((ch, idx) 
            => ch == word[idx] ? idx    // prefer exact match 
                : word.IndexOf(ch))     // returns -1 if not found
        .ToArray();        
}

GRASP

Should be:

STASH

Actually:

STASH

Precise requirements

A letter in the set word should be matched by only one letter in the guess, with an exact match being preferred

The Copilot tests

Actually caught this bug with one of the examples

[TestCase("apple", "ppale", new[] { 1, 0, 2, 3, 4 })]

…but for the wrong reason!:

[2, 1, 0, 3, 4] →

ppale

A new implementation

public static int[] Match(string word, string guess) 
{
    var found = Enumerable.Range(0, word.Length)
        .Select(idx => word[idx] == guess[idx] ? idx : -1)
        .ToArray();
    
    foreach(var i in Enumerable.Range(0, word.Length))
    {
        if(found[i] == -1)
        {
            found[i] = Enumerable.Range(0, word.Length)
                .FirstOrDefault(idx => word[idx] == guess[i] 
                       && !found.Contains(idx), -1);
        }
    }

    return found;
}

How do we test the new version?

Our example-based tests now pass…

…but we identified properties that should hold for all strings

Ask Copilot (again)

[Test]
public void Match_AllExactMatches_ReturnsCorrectIndices()
{
    // Arrange
    string word = "hello";
    string guess = "hello";

    // Act
    var result = Match(word, guess);

    // Assert
    Assert.That(result, Is.EqualTo(new[] { 0, 1, 2, 3, 4 }, result));
}

[Test]
public void Match_SomeExactMatches_ReturnsCorrectIndices()
{
    string word = "hello";
    string guess = "hxllo";

    var result = Match(word, guess);

    Assert.AreEqual(new[] { 0, -1, 2, 3, 4 }, result);
}

[Test]
public void Match_NoExactMatches_ReturnsCorrectIndices()
{
    string word = "hello";
    string guess = "xxxxx";

    var result = Match(word, guess);

    Assert.AreEqual(new[] { -1, -1, -1, -1, -1 }, result);
}
[Test]
public void Match_PartialMatches_ReturnsCorrectIndices()
{
    string word = "hello";
    string guess = "olelh";

    var result = Match(word, guess);

    Assert.AreEqual(new[] { 4, 3, 2, 1, 0 }, result);
}
[Test]
public void Match_EmptyStrings_ReturnsEmptyArray()
{
    string word = "";
    string guess = "";

    var result = Match(word, guess);

    Assert.AreEqual(new int[] { }, result);
}

Fixed

[TestCase("hello", "hello", new[] { 0, 1, 2, 3, 4 })]
[TestCase("hello", "hxllo", new[] { 0, -1, 2, 3, 4 })]
[TestCase("hello", "xxxxx", new[] { -1, -1, -1, -1, -1 })]
[TestCase("hello", "olelh", new[] { 4, 3, 2, 1, 0 })] // 4, 2, 1, 3, 0
[TestCase("", "", new int[] { })]
public void Match_AllExactMatches_ReturnsCorrectIndices(string word, string guess, int[] expected)
{
    var result = Match(word, guess);
    Assert.That(result, Is.EqualTo(expected));
}

…but still fails as shown

…and fails in the same way for the original implementation

Property-based testing

Define properties that should hold for all inputs…
…which may be (often are) generated randomly

Identifies counter-examples that don’t hold
Exposes corner-cases you haven’t thought of

Enter FsCheck

Property-based testing for .NET

Property

a function that tests an assumption. Either:

a predicate—a false return indicates a falsifiable case
a unit (void) method—thrown exception is a falsifiable case

Common Properties

Invariants - sorted list insertion
Inversion - reverse, reverse again
Commutativity - x == y, y == x
Round-trip - LCM(x,y) % GCD(x,y) always 0
Idempotency - call Dispose

…but sometimes it’s useful to just be able to test all the things!

Testing is all about confidence

[FsCheck.NUnit.Property]
public void Any_string_matches_itself_exactly(string word)
{
    var result = Match(word, word);
    var expected = Enumerable.Range(0, word.Length);
    
    Assert.That(result, Is.EqualTo(expected));
}

        Falsifiable, after 32 tests (0 shrinks) (StdGen (1595582682,297278030)):
        Original:
        <null>
        with exception:
        System.NullReferenceException:
            Object reference not set to an instance of an object.

Filtering

[FsCheck.NUnit.Property]
public void Any_string_matches_itself_exactly(NonEmptyString word)
{
    var result = Match(word.Get, word.Get);
    var expected = Enumerable.Range(0, word.Get.Length);
    
    Assert.That(result, Is.EqualTo(expected));
}

A little harder to test

Uniqueness of matched indices: Requires guess to be at least as long as the set word (ideally they are equal length).
Disjoint strings: Requires two strings with no matching characters.

Some FsCheck terminology

Gen<T>: generate instances—usually random—of T
Arbitrary<T>: supplies data to a property parameter, uses Gen<T> along with a shrinker

Generating strings of fixed length

class ArbitraryStrings
{
    private static Gen<string> StringOfLength(int n)
        => Arb.Default.Char().Generator     // 'char' gen as shipped with FsCheck
            .Where(char.IsLetterOrDigit)    // for readability
            .ArrayOf(n)                     // fixed size array of char
            .Select(string.Concat);         // create a string

    public static Arbitrary<string> FixedLengthString()
        => StringOfLength(15).ToArbitrary();
}

Unique matched indices property

[FsCheck.NUnit.Property(Arbitrary = [ typeof(ArbitraryStrings) ])]
public void Matched_indices_are_unique(string word, string guess)
{
    var result = Match(word, guess);
    
    var matches = result.Where(i => i != -1).ToArray();
    
    Assert.That(matches, Is.Unique);
}

Back testing

Our original implementation fails this test:

    Falsifiable, after 4 tests (0 shrinks) (StdGen (1105325589,297278101)):
    Original:
    ("yW606jYvi8Bd7P2", "Z7QjVj4oJcycyct")
    with exception:
    NUnit.Framework.AssertionException:   Assert.That(matches, Is.Unique)
        Expected: all items unique
        But was:  < 12, 5, 5, 0, 0 >
        Not unique items: < 5, 0 >

We can use this output to write a concrete example-based test!

Generating disjoint strings

public static Arbitrary<(string, string)> DisjointStringPair()
    // generate a Tuple<string, string>
    => StringOfLength(15).Two()        
        // eliminate pairs with any matching character
        .Where(pair => !pair.Item1.Intersect(pair.Item2).Any())
        // transform to value tuple
        .Select(pair => (pair.Item1, pair.Item2))
        .ToArbitrary();

Filtering like this can be expensive.

Optimizing input data may be more efficient.

Zero matches property

[FsCheck.NUnit.Property(Arbitrary = [ typeof(ArbitraryStrings) ])]
public void Disjoint_strings_have_no_matched_indices((string, string) pair)
{
    var (word, guess) = pair;
    var result = Match(word, guess).Where(i => i != -1);
    
    result.Should().BeEmpty();
}

or…

Assert.That(result, Is.Empty);

Observation

We discovered our original counter-example manually
but the property based test for unique matched indices
would have found it.

Any other properties?

More on generating data

The Oldies are the Besties

public static class Fizzer
{
    public static string Buzz(int number)
        => number switch
        {
            _ when number % 15 == 0 => "FizzBuzz",
            _ when number % 3 == 0 => "Fizz",
            _ when number % 5 == 0 => "Buzz",
            _ => number.ToString()
        };
}

All the things

[FsCheck.NUnit.Property]
public void All_results_are_valid_outcomes(PositiveInt n)
{
    Fizzer.Buzz(n.Get).Should().BeOneOf(n.ToString(), "Fizz", "Buzz", "FizzBuzz");
}

FizzBuzz is traditionally for numbers 1..100
…but only because humans are easily bored!

Ask Copilot (again)

[TestCase(15, "FizzBuzz")]
[TestCase(30, "FizzBuzz")]
[TestCase(5, "Buzz")]
[TestCase(10, "Buzz")]
[TestCase(3, "Fizz")]
[TestCase(6, "Fizz")]
[TestCase(1, "1")]
[TestCase(2, "2")]
[TestCase(4, "4")]
[TestCase(7, "7")]
public void Buzz_ReturnsExpectedResult(int input, string expected)
{
    // Act
    var result = Fizzer.Buzz(input);

    // Assert
    Assert.That(result, Is.EqualTo(expected));
}

Properties of FizzBuzz

Every third result contains "Fizz"

Every "Fizz" comes from a multiple of 3

Every fifth result contains "Buzz"

Every "Buzz" comes from a multiple of 5

Every fifteenth result is "FizzBuzz"

Every "FizzBuzz" comes from a multiple of both 3 and 5

Every other number corresponds to its ordinal position

https://accu.org/conf-docs/PDFs_2016/KevlinHenney_DeclarativeThinking,DeclarativePractice.pdf

An unsatisfactory test

[FsCheck.NUnit.Property]
public void All_divisible_by_3_are_Fizz(int n)
{
    if(n % 3 == 0)
        Assert.That(Fizzer.Buzz(n), Does.StartWith("Fizz"));
}

Can we do better?

Generated parameters

[FsCheck.NUnit.Property(Arbitrary = [ typeof(FizzBuzzNumbers) ] )]
public void All_divisible_by_3_are_Fizz(IntDivisibleBy3 n)
{
    Fizzer.Buzz(n).Should().StartWith("Fizz");
}

or, being more specific

var expected = n switch
{
    _ when n % 5 == 0 => "FizzBuzz",
    _ => "Fizz",
};
Fizzer.Buzz(n).Should().Be(expected);

Generate the numbers

public readonly record struct IntDivisibleBy3(int Val)
{
    public static implicit operator int(IntDivisibleBy3 i) => i.Val;
}

static class FizzBuzzNumbers
{
    static Gen<int> DivBy(int div) 
        => Gen.Choose(1, int.MaxValue)
            .Select(n => n + div - n % div);

    public static Arbitrary<IntDivisibleBy3> Number3() 
        => DivBy(3).Select(n => new IntDivisibleBy3(n)).ToArbitrary();
}

Define the type IntDivisibleBy3 to distinguish from int
Define the generator to register in the test for IntDivisibleBy3 types

What about the others?

The essential ideas are the same for 5 and 15.

Generating numbers not divisible by 3 or 5 is harder…

static Gen<int> NotDivBy(int div) 
    => Gen.Choose(1, int.MaxValue)
        .Select(n => n % div == 0 ? n + 1 : n);

public static Arbitrary<NonFizzBuzzNumber> NumberOther() 
    => NotDivBy(3).Where(n => n % 5 != 0)
        .Select(n => new NonFizzBuzzNumber(n)).ToArbitrary();

Testing ordinal position

[FsCheck.NUnit.Property(Arbitrary = [ typeof(FizzBuzzNumbers) ] )]
public void All_numeric_results_correspond_to_original_value(NonFizzBuzzNumber n)
{
    Fizzer.Buzz(n).Should().Be(n.Val.ToString());
}

The remaining tests

[FsCheck.NUnit.Property]
public void All_results_match_ordinal_position(PositiveInt p)
{
    var n = p.Get;
    var expected = Fizzer.Buzz(n) switch
    {
        "FizzBuzz" => n % 15,
        "Fizz"     => n % 3,
        "Buzz"     => n % 5,
        var v      => int.Parse(v) - n,
    };

    expected.Should().Be(0);
}

FsCheck is F#

[< Property >]
let ``All results match ordinal position`` (n: PositiveInt) =
    match Fizzer.Buzz n.Get with
    | "Fizz"      -> int n % 3  |> should equal 0
    | "Buzz"      -> int n % 5  |> should equal 0
    | "FizzBuzz"  -> int n % 15 |> should equal 0
    | v -> n.ToString() |> should equal v

Testing in a foreign language

module WordTests =
    
    [< Property >]
    let ``Any string matches itself exactly``(word: NonEmptyString) =
        Match(word.Get, word.Get)
        |> should equalSeq [| 0..word.Get.Length - 1 |]

    [< Property(Arbitrary=[| typeof<ArbitraryStrings> |]) >]
    let ``Matched indices are unique``(word: string, guess: string) =
        Match(word, guess)
        |> Array.filter(fun i -> i <> -1)
        |> should be unique

Custom generator

type ArbitraryStrings =
    static member StringOfLength n =
       Arb.generate<char>
       |> Gen.filter Char.IsLetterOrDigit
       |> Gen.arrayOfLength n
       |> Gen.map String.Concat

    static member FixedLengthString =
        ArbitraryStrings.StringOfLength(15)
        |> Arb.fromGen

class ArbitraryStrings
{
    private static Gen<string> StringOfLength(int n)
        => Arb.Default.Char().Generator     // 'char' gen as shipped with FsCheck
            .Where(char.IsLetterOrDigit)    // for readability
            .ArrayOf(n)                     // fixed size array of char
            .Select(string.Concat);         // create a string

    public static Arbitrary<string> FixedLengthString()
        => StringOfLength(15).ToArbitrary();
}

Necessity & sufficiency

Reflexive, no matches, unique indices

Definitely all necessary
- the latter identified a counter-example in the original algorithm
How to determine sufficiency?

If there’s time….

One more interesting property of the word matcher…

Reciprocity

SHORT

STARE

[ 0, 4, -1, 3, -1 ]

STARE

SHORT

[ 0, -1, -1, 3, 1 ]

Quite interesting…

SHORT → STARE

STARE → SHORT

[ 0, 4, -1, 3, -1 ]

[ 0, -1, -1, 3, 1 ]

(0, 0)
(1, 4)
(3, 3)

(0, 0)
(3, 3)
(4, 1)

Hypothesis: it’s true for all matches between strings of the same length

Test switched words

[< Property(Arbitrary=[| typeof<ArbitraryStrings> |]) >]
let ``Matched indices roundtrip in switched words``(word: string, guess: string) =
    let orig = Match(word, guess) |> Seq.ofArray
    let rev = Match(guess, word) |> Seq.ofArray
    
    let inOrder = Seq.zip orig { 0..word.Length - 1 }
                |> Seq.where(fun(x, y) -> x <> -1)
                
    let switched = Seq.zip rev { 0..guess.Length - 1 } 
                |> Seq.where(fun(x, y) -> x <> -1)
                |> Seq.map(fun(x, y) -> y, x)

    inOrder |> should equivalent switched

inOrder:  (0, 0), (1, 4), (3, 3)
switched: (0, 0), (3, 3), (4, 1)

Property Based Testing

Finding all those edge cases you forgot!

.NET tools

FsCheck

https://fscheck.github.io/FsCheck
https://fsharpforfunandprofit.com/pbt/

CsCheck

https://github.com/AnthonyLloyd/CsCheck

Hedgehog

https://github.com/hedgehogqa/fsharp-hedgehog

FsUnit

https://fsprojects.github.io/FsUnit/

Fluent Assertions

https://fluentassertions.com/

Thank you for listening!

Out now

https://nostarch.com/c-type-system	https://pragprog.com/titles/csharpbt/c-brain-teasers/

Out now

csts qrcsts

csbt qrcsts

https://nostarch.com/c-type-system

https://pragprog.com/titles/csharpbt/c-brain-teasers/

https://www.arventech.com/ — steve@arventech.com

Property Based Testing By Example

OOP 2025

Testing—​Augmented

Guess a word

The game replies…​

Breaking it down

A naive attempt

We can test for that!

A more general idea

We can test for that, too!

Another general test

Test Parameters

Ask Copilot…​

Tests are about confidence…​

A more specific test

Another test example

So far, so good

Adding new cases

Unexpected failure?

Breaking it down again

Revisit the code

Precise requirements

The Copilot tests

A new implementation

How do we test the new version?

Ask Copilot (again)

Fixed

Property-based testing

Enter FsCheck

Common Properties

Testing is all about confidence

Filtering

A little harder to test

Some FsCheck terminology

Generating strings of fixed length

Unique matched indices property

Back testing

Generating disjoint strings

Zero matches property

Observation

More on generating data

All the things

Ask Copilot (again)

Properties of FizzBuzz

An unsatisfactory test

Generated parameters

Generate the numbers

What about the others?

Testing ordinal position

The remaining tests

FsCheck is F#

Testing in a foreign language

Custom generator

Necessity & sufficiency

If there’s time…​.

Reciprocity

Quite interesting…​

Test switched words

Property Based Testing

.NET tools

Thank you for listening!

Testing—Augmented

The game replies…

Ask Copilot…

Tests are about confidence…

If there’s time….

Quite interesting…