🐛 Xunit3 TheoryData does not work with MemberAutoData/ClassAutoData

[pinkfloydx33]

Describe the Bug

When working with MemberAutoData or ClassAutoData, Xunit3's TheoryData<> (or TheoryDataRow<>-based classes) no longer work. An exception will be thrown at test execution time indicating that the member or class must implement IEnumerable<object[]>:

Class Data:

System.InvalidOperationException
Data source type "MyProject.Tests.MyClassData" should implement the "System.Collections.Generic.IEnumerable`1[System.Object]" interface.

Member Data:

System.ArgumentException
Member MyMemberData on MyProject.Tests.MyTestClass does not return IEnumerable<object[]>

This works just fine with Xunit2 where TheoryData<> implicitly implements IEnumerable<object[]> via IReadOnlyCollection<object[]>. However TheoryData<> in Xunit3 has been completely revamped and works in terms of TheoryDataRow which is very different.

Scenario

// class data
public class MyClassData : TheoryData<string, string> 
{
    public MyClassData() 
    {
       Add("A", "B");
    }
}

public class TestClass 
{
  // member data
  public static TheoryData<string, string> MyMemberData => new() {
      { "A", "B" }
  };

  [Theory]
  [MemberAutoData(nameof(MyMemberData)]
  public void MyTest(string x, string y, int autodata) {   }

  [Theory]
  [ClassAutoData(typeof(MyClassData)]
  public void MyTest(string x, string y, int autodata) {   }
}

Expected Behavior

Both tests should run, sourcing data from the appropriate member or class and fill in the remaining parameters via AutoFixture

More Information

• .NET 9
• Libraries:
  • xunit.v3: 2.0.3
  • AutoFixture: 5.0.0-preview0012
  • AutoFixture.Xunit3: 5.0.0-preview0012

The ClassData scenario can at least be worked around by implementing IEnumerable<object[]> yourself:

public class MyClassData : TheoryData<string, string>, IEnumerable<object[]>
{
    public MyClassData() 
    {
       Add("A", "B");
    }
    IEnumerator<object[]> IEnumerable<object[]>.GetEnumerator()
    {
        foreach (var r in this)
            yield return [ r.Data.Item1, r.Data.Item2 ];
    }
}

Though this defeats the purpose of using TheoryData

Thanks!

[Ergamon]

Personally I never understood why later AutoFixture.xUnit versions try to mimic the xUnit attributes. Considering the limited resources in this project it feels like a waste of time to reinvent the wheel.

In my own version I stick to the concept of combining the xUnit attribute with AutoFixture. I tested your code with my lib and it works.

I didnt put any effort to split my version from the actual substttuion library (NSubstitute), but I could split it in 2 parts, if interested.

[pinkfloydx33]

@aivascu i somehow missed that AutoFixture.Xunit3 has its own repo. Should I close and reopen this there? (or can it be transfered?)

@Ergamon what lib/project are you referring to?

[Ergamon]

@aivascu i somehow missed that AutoFixture.Xunit3 has its own repo. Should I close and reopen this there? (or can it be transfered?)

Yes it is relativly new. All AutoFixture side repos got their own repo.

@Ergamon what lib/project are you referring to?

I have not published it so far, but about 4 years ago I started with AutoFixture and when I read that I had to write my own version of AutoNSubstituteDataAttribute I made a class library out of it. Over time I started to replace the AutoDataAttribute and derived classes for the reasons I mentioned before.

As said I can upload it to GitHub, there is no company specific code. It should be relativly stable as we run over 20k tests based on it.

I could also try to merge the changes into the official package, if interested.

[aivascu]

Personally I never understood why later AutoFixture.xUnit versions try to mimic the xUnit attributes. Considering the limited resources in this project it feels like a waste of time to reinvent the wheel.

In my own version I stick to the concept of combining the xUnit attribute with AutoFixture. I tested your code with my lib and it works.

@Ergamon could you elaborate on combining the xUnit attributes with AutoFixutre? I'm trying to understand if there's an alternative option to implement the integration, and I'm just wasting time on re-implementing the attributes.
If you have some sources you can share they'd be welcome.

[aivascu]

@aivascu i somehow missed that AutoFixture.Xunit3 has its own repo. Should I close and reopen this there? (or can it be transfered?)

@pinkfloydx33 no worries, that's on me, I didn't mention it in the readme. I'll transfer the issue in a minute.

[Ergamon]

I started to port my changes into the existing code to demonstrate what I meant.

How free are we currently to make modifications? I would change the C# language version to 12 and in my opinion target frameworks netstandard2.0 and net8.0.
(As a matter of fact I build my library only with netstandard2.0 as we have in house only .NET Framwork 4.8 and .NET 9.0 projects)

For the netstandard2.0 is it preferred to include the necessary files into the source or is it acceptable to add a dependency to e.g. PolySharp?

[aivascu]

@Ergamon if we're talking about the xUnit 3 repo then we need to support at least the same target frameworks as the xUnit library, meaning netstandard2.0 (same as the extensibility core), and if we want to provide framework specific binaries to get the latest IL/CLR optimizations then net8.0 and .NET Framework 4.7.2 (same as the current xunit 3 main library).

In terms of C# language version, it can be as new as necessary, as long as it builds against the target frameworks, and there are no autogenerated source files, that can conflict with the minimal C# version for the earliest target framework (e.g. netstandard2.0 -> C# 7.3 source).
One note of warning here, the syntax validation rules might need to be updated to support the later C# versions.

Regarding implementation and dependencies, the library should remain free of third-party packages as much as possible, especially those that can lead to provider lock-in (e.g. Newtonsoft.Json). Also it's another library version that I'd have to manage, which I want to avoid.
If you're going to borrow code from other libraries, then it should be released under licenses at least as permissive as MIT, since this is the license AutoFixture libraries are going to redistribute the code under.
On PolySharp specifically, I'll be honest I haven't used it, but it seems to be a convenience library that most code could do without. I would consider taking a dependency on a library that for example implements a complicated algorithm, that I would not be able to maintain myself, like Fare for example, but polyfills I'd rather have embedded if possible.

Also due to SemVer, any changes to the target frameworks will go toward the next major version, which is not tied to the base AutoFixture library version anymore, but still must be taken into consideration.