Integration tests for SSH.NET

[drieseng]

I've finally had the courage to push my (very rough) integration tests for SSH.NET:
https://github.com/sshnet/IntegrationTests

I held back this long because it doesn't don't meet my quality bar.
Even though I automated quite some stuff, there's a lot to be done.
For example:

• Automatically created & configure the regular (non-admin) user that is used to run the tests.
  For now you need to manually create that user, configure its authorized keys, ...
• Consider using docker.
  I mostly use a collection of VMs to be able to test against difference versions of OpenSSH.
• Credentials are hardcoded in the SshNetTests.Users class.
• Hosts are hardcoded in SshNetTests.Hosts class (but by default I use the Ubuntu1910Desktop host).

@IgorMilavec, please have a look. Don't hold back on comments, I'm well aware that I needs a lot of love.

[IgorMilavec]

I had a quick look and... wow, I see you've invested a lot of time into this and I'll need some time to wrap my head around it all. :)
Do you envision the integration tests to be used as part of the release pipeline for SSH.NET?

[drieseng]

Ideally, we should indeed have integration tests to validate our releases. I'm not saying that we should use the tests I wrote, but we could at least borrow some ideas from it.

I'm planning on updating the tests to use a docker container instead of a VM. That should make it easier for you (and others) to use and (I hope) even contribute to it. I'd like to be able to run these tests against multiple versions of OpenSSH and perhaps even other SSH servers, but let's keep it "simple" for now.

In these integration tests, we reconfigure OpenSSH a lot.
Stopping, rebuilding and starting the container for each test is not an option.
I was thinking about having a container with a dummy entrypoint (tail -f /dev/null), and then just stop/reconfigure/start sshd in each test. That way, the container would not have to be restarted at all during the test run.

This is what I currently do in these tests, but then with a VM.
Changing this to use a docker container instead shouldn't take that much time.
The only problem is that there are only 24 hours in a day. Hours that I need to divide between family, work, OSS and very little sleep.
I still have a lot on my to-do list for SSH.NET, Quartz.Net and even old Ninject.
Before your "appearance", I had just about given up on SSH.NET. I don't blame you though :-)
It's great to have you on board!

[drieseng]

Made some good progress today running the test against a docker container. I hope to have it working in some form tomorrow.

[drieseng]

Didn't find much time to work on it, but I got through the "challenging" part today. Remaining work should be straightforward.

[drieseng]

@IgorMilavec @darinkes I've complete the "transformation" to docker for the integrations tests. In these tests I use an assembly from the https://github.com/sshnet/TestTools repo. I've granted both of you access to that repository as well.

I'd like to discuss a few things with you:

• The integration tests solution expects the SshNet.TestTools.OpenSSH project to be in a specific folder (relative to the SshNetTests project). Does this bother you?
  I don't like it much, but it's also not a high priority issue.
  Possible options that I see:
  • Publish SshNet.TestTools.OpenSSH as a NuGet package.
    I'd prefer not to make the package public (yet) as it doesn't meet my quality bar (and it's not really "general" purpose).
    Can we somewhere have a private NuGet feed only accessible to team members? Another option is that each of just just defines a local folder as feed.
  • Integrate SshNet.TestTools.OpenSSH into the IntegrationTests repository.
• The integration tests use version 2020.0.1 of SSH.NET via a NuGet package reference.
  We should be able to run tests against a locally built version of SSH.NET.
  What options do you see for this?
  I prefer not to integrate these tests in the main SSH.NET repository as I want these tests to be able to evolve separately from SSH.NET. I want to be able to add integration tests (or benchmarks) now, but still run them against an old version of SSH.NET.

Personally, I consider these integration tests very important. If you couldn't care less, feel free to say so.
That goes for anything actually. I have a pretty thick skin :-)

[IgorMilavec]

The folder structure is fine for now, this way we will be able to move faster.

I agree that integration tests are important, even more so as I believe some tests are duplicated in existing unit tests. And I really feel we should make unit tests faster to enable quicker developer inner loop.

Before discussing usage of NuGet package to reference SSH.NET in the integration tests, I'd first like to discuss SSH.NET release process. Currently we only have stable version as NuGet package, forcing many users to build their own version off the development branch.
What if we had an automatic deployment pipeline, that would automatically publish a prerelease version of SSH.NET on NuGet, tagged as beta. This way both application developers and "integration testers" could access latest development build by simply updating the package reference.
And for integration testing locally built SSH.NET: one can always create a local repository (in a folder), publish SSH.NET there and again update the package reference in the integration tests. Would this work?

Do you plan at some point to include integration tests into automatic commit tests?
Do you expect that most developers will be running integration tests before submitting PRs?