Test suite for `cabal` the binary should be isolated/independent

[hasufell]

Currently, cabal-testsuite (which seems to be the main way cabal the binary is tested) cannot be used as a stand-alone binary to run the tests.

The reasons seem to be:

• it requires sources that are part of the repository
• there is a generated module that depends on the current cached LocalBuildInfo and hardcodes it into the binary, which then is used to reconstruct arguments to ghc when compiling tests
  • it's not clear to me whether this is only required for the Setup.hs tests or the cabal-install binary tests as well (e.g. there is --skip-setup-tests but my observation is that this isn't really enough)

The end goal I have is:

• build and test is strictly separated in CI and the test step does not involve building anything other than the tests
• tests can be run on the end users system (ultimately through GHCup)

[mpickering]

@hasufell Do you have a proposal for how to architecture the testsuite differently to enable this?

Currently the cabal-testsuite binary dynamically discovers what tests to run and then interprets the test scripts to execute the tests. Therefore you need a suitable environment for interpreting the test scripts (which is what the Setup.hs does).

You could imagine having a separate script like the bootstrap script which installed the necessary test dependencies into a specific package database This complicates the packaging story in other ways, having to provide a suitable testsuite tarball with these dependencies as well.

[hasufell]

You could imagine having a separate script like the bootstrap script which installed the necessary test dependencies into a specific package database

I could have a go at it, but I'm still not too familiar with the whole test harness. Is the package DB the only reason we extract LocalBuildInfo?

From what I see there's this:

cabal/cabal-testsuite/Setup.hs

Lines 32 to 66 in 5e1fd2a

	rewriteFileEx verbosity (moduledir </> "ScriptEnv0.hs") $ unlines
	[ "{-# LANGUAGE OverloadedStrings #-}"
	, "{-# LANGUAGE FlexibleInstances #-}"
	, "{-# OPTIONS_GHC -Wno-orphans #-}"
	, "module Test.Cabal.ScriptEnv0 where"
	, ""
	, "import Distribution.Simple"
	, "import Distribution.System (Platform(..), Arch(..), OS(..))"
	, "import Distribution.Types.ModuleRenaming"
	, "import Distribution.Simple.Program.Db"
	, "import Distribution.Backpack (OpenUnitId)"
	, "import Data.Map (fromList)"
	, "import Data.String (IsString(..))"
	, "import Distribution.Utils.Path"
	, ""
	, "lbiPackageDbStack :: PackageDBStackCWD"
	, "lbiPackageDbStack = " ++ show lbiPackageDbStack
	, ""
	, "lbiPlatform :: Platform"
	, "lbiPlatform = " ++ show (hostPlatform lbi)
	, ""
	, "lbiCompiler :: Compiler"
	, "lbiCompiler = " ++ show (compiler lbi)
	, ""
	, "lbiPackages :: [(OpenUnitId, ModuleRenaming)]"
	, "lbiPackages = read " ++ show (show (cabalTestsPackages lbi))
	, ""
	, "lbiProgramDb :: ProgramDb"
	, "lbiProgramDb = read " ++ show (show (withPrograms lbi))
	, ""
	, "lbiWithSharedLib :: Bool"
	, "lbiWithSharedLib = " ++ show (withSharedLib lbi)
	, ""
	, "instance IsString (SymbolicPath from to) where"
	, " fromString = makeSymbolicPath"

[mpickering]

@hasufell Yes, I believe the main reason is to use cabal in order to provision the correct package database to interpret the scripts. I would have to investigate the other uses of Setup.

Some things to keep in mind.

• Running cabal-tests multiple times in a row shouldn't recompile all test dependencies (local development workflow)
• The test environment should be provisioned with the system cabal, not the cabal under test.
• The "external dependencies" which are allowed to be used by test scripts are listed at the moment in cabal-testsuite.cabal in the test-runtime-deps stanza.