Clarify the situation around Data.List

[mpickering]

In https://gitlab.haskell.org/ghc/ghc/-/issues/20025 it was observed that the proposed changes to monomorphise Data.List were not well communicated with the community and many people were surprised at the change. It was therefore decided to revert the patch whilst the CLC proposals system was set-up, it has now been set-up so the CLC should decide on the plan for this issue.

The plan articulated by @chessai was as follows:

1. Revert for now
2. Wait for -Wcompat to become part of -Wall (which is happening and will be a part of 9.4)
3. Wait for 1+ GHC releases, and announce extensively in the meantime
4. Re-introduce the change
5. Provide a retrie command for automatic refactoring (see Feasibility of using retrie to refactor for Data.List monomorphisation facebookincubator/retrie#26)

I am executing part 1 of this plan currently (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/6409)

The new CLC need to agree to the rest of this plan and communicate the result with the community.

[Bodigrim]

We briefly discussed the issue on one of CLC meeting back in October.

According to @chessai, CLC has approved monomorphisation of Data.List back in 2019. I think the process was abysmal: the vote was not recorded publicly, the decision was not announced clearly, no justification was circulated and no migration plan has been prepared at all. Nevertheless, it would set a bad precedent if current CLC just dismisses its earlier decision, without a material change of circumstances invalidating justification.

Since there is no agreed and voted migration plan, besides a draft discussed between @chessai, @emilypi and @bgamari, GHC developers may revert the change from GHC HEAD for now.

CLC will work on migration plan, but it is likely to take time and will not happen overnight. We will communicate it to GHC developers and to the community when ready.

[goldfirere]

I'm fine with the plan above -- though it sounds like "We will communicate it ..." suggests that the CLC is going to set up a migration plan without community input. Maybe that's fine: we, as a community, do not tend to address issues like this efficiently.

But I want to speak up to say we can make this less painful by adding a new feature to the language: narrowing re-exports. I will illustrate with an example:

module Poly ( id ) where

id :: a -> a
id x = x

module Mono ( id :: Int -> Int ) where
import Poly

module A where
import Poly

x = id 'x'   -- accepted
y = id undefined   -- accepted; y :: forall a. a is inferred

module B where
import Mono

x = id 'x'   -- rejected: id has type Int -> Int
y = id undefined  -- accepted; y :: Int is inferred

module C where
import Mono
import Poly

x = id 'x'   -- accepted
y = id undefined   -- accepted; y :: forall a. a is inferred

The idea is that an export list can narrow the type of an export. If only the narrowed type is imported, that's the type of the identifier. If the narrowed type and the original type are both imported, then the identifier has the original type. If two different narrowed types are imported, a use of the identifier is rejected as ambiguous (not exemplified above); one could imagine some scheme looking for a more general type, but I think that's more complicated than it's worth.

Using narrowing re-exports, Data.List could simply re-export many identifiers with narrower types. This would have the following changes, compared to the status quo:

• Data.List would be fully monomorphic, including in its Haddocks.
• import Data.List would not introduce any ambiguous identifiers.
• The change is still backward-incompatible: if a user imports a polymorphic function from Data.List (and does not have the polymorphic function imported via other means), that function would now be monomorphic. (It would be easy, though, to suggest where to find the polymorphic version in an error message.)

Compared to the plan to make Data.List monomorphic without using narrowing re-exports, the narrowing re-exports plan prevents much of the backward incompatibility, especially for those users who do not use an import list.

Narrowing re-exports might be useful elsewhere, though it's not clear quite how useful they would be. Is the benefit here worth the cost? It's not clear to me. If this narrowing re-exports feature were very widely applicable, that would be exciting... but I don't think the feature would be used much. So it does seem like a largish language-level change for a smallish use case. I wanted to float the idea, regardless, to see if anyone else thought it a better way forward.

[Bodigrim]

though it sounds like "We will communicate it ..." suggests that the CLC is going to set up a migration plan without community input

Sorry that it made such impression. This was a mere statement that CLC is responsible to get it done, not that it will work on it behind closed doors.

The change is still backward-incompatible: if a user imports a polymorphic function from Data.List (and does not have the polymorphic function imported via other means), that function would now be monomorphic.

Given that -Wcompat already urges users to import Data.List qualified, breakage of polymorphic functions imported from Data.List is likely to be the biggest migration concern. If narrowed re-exports do not resolve it, I'm not sure they are worth implementing.

[Bodigrim]

As discussed on the last CLC meeting back in February, current intention is to proceed with the migration plan as described in #22 (comment). However, given that this is a breaking and disruptive change, we would like to seek community's feedback on the proposed approach. The floor is open, please be civil.

[tomjaguarpaw]

But what actually are the proposed changes? I mean, I can guess, but others less familiar with the situation may have no clue. Can we link to an accepted proposal or at least a statement of the proposal and some discussion? (Since the proposal predates the current GitHub-based CLC process I suppose the link would be to a mailing list archive.) Although I can guess what this proposal means I don't know who proposed it, what arguments were made for and against and why the decision was made as it was. I found some justification but it would be good to have that fleshed out.

I apologise if I've missed the obvious here, but I've been clicking between the GitHub and GitLab issues trackers for five minutes without gaining clarity.

[Bodigrim]

@tomjaguarpaw the change is in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5304

[jappeace]

why would you monomorphize data.list ?

[Bodigrim]

@jappeace the justification, used by the previous composition of CLC, is here: https://gitlab.haskell.org/ghc/ghc/-/issues/20025#note_365364

[jappeace]

Nice, thanks, makes sense. 👍

[JakobBruenker]

Is it likely that changing imports for the affected bindings from Data.List to Data.Foldable would be sufficient to fix any issues arising from this change?

[phadej]

Is it likely that changing imports for the affected bindings from Data.List to Data.Foldable would be sufficient to fix any issues arising from this change?

Yes, except if you actually use list specific functions like intercalate, sort, nub or isPrefixOf etc. I.e. In general you'd change

import Data.List

to

import Data.Foldable (foldl')
import Data.List (sort)

or

import Data.Foldable (foldl')
import qualified Data.List as L

... sort to L.sort...

[runeksvendsen]

As discussed on the last CLC meeting back in February, current intention is to proceed with the migration plan as described in #22 (comment). However, given that this is a breaking and disruptive change, we would like to seek community's feedback on the proposed approach. The floor is open, please be civil.

Please don't do this. These sorts of "cleanup" changes make perfect sense in ones own personal packages, but not for a package like base whose API is depended on by thousands of packages. I know I have no vote in this, but I would like to voice my opinion that being able to use some old Haskell package I find on GitHub five years down the road with a newer GHC version is worth much more to me than an attempt to "[make] Haskell easier to learn" [1].

I'm an application developer, and the first step for me when evaluating the amount of work required to create a Haskell application is to go look for a package either on Hackage or (more often) on GitHub that does what I want. I very often find some old Haskell library on GitHub that does what I want but hasn't been updated in several years. Changes like the one suggested here increases the amount of work needed to use libraries like these, thus directly increasing the amount of work associated with developing applications in Haskell compared to other languages that have more users and therefore more libraries that remain maintained years down the road.

Also, importantly, I would like to point out that the practice of having the CLC produce patches for the newest version of the affected packages because of a breaking change to base does practically nothing to fix the issue I described above, since the only way a five year old Haskell library can make use of this patch is to update the given dependency to the newest version, which usually requires updating all other dependencies to their newest version as well (since they transitively depend on each other), which is a huge task.
Thus, only if this policy, of providing patches for libraries affected by breaking changes to base, is modified to include backporting changes to all released versions of the given package (or at least the released versions depended on by old libraries in the wild) would it actually solve the problem. This, however, would place a much greater burden on the CLC when adopting backwards-incompatible changes, but it would more accurately reflect the amount of work required by other developers down the road.

Consequently, if the CLC is genuinely interested in shifting to themselves the burden created by backwards-incompatible changes to base, then the practice of requiring patches for the newest version of affected packages would be changed to:

1. Go look for Haskell packages on the internet: on Hackage, GitHub, GitLab, Bitbucket, and privately hosted Gitea/Gitweb/cgit/etc. instances
2. Enumerate all of these packages that depend on some version (either directly or transitively) of the affected packages
3. Produce patches for all the versions of the affected packages that are depended on by the packages enumerated in step 2

To be clear, I'm not categorically against breaking changes to base. I just see how they make it more difficult for me to write applications in Haskell. And in the case of the kind of the breaking change suggested here — which I would put in the category of "cleanup" — they have a very real and quantifiable negative impact on the development of Haskell applications, and a much less real and quantifiable benefit. The main argument for the change, brought forward in [1], is that it makes Haskell easier to learn. But to what extent this is the case is more or less unknown. My opinion is that the degree to which this change makes Haskell easier to learn, and the impact of this on actually learning Haskell, does not outweigh the cost of increasing the amount of work required to use older Haskell packages with newer GHC versions.

[Ericson2314]

@runeksvendsen You might be interested in https://discourse.haskell.org/t/pre-hftp-ghc-x-hackage-a-tool-for-easing-migrations-to-new-ghc-versions/4239 / https://github.com/bgamari/tech-proposals/blob/ghc-x-hackage/proposals/001-ghc-x-hackage.md

I want this change, and for me the breakage you describe is real but also a pointless tragedy. The above proposal is a very good stop-gap solution to this; the breakage this causes is quite trivial to fix, and so if we can crowd-source fixes to each other's packages it should be not to hard to fix the issues. But this should also be but the first step, there are many more thing we can do such as support the old and new Data.List interfaces at the same time (there are no deep incompatibles like changed instances here).

One argument might be to fix make the solutions, then agree to do the breakage. (Remember, there is still a deprecation cycle here.) That sounds nice, but history doesn't pain this method in a good like, because the vast majority of languages to avoid breakage until tools are available just end up stagnating (See https://twitter.com/__phantomderp/status/1510309539538690055 and really @ThePhD's entire mission with C.)

So given the bias towards stagnation, affirming decisions like this, with long road maps, while concurrently making better tools to deal with cruft and breakage and hopefully accelerate those timelines, I think is the only path forward. It may not have much precedent, but at least it doesn't have precedent of failure.

[tomjaguarpaw]

I wonder why @runeksvendsen is the only person so far asking for long-term backwards compatibility. Are there no others that want it? Do they not know of this discussion? [1] Do they want it, and know of this discussion, but they're too burnt out arguing against breaking changes?

Personally I like cleaning things up, and this change will probably cause me personally some breakage, but not too much. On the other hand, I don't like the idea that the people who are going to be most resentful of this change are just too frustrated to participate. How can we work out if there are such people and if so, encourage them to participate?

[1] It has been posted on Reddit at least, and I posted it on Twitter.

[sjakobi]

I wonder why @runeksvendsen is the only person so far asking for long-term backwards compatibility.

I think backwards compatibility is important. But it's more important to fix API design issues like Data.List.null :: Foldable t => t a -> Bool.