Add locale-sensitive sorting + completion fixes

[gabritto]

This adds locale-sensitive sorting, similar to what we had in Strada, but using the collate package.
Tests set en-US locale, the server gets its locale from the client during initialization, the API doesn't specify a locale yet.
An annoying thing is that the collator is not concurrency-safe, since calling collator.CompareString sets state in the collator.

I'm not sure what the best way to pass down the locale from the server to the language service methods is.

There are also some small completion fixes.

[jakebailey]

I'm not sure what the best way to pass down the locale from the server to the language service methods is.

This will end up being context.Context thing (see GetRequestID, GetFormatCodeSettingsFromContext).

[jakebailey]

Probably we should still let this be unspecified, but then treat unspecified as en-US in the LS.

[gabritto]

There's already an undefined locale, is there a reason why we'd want to default to en-US in general?

[jakebailey]

IIRC Strada defaulted to en-US for localization, but I'm not sure how it worked otherwise.

[gabritto]

I think Strada left locale as undefined unless provided, which, according to MDN, means "the runtime's default locale is used when undefined" for the collator. I tested and Node just gets the system's locale.
I don't know how this compares. I'd hope using the undefined locale would result in less opinionated choices somehow, but I don't know if that's even possible.

[jakebailey]

I'll note that including the x/text stuff increases the binary size by 1.8MB on my machine (at least, the non-release binary); not great but I don't know how it's avoidable.

[Copilot]

Pull Request Overview

This PR adds locale-sensitive string comparison for completion items and includes several completion-related fixes. It integrates the collate package from golang.org/x/text to provide proper locale-aware sorting of completion entries, replacing basic string comparisons.

• Implements locale-sensitive completion sorting using the collate package
• Fixes completion bugs including comment position detection and symbol mapping
• Updates test infrastructure to use consistent locale settings (en-US)

Reviewed Changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/lsp/server.go	Adds locale parsing from client initialization parameters and passes locale through request context
internal/core/context.go	Adds context utilities for storing and retrieving locale information
internal/ls/completions.go	Implements locale-sensitive completion sorting and fixes symbol mapping bug
internal/ls/utilities.go	Fixes comment position detection by using proper preceding token
internal/ls/format.go	Corrects trailing comment range detection by using token end position
internal/fourslash/fourslash.go	Sets en-US locale in test initialization
internal/fourslash/tests/util_test.go	Updates test completion sorting to use locale-aware comparison
internal/astnav/tokens.go	Improves token position detection logic
Multiple test files	Removes t.Skip() calls to enable previously disabled completion tests

[Copilot]

Creating a new collator on every comparison call is inefficient. The collator should be created once and reused, but since collators are not thread-safe, consider using a sync.Pool or similar mechanism for better performance.

Suggested change

	func getCompareCompletionEntries(ctx context.Context) func(entryInSlice *lsproto.CompletionItem, entryToInsert *lsproto.CompletionItem) int {
	return func(entryInSlice *lsproto.CompletionItem, entryToInsert *lsproto.CompletionItem) int {
	compareStrings := collate.New(core.GetLocale(ctx)).CompareString
	var collatorPools sync.Map // Map to store sync.Pool for each locale
	func getCompareCompletionEntries(ctx context.Context) func(entryInSlice *lsproto.CompletionItem, entryToInsert *lsproto.CompletionItem) int {
	return func(entryInSlice *lsproto.CompletionItem, entryToInsert *lsproto.CompletionItem) int {
	locale := core.GetLocale(ctx)
	pool, _ := collatorPools.LoadOrStore(locale, &sync.Pool{
	New: func() interface{} {
	return collate.New(locale)
	},
	})
	collator := pool.(*sync.Pool).Get().(*collate.Collator)
	defer pool.(*sync.Pool).Put(collator)
	compareStrings := collator.CompareString

[gabritto]

Not entirely a bad suggestion, but let's see if we need to do this.