Cleanup: throw error on missing `tokenizer_config.json`?

[pcuenca]

Context: #246 (comment)

Some repos still don't have a tokenizer_config.json file (example), and it's not going to be added for BC considerations.

Question: should we just throw when this happens?

More info:

• Loading the tokenizer with transformers works. Saving it generates all necessary files:

>>> from transformers import AutoTokenizer
>>> t = AutoTokenizer.from_pretrained("t5-base")
>>> t.save_pretrained("t5_saved")

• transformers.js raises an exception. But since it relies on onnx repos that have been previously exported, it's a non-issue – exporting goes through the previous path and the file is available, see Xenova/t5-base

Thoughts on doing the same (throw an error) for the sake of cleanup?

cc @mattt @FL33TW00D

[mattt]

@pcuenca Repeating back my own understanding to clarify:

AutoTokenizer.from(pretrained:) currently throws if configuration isn't found.

swift-transformers/Sources/Tokenizers/Tokenizer.swift

Lines 629 to 639 in a814cc5

	static func from(
	pretrained model: String,
	hubApi: HubApi = .shared,
	strict: Bool = true
	) async throws -> Tokenizer {
	let config = LanguageModelConfigurationFromHub(modelName: model, hubApi: hubApi)
	guard let tokenizerConfig = try await config.tokenizerConfig else { throw TokenizerError.missingConfig }
	let tokenizerData = try await config.tokenizerData
	return try AutoTokenizer.from(tokenizerConfig: tokenizerConfig, tokenizerData: tokenizerData, strict: strict)
	}

...but the fallback logic in LanguageModelConfigurationFromHub prevents that from happening for T5, specifically:

swift-transformers/Sources/Hub/Hub.swift

Lines 105 to 128 in a814cc5

	public var tokenizerConfig: Config? {
	get async throws {
	if let hubConfig = try await configPromise.value.tokenizerConfig {
	// Try to guess the class if it's not present and the modelType is
	if hubConfig.tokenizerClass?.string() != nil { return hubConfig }
	guard let modelType = try await modelType else { return hubConfig }
	// If the config exists but doesn't contain a tokenizerClass, use a fallback config if we have it
	if let fallbackConfig = Self.fallbackTokenizerConfig(for: modelType) {
	let configuration = fallbackConfig.dictionary()?.merging(hubConfig.dictionary(or: [:]), strategy: { current, _ in current }) ?? [:]
	return Config(configuration)
	}
	// Guess by capitalizing
	var configuration = hubConfig.dictionary(or: [:])
	configuration["tokenizer_class"] = .init("\(modelType.capitalized)Tokenizer")
	return Config(configuration)
	}
	// Fallback tokenizer config, if available
	guard let modelType = try await modelType else { return nil }
	return Self.fallbackTokenizerConfig(for: modelType)
	}
	}

swift-transformers/Sources/Hub/Hub.swift

Lines 242 to 262 in a814cc5

	static func fallbackTokenizerConfig(for modelType: String) -> Config? {
	// Fallback tokenizer configuration files are located in the `Sources/Hub/Resources` directory
	guard let url = Bundle.module.url(forResource: "\(modelType)_tokenizer_config", withExtension: "json") else {
	return nil
	}
	do {
	let data = try Data(contentsOf: url)
	let parsed = try JSONSerialization.jsonObject(with: data, options: [])
	guard let dictionary = parsed as? [NSString: Any] else {
	throw Hub.HubClientError.parseError("Failed to parse fallback tokenizer config")
	}
	return Config(dictionary)
	} catch let error as Hub.HubClientError {
	print("Error loading fallback tokenizer config: \(error.localizedDescription)")
	return nil
	} catch {
	print("Error loading fallback tokenizer config: \(error.localizedDescription)")
	return nil
	}
	}

Therefore, you're proposing to remove that fallback and the bundled resources so that AutoTokenizer.from(pretrained:) starts throwing an error for T5, correct?

I don't have enough context to understand the impact of this, but I'm generally in favor of matching the behavior of sibling projects and reducing the size of our resources.

[pcuenca]

Yes, that's correct.

Sibling projects work differently - transformers works (it has code to deal with these cases), transformers.js doesn't. In general, it's a good goal to be able to instantiate a tokenizer from any transformers model on the Hub, but these are old formats easy to convert.

[mattt]

Got it. I think that if 1) API consumers have a reasonable, documented way to recover from TokenizerError.missingConfig and 2) only a small number of users would be impacted by this change, then removing the fallback makes sense.

[pcuenca]

As mentioned in #261, we would also need to handle this to skip downloading model config files, which are only used if this entry is needed.