Open
Description
Description:
This repo currently doesn’t have any unit tests. Let’s add JUnit 5 support and create an initial test for the Tokenizer class to ensure tokenization behaves as expected.
Tasks:
- Add JUnit 5 as a dependency using Maven.
- Create a test class:
src/test/java/llama/TokenizerTest.java - Add a test that checks tokenization for a simple prompt. Example:
@Test
void testBasicTokenization() {
Tokenizer tokenizer = new Tokenizer("path/to/tokenizer.model");
int[] tokens = tokenizer.tokenize("Hello world");
assertNotNull(tokens);
assertTrue(tokens.length > 0);
}