add tokgo

[teb0ho]

We want to ensure high quality of the packages. Make sure that you've checked the boxes below before sending a pull request.

• I have read the Contribution Guidelines
• I have read the Maintainers Note
• I have read the Quality Standards

Not every repository (project) will require every option, but most projects should. Check the Contribution Guidelines for details.

• The repo documentation has a pkg.go.dev link.
• The repo documentation has a coverage service link.
• The repo documentation has a goreportcard link.
• The repo has a version-numbered release and a go.mod file.
• The repo has a continuous integration process that automatically runs tests that must pass before new pull requests are merged.
• Continuous integration is used to attempt to catch issues prior to releasing this package to end-users.

Please provide some links to your package to ease the review

• forge link (github.com, gitlab.com, etc): https://github.com/currybab/tokgo
• pkg.go.dev: https://pkg.go.dev/github.com/currybab/tokgo
• goreportcard.com: https://goreportcard.com/report/github.com/currybab/tokgo
• coverage service link (codecov, coveralls, etc.):

Pull Request content

• The package has been added to the list in alphabetical order.
• The package has an appropriate description with correct grammar.
• As far as I know, the package has not been listed here before.

Category quality

Note that new categories can be added only when there are 3 packages or more.

Packages added a long time ago might not meet the current guidelines anymore. It would be very helpful if you could check 3-5 packages above and below your submission to ensure that they also still meet the Quality Standards.

Please delete one of the following lines:

• The packages around my addition still meet the Quality Standards.

Thanks for your PR, you're awesome! 😎

Summary by CodeRabbit

• Documentation
  • Added "tokgo" to the Text Processing section as a Go-based tokenizer, including a direct link and brief description.
  • Improves discoverability of tokenization options for text-analysis workflows.
  • Purely informational update — no code, API, or behavior changes.

[coderabbitai]

Walkthrough

Added a new entry for the Go tokenizer library "tokgo" to the README.md under the Text Processing (tokenizers) section, including a link and brief description.

Changes

Cohort / File(s)	Summary of Changes
Docs README.md	Inserted a new Text Processing (tokenizers) entry: “tokgo” with a link and short description.

Sequence Diagram(s)

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Suggested reviewers

• phanirithvij
• bukarumar147

Poem

I nibble notes in README land,
A tiny tokenizer hops to stand—tokgo so grand.
Links align, a tidy row,
Tokens counted, off they go.
Carrot-cheers for docs well planned! 🥕

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title succinctly describes the main change of adding the tokgo library to the list, is directly related to the content of the PR, and is concise and clear.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6052346 and 7aca9c8.

📒 Files selected for processing (1)

• README.md (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• README.md

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: Codacy Static Code Analysis
• GitHub Check: quality

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between afa3ba4 and 6052346.

📒 Files selected for processing (1)

• README.md (1 hunks)

🧰 Additional context used

📓 Path-based instructions (3)

README.md

📄 CodeRabbit inference engine (AGENTS.md)

README.md: When modifying the Awesome list, keep categories with at least three entries
Descriptions in the Awesome list must end with a period
Avoid promotional copy; keep descriptions concise and neutral in the Awesome list
Do not remove existing list content unless removal is requested and justified

Files:

• README.md

**/*.md

📄 CodeRabbit inference engine (AGENTS.md)

Keep documentation in English

Files:

• README.md

{README.md,COVERAGE.md}

📄 CodeRabbit inference engine (AGENTS.md)

Align rendered documentation (README.md, COVERAGE.md, etc.) with behavior changes in main.go or helper packages

Files:

• README.md

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: Codacy Static Code Analysis
• GitHub Check: quality

[github-actions]

Automated Quality Checks (from CONTRIBUTING minimum standards)

• Repo: FAIL (missing semver release)
• pkg.go.dev: OK
• goreportcard: OK (grade unknown)
• coverage: missing

These checks are a best-effort automation and do not replace human review.

[avelino]

Hi @teb0ho, the automated checks for this PR failed. Please review the build logs and fix the issues. Specifically, the PR Quality Checks/quality-check failed. Please address this and re-request a review.