[V1] Logits processors extensibility

[afeldman-nm]

Purpose

Enable V1 logits processors support to be extended with custom logits processors.

New Python interface for custom logitsprocs

    # Specify logitproc by entrypoint
    llm = LLM(
        model="facebook/opt-125m",
        logits_processors_entrypoints=["dummy_logitproc"],
    )

    # Specify logitproc by fully-qualified name
    llm = LLM(
        model="facebook/opt-125m",
        logits_processors_fqns=["vllm.v1.sample.logits_processor.impls:DummyLogitsProcessor"],
    )

New CLI interface for custom logitsprocs

# Engine CLI args for specifying logitsprocs by entrypoint
--logits-processors-entrypoints=dummy_logitproc,<other logitproc>,...

# Engine CLI args for specifying logitsprocs by fully-qualified name
--logits-processors-fqns=vllm.v1.sample.logits_processor.impls:DummyLogitsProcessor,<other logitproc>,...

Test Plan

(WIP)

• Configuring a contrived custom logits proc
• E2E tests using REST API and Python interfaces
• Wrap a V0-style logits proc to create a V1-style logits proc

Test Result

WIP

(Optional) Documentation Update

WIP

Fixes #17799
Fixes #12678

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @afeldman-nm.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @afeldman-nm.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[llsj14]

This PR looks great as it refactors the code related to LogitsProcessors, splitting it into separate files. I'm currently working on a new logits processor to limit thinking tokens (PR: #20859), and I needed the changes in this PR, so I hope it gets merged soon.