Skip to content

[P1] Questions about evaluate.py and Concept500-HO dataset #137

@clarenceluo78

Description

@clarenceluo78

Thank you for the interesting project and code! I have a couple of questions:

  1. When running demo.sh, the script axbench/scripts/evaluate.py seems incomplete:

    • eval_latent does not return any results.
    • A small bug occurs with KeyError: 'LsReFT_perplexity' at line 556 in eval_steering:
      for concept_id, evaluator_str, model_str, result, lm_report, lm_cache, current_df in executor.map(...)
  2. Will the Concept500-HO dataset from the HyperSteer paper be released?
    Currently, I only see the Concept500 dataset in the repository.

Thanks again for sharing this work!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions