Skip to content

[ReTrack Issue] Unable to reproduce the evaluate results using the demo script #10

@manasabha

Description

@manasabha

Hello! There seem to be two paths currently to hit the system. Use processed grail qa file with evaluate.py under the parser directory. There is a demo pipeline that can be setup using the demo section in the read me.

Current I'm getting different results for the same questions. I have set all the flags mentioned in

For the best possible results, please enable the complete checker (use_beam_check, use_virtual_forward, use_type_checking, and use_entity_anchor. in the demo overrides.

The Redis cache seems up and running. For a sample of 100 questions, nearly 2-5% loss is there in F1 score, EM .

Can you please help with this issue?
Please let me know if you need any further information

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions