More evaluations, metrics and documentation (MIAs, ES, EM etc.) #90
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* testing commit * Fixes * cleanup
Fix tofu_unlearn.sh for IdKDPO method.
Revert "Dpo fix"
IdkDPO fix
* IdkDPO script fix in tofu_unlearn.sh (locuslab#65) * Fix hyperlinks in README * Download I don't know data in setup_data.py * Fix tofu_unlearn.sh for IdkDPO --------- Co-authored-by: Anmol Mekala <49127549+molereddy@users.noreply.github.com> * overwrite=True * RMU added * Fix ref model device * ruff fix * RMU updated * Update rmu.py * Update README.md: add RMU * Added references and renamed functions --------- Co-authored-by: Anmol Mekala <49127549+molereddy@users.noreply.github.com>
…on (#8) * docs: updates, small corrections, re-formats * modified ruff commands * modified ruff commands * CI/CD minor updates * added contributing + leaderboard * fix minor spelling misatkes * docs: bunch of minor updates * docs fixes --------- Co-authored-by: molereddy <m.anmolreddy@gmail.com>
* update docs * setup: add license * Update results docs * re-factor MIA probs extraction code * Ensure model and tokenizer are compatible wrt vocab size * Model-tokenizer vocab check: treat as warning * Document metric usage * Merge main properly * pre-commit enforcement to be set only in dev mode * revert vocab size warning * Revert "revert vocab size warning" This reverts commit eded904. * Revert "pre-commit enforcement to be set only in dev mode" This reverts commit 2bb5e19. * small fixes: keep pre-commit only in dev mode prereqs, remove vocab size check * privleak changes: change relative_auc handler to privleak. move min_k logic to metric file from utils * Simplified privleak version * Access key option for datasets * feat: MIA attack implementation from mimir, pipelined into open-unlearning format * Tokenwise, example wise log probs computation * configs: add MIA new version of privleak support * feat: privleak in TOFU * More hyperparam details: bsz * load and register mia metrics, cleanup old * MIA init bug fixes + convert numpy scalars to floats * fix: load ref model for ref mia * misc: comments and nits * fix: ref attack load to same device, log things * describe standard mia score signage * fix: handle division by 0 * standardize auc convention (fix muse's) make relative diff metric separate and clear * generalise the ks-test usage beyond forget quality * update docs: repo updates, clarify example commands, small fixes * fix: wrong command in readme * fit gradnorm to conventions * updated list of metrics * Add MIA attack configs * load holdout dataset from local path (must be updated) * docs: Update MIA count * remove hanging comments * fixes: docs, comments etc. addressing comments * Replace scrollable with dropdown * Ruff style changes * Clarify metrics kwargs, small docs updates * Delete todos.txt * Alert rendering * Mark alerts again * More alert rendering * more rendering changes * Add star history * Make heading smaller * Add holdout dataset to metrics, datasets and experiment configs * Add holdout evals to scripts * Fix cfg spell mistake (#9) * fix spelling of holdout * fix typo * Add links documentation * Update misc documentation * Small docs updates * Clean up links * Clarify new target models * Add ES, EM metrics + Minor fixes (#10) * Add seed in eval * Add ES, EM metrics * Add tqdm in MIA * Fix get_data * Ruff format fix (pre-commit and code) * Update docs * Update Readme * Evaluation and leaderboard modifications (#11) * Remove different splits in leaderboard * update default metrics * Fix: Only required metrics in summary * Minor fixes in docs and gitignore --------- Co-authored-by: Vineeth <48151992+Dornavineeth@users.noreply.github.com> Co-authored-by: Dornavineeth <vineethdorna@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
metrics
updated documentation
links.md(addresses [Feature Request] More documentations #70)misc
Fixes # (issue)
#70
Before submitting