[Feature Request] More documentations

[ruidazeng]

Tasks

• Benchmark
• Unlearning method
• Evaluation
• Dataset
• None of the above

Feature request

Add detailed documentation on unlearning stuff.

Motivation

I think maybe more documentations on what existing open-sourced unlearning works have done could be helpful. So this repo can serve as a summarization of all unlearning knowledge, collected from various other researches, papers, datasets, evaluations, benchmarks, etc. Right now the docs section only contain a few information, and doesn't really tell me how each datasets/benchmarks are really evaluating the models, and how they work.

I can work on this too if required.

[molereddy]

Thank you for the suggestion! The survey of all unlearning knowledge is a larger endeavour and not our immediate priority, but we welcome your contributions. Here's a potential idea to start this off- with a documentation PR from you.

You can add benchmark documentation files summarizing the benchmark papers (MUSE/TOFU).
It can contain: a table comparing the benchmarks (across metrics, dataset details, target model details (chat/pt model), task type, list of metrics, top unlearning methods and how successful they are).

Then for each benchmark, a section detailing the important points from each paper, including detailed description of the dataset, evaluation metrics (with math formulae + checking consistency with implemented code) and other important points from the papers. We feel there are many metrics that require careful explanation (e.g. PrivLeak -MUSE and FQ -TOFU) which would be a useful contribution to the documentation so that people better understand what the metrics mean.

Also feel free to request more important features implementing components (like RMU), if we agree on the importance of inclusion, it would be great if you could raise PRs implementing the new components.

[ruidazeng]

Thank you for the suggestion! The survey of all unlearning knowledge is a larger endeavour and not our immediate priority, but we welcome your contributions. Here's a potential idea to start this off- with a documentation PR from you.

You can add benchmark documentation files summarizing the benchmark papers (MUSE/TOFU). It can contain: a table comparing the benchmarks (across metrics, dataset details, target model details (chat/pt model), task type, list of metrics, top unlearning methods and how successful they are).

Then for each benchmark, a section detailing the important points from each paper, including detailed description of the dataset, evaluation metrics (with math formulae + checking consistency with implemented code) and other important points from the papers. We feel there are many metrics that require careful explanation (e.g. PrivLeak -MUSE and FQ -TOFU) which would be a useful contribution to the documentation so that people better understand what the metrics mean.

Also feel free to request more important features implementing components (like RMU), if we agree on the importance of inclusion, it would be great if you could raise PRs implementing the new components.

That makes sense. However, I think it would be nice to have links in the documentations that at least takes us to the original paper. For example, in the README, clicking on MUSE & TOFU would take us to the arxiv or their landing page, however, there's nothing to click for any of the key unlearning methods that we support (GradAscent, GradDiff, NPO, SimNPO, DPO, RMU).

I think this might be needed because for example, what RMU means is not immediately obvious to all, as it is a relatively novel unlearning approach.

[ruidazeng]

Thank you for the suggestion! The survey of all unlearning knowledge is a larger endeavour and not our immediate priority, but we welcome your contributions. Here's a potential idea to start this off- with a documentation PR from you.

You can add benchmark documentation files summarizing the benchmark papers (MUSE/TOFU). It can contain: a table comparing the benchmarks (across metrics, dataset details, target model details (chat/pt model), task type, list of metrics, top unlearning methods and how successful they are).

Then for each benchmark, a section detailing the important points from each paper, including detailed description of the dataset, evaluation metrics (with math formulae + checking consistency with implemented code) and other important points from the papers. We feel there are many metrics that require careful explanation (e.g. PrivLeak -MUSE and FQ -TOFU) which would be a useful contribution to the documentation so that people better understand what the metrics mean.

Also feel free to request more important features implementing components (like RMU), if we agree on the importance of inclusion, it would be great if you could raise PRs implementing the new components.

I also like the benchmark summarization ideas, I can definitely work on that too.

[molereddy]

This sounds great! Will close this issue after adding links for methods and benchmarks.