Plot metric values from multiple experiments as a single series

[maxromanovsky]

I wonder if it's possible to achieve the following:

• Run hyperparameter search: multiple experiments with a single step
• Store resulting metric for each experiment (e.g., f1_score)
• Store input hyperparameter values (e.g., threshold, length)
• Create 2 plots using DVC where values of f1_score are on y axis, plotted as a single series, and values of hyperparameter (threshold for first plot, length for second) are on x axis.

So far I haven't found how to achieve that.
Closest thing I found is to store hyperparameter as metric too, and plot a chart that will have multiple series (each with one data point from corresponding experiment).

Is it even possible using DVC? If not, is it planned?
And if that's not possible, do you see any solution other than export dvc exp show --csv and plot it in a python script?

[daavoo]

Is it even possible using DVC? If not, is it planned?

It is not possible to do in DVC CLI and it is not currently planned to support it natively in the CLI.

And if that's not possible, do you see any solution other than export dvc exp show --csv and plot it in a python script?

We currently support something similar to what you want in the DVC VSCode UI using Custom Plots.
It will be eventually supported also in the DVC Studio Web UI .

Quick example using a simple DVCLive snippet and the VSCode UI :

import random
from dvclive import Live

for threshold in [0.1, 0.2, 0.3]:
    for lenght in [10, 20, 30]:
        with Live(report=None, save_dvc_exp=True) as live:
            score = random.random() * lenght * threshold
            live.log_param("threshold", threshold)
            live.log_param("lenght", lenght)
            live.log_metric("score", score, plot=False)

custom-plots.mp4

For more complex things, I would recommend using https://dvc.org/doc/api-reference/exp_show and Python to plot, as you suggested