Address methodologies for improving evaluation scoring

[donaldknoller]

The current dataset being used for evaluation has the following properties:

1. Fetch 2048 samples from historical synthetic dataset
2. Fetch 2048 samples from latest 48 hours of synthetic data
3. Evaluate on the 4096 samples gathered.

While this setup has been effective at preventing blatant overfitting, there are a few limitations. Mainly, the changing nature of the dataset lends itself to a higher degree of variance when it comes to scoring.

This issue aims to discuss potential solutions and the details of the implementation.
Some potential suggestions from miners:

1. Utilize an epoch based dataset that is fixed in nature. A fixed dataset will reduce variance but requires some safeguards against overfitting
2. Re evaluate existing models given some cycle. Implementation for this will also depend on interactions with emissions and re-calculating models in the case of "fluke" scores

More recently, the sample size for evaluation has been adjusted as seen here to aid in reducing variance, but there may be other implementations that may be effective as well

[donaldknoller]

Additional suggestion from @torquedrop :

I recommend using the truncated mean scoring system. In this system, the highest and lowest scores are removed, and the average of the remaining scores is used to calculate the final result. To save time during score evaluation, consider reducing the sample size to 1024 or 512 for each scoring instance, and limiting the number of score calculations to 5.

[donaldknoller]

Implementations to add:

1. When querying the dataset API, use query parameters to specify the time range
2. Add in some notes which query range was used when evaluating

[donaldknoller]

A note regarding the current eval_score and its relation to creativity score:

The current implementation does set a soft limit of scoring based on the creativity score threshold. This is intentional as it is an initial guard to punish overfitting. A promising method of better evaluating the quality of models at this point in time suggests that synthetic benchmarks, implemented via the post_eval_score multiplier, may yield better end results.

Also, in the future, the behaviour we have seen with leaderboard resets suggests that we can implement a soft "epoch" based scoring which will require miners to continuously re submit their models according to some optimal cadence.

Currently, the priority is to implement a better post_eval_score method which will dynamically adjust miner scores to better select "good" models