Genetic Tuning Improvements #160

thetheodor · 2018-03-15T13:34:48Z

Use better candidate sampling and make the selection phase more explicit and tunable by command line flags.

prigoyal · 2018-03-15T13:41:12Z

@ttheodor can you provide some information on this? I am curious to know how sampling is changing and the effect on performance. thanks!

prigoyal

nits for code reusability

prigoyal · 2018-03-15T13:42:13Z

src/autotuner/genetic_search.cc

@@ -72,9 +74,35 @@ void mutate(
  }
 }

-void normalizeVector(std::vector<double>& v) {
+double mean(std::vector<double>& v) {


can these be moved to the utils file https://github.com/facebookresearch/TensorComprehensions/blob/master/src/autotuner/utils/utils.cc

where we already have mathematical functions implemented?

prigoyal · 2018-03-15T13:42:23Z

src/autotuner/genetic_search.cc

+  return sum / v.size();
+}
+
+double stdv(std::vector<double>& v, double mean) {


can these be moved to the utils file https://github.com/facebookresearch/TensorComprehensions/blob/master/src/autotuner/utils/utils.cc

where we already have mathematical functions implemented?

prigoyal · 2018-03-15T13:42:29Z

src/autotuner/genetic_search.cc

+  return std::sqrt(squareSum / v.size());
+}
+
+void sigmaScale(std::vector<double>& v) {


can these be moved to the utils file https://github.com/facebookresearch/TensorComprehensions/blob/master/src/autotuner/utils/utils.cc

where we already have mathematical functions implemented?

thetheodor · 2018-03-15T13:44:54Z

@prigoyal the first commit message includes a motivation. I'll add descriptions to the others.

prigoyal · 2018-03-15T13:45:30Z

include/tc/autotuner/genetic_search.h

@@ -113,6 +115,7 @@ class GeneticSearch {
  Population population;
  TuningConfiguration lastBestConf;
  const size_t kMaxPopulationSize;
+  const size_t kMatingPoolSize;


is this being exposed as an option to set? like we specify pop_size etc. via flags

Right now it is not. I'll expose it in a subsequent commit.

prigoyal

nits

prigoyal · 2018-03-15T13:47:01Z

src/autotuner/genetic_search.cc

@@ -164,6 +193,7 @@ GeneticSearch::GeneticSearch(
    : population(),
      lastBestConf(confs[0]),
      kMaxPopulationSize(n),
+      kMatingPoolSize(n * 3),


just curious: what is the reason for number 3? can we move the number to header file as a static constexpr int? :)

Right now it is semi-arbitrary: it is the maximum number of candidates that can be used during breeding.

I'll expose it as an option in a subsequent commit.

prigoyal · 2018-03-15T13:48:28Z

thanks @ttheodor , can you update PR summary with slight context :)

ftynse · 2018-03-15T13:53:48Z

@ttheodor I have a hack for you: if you first push only one commit to a branch and create a PR, github will pick up the description from that commit. Once PR is created, pushing more commits does not change the description

nicolasvasilache

Looks good.
Needs a tuning run though, I will not get to it before coming back from travel.

prigoyal · 2018-03-20T17:38:37Z

@caffe2bot retest this please

ezyang · 2018-04-05T16:12:10Z

@caffe2bot retest this please

ezyang · 2018-04-05T16:16:49Z

@caffe2bot retest this please

Using an unscaled fitness value for selection is problematic: -Outstanding individuals take over very quickly, this leads to premature convergence. -When fitness values are close together, very litle selection pressure is applied and selection is almost uniformly random. Having slightly better fitness does not improve an individual's survival chances. -Transposing the fitness function (e.g. adding a constant value) changes the selection probabilities even though the location of the optimum (and the "shape" of the fitness) remain unchanged. Scaling the fitness function helps ameliorate those issues. Sigma scaling is used: fitness' = max(fitness - (mean_fitness - 2 * std_fitness), 0)

Stochastic Universal Sampling is an improvement upon the roulette algorithm that was previously used

Previously each generation had mu candidates and generated mu children which all survided. This meant that really bad candidates that were randomly generated would survive across generations. With this change, lambda (typically larger thatn mu) children are generated and the best mu survive. The previous behaviour is a special case in which lambda = mu.

Some candidates survive across across generations. Their runtimes are stored in the options cache. Previously those candidates would be benchmarked everytime they were encountered. Now the runtimes are restored from the caches instead.

Candidates that survive across generations need not be benchmarked again and thus no compilation and gpu jobs have to be created for them.

Each generation includes two evalution phases (marked with *): 1) Benchmarking of candidates * 2) Breeding 3) Benchmarking of new candidates and selection * Except for the first generation usually most of the candidates of step 1) will have been evaluated during step 3) of the previous generation. This commit removed the "Jobs(Compiled,GPU) (0,0)/0" that was printed during step 1) of almost every generation. In theory it is possible not to have enough valid candidates to form a new generation. In those cases step 1) will be run and the progress message will be printed

thetheodor · 2018-05-28T13:54:49Z

Replaced by #453

thetheodor requested a review from nicolasvasilache March 15, 2018 13:34

facebook-github-bot added the CLA Signed label Mar 15, 2018

prigoyal reviewed Mar 15, 2018

View reviewed changes

nicolasvasilache reviewed Mar 16, 2018

View reviewed changes

thetheodor changed the title ~~[genetic search] Better candidate sampling~~ Genetic Tuning Improvements Mar 16, 2018

thetheodor force-pushed the genetic_improv branch 2 times, most recently from ba40df5 to d90a51c Compare March 20, 2018 12:26

thetheodor changed the base branch from dev to master March 20, 2018 12:26

thetheodor force-pushed the genetic_improv branch from 795be0e to e4cecc3 Compare March 24, 2018 12:22

thetheodor mentioned this pull request Mar 27, 2018

Increase the default number of the autotuner generations in python bindings. #195

Closed

2 tasks

thetheodor force-pushed the genetic_improv branch 2 times, most recently from de8f30f to 58fb6c4 Compare April 4, 2018 14:17

thetheodor force-pushed the genetic_improv branch 2 times, most recently from a109131 to 99ec2b3 Compare April 5, 2018 16:03

thetheodor force-pushed the genetic_improv branch 4 times, most recently from 8f7bc63 to eb3c330 Compare April 12, 2018 13:19

Theodoros Theodoridis added 3 commits April 13, 2018 17:23

[genetic search] Fix bug that disabled crossover

4c3f915

[genetic search] Switch to Stochastic Universal Sampling

5052f83

Stochastic Universal Sampling is an improvement upon the roulette algorithm that was previously used

Theodoros Theodoridis added 6 commits April 13, 2018 17:23

[genetic tuning] Use cached runtimes if available

fefa818

Some candidates survive across across generations. Their runtimes are stored in the options cache. Previously those candidates would be benchmarked everytime they were encountered. Now the runtimes are restored from the caches instead.

[genetic tuning] Create jobs only for new candidates

ea3453c

Candidates that survive across generations need not be benchmarked again and thus no compilation and gpu jobs have to be created for them.

Move mathematical functions to utils.h

a05e0cc

[genetic search] Handle invalid generations properly

d832583

thetheodor force-pushed the genetic_improv branch from eb3c330 to f340b60 Compare April 13, 2018 15:23

thetheodor mentioned this pull request May 28, 2018

Genetic Tuning Improvements #453

Open

thetheodor closed this May 28, 2018

Genetic Tuning Improvements #160

Genetic Tuning Improvements #160

Uh oh!

Conversation

thetheodor commented Mar 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prigoyal commented Mar 15, 2018

Uh oh!

prigoyal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thetheodor commented Mar 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prigoyal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prigoyal commented Mar 15, 2018

Uh oh!

ftynse commented Mar 15, 2018

Uh oh!

nicolasvasilache left a comment

Choose a reason for hiding this comment

Uh oh!

prigoyal commented Mar 20, 2018

Uh oh!

ezyang commented Apr 5, 2018

Uh oh!

ezyang commented Apr 5, 2018

Uh oh!

thetheodor commented May 28, 2018

Uh oh!

Uh oh!

thetheodor commented Mar 15, 2018 •

edited

Loading