Spark random forest issues

[szilard]

This is to collaborate on some issues with Spark RF also addressed by @jkbradley in comments to this post http://datascience.la/benchmarking-random-forest-implementations/ (see comments by Joseph Bradley). cc: @mengxr

Please see “Absolute Minimal Benchmark” for random forests https://github.com/szilard/benchm-ml/tree/master/z-other-tools and let's use the 1M row training set and the test set linked in from there.

@jkbradley says: One-hot encoder: Spark 1.4 includes this, plus a lot more feature transformers. Preprocessing should become ever-easier, especially using DataFrames (Spark 1.3+).

Yes, indeed. Can you please provide code that reads in the original dataset (pre- 1-hot encoding) and does the 1-hot encoding in Spark. Also, if random forest 1.4 API can use data frames, I guess we should use that for the training. Can you please provide code for that too.

@jkbradley says: AUC/accuracy: The AUC issue appears to be caused by MLlib tree ensembles aggregating votes, rather than class probabilities, as you suggested. I re-ran your test using class probabilities (which can be aggregated by hand), and then got the same AUC as other libraries. We’re planning on including this fix in Spark 1.5 (and thanks for providing some evidence of its importance!).

Fantastic. Can you please share code that does that already? I would be happy to check it out.

[jkbradley]

Apologies for the slow reply! I've been bogged down with QA for the next Spark release. I wrote the code for one-hot encoding + running RFs and computing AUC. Let me know if you run into issues using it:

[https://gist.github.com/jkbradley/1e3cc0b3116f2f615b3f]

[szilard]

Fantastic, thanks Joseph! I'll take a look asap and re-run the benchmarks with the new code. I'll let you know if I have more questions along the way...

[szilard]

@jkbradley
Hi Joseph:

1. The 1-hot encoding code you provided works great, thanks.
2. Also, the new scoring by aggregating probabilities works way better than the original aggregation of votes. For n=1M and

val numTrees = 100
val featureSubsetStrategy = "sqrt"   
val impurity = "entropy"    //val impurity = "gini"
val maxDepth = 20           
val maxBins = 100           //val maxBins = 50

I can verify I get AUC 71.4 instead of 62.5
3. If I run it on 1.5.0 (Tungsten) runtime and memory footprint is still the same.
4. However, something weird is still going on with accuracy, the AUC does not increase (enough) with increasing size (similar to the results with the other tools):
For n=0.1M: AUC 71.1
For n=1M: AUC 71.4
See also this graph for a different (the original) setup (500 trees etc.):
https://raw.githubusercontent.com/szilard/benchm-ml/master/2-rf/x-plot-auc.png

Any comments on 3 and especially on 4 above?
Thanks.

[jkbradley]

I'm glad 1 & 2 worked out!

For 3, I should have been more specific. Tungsten makes improvements on DataFrames, so it should improve the performance of simple ML Pipeline operations like feature transformation and prediction. However, to get the same benefits for model training, we'll need to rewrite the algorithms to use DataFrames and not RDDs. Future work...

For 4: Are you able to examine the tree structure? I'm wondering if the limit on MLlib tree depth is the difference. What happens when you set a max depth for the other libraries? I'll try to look into it more too.

[szilard]

Thanks again for 1 & 2.

For 3: Yes, that was my guess too. One more question: I re-run the logistic regression https://github.com/szilard/benchm-ml/blob/master/1-linear/5-spark.txt with 1.5.0 as well and got same training time as with 1.4.0. While my guess was that RF will be the same, I thought LR uses the DF as underlying and it will be faster with Tungsten. No?

[szilard]

For 4: I'm not sure if it's max_depth, for example H2O does not have similar behavior (with same max_depth = 20). To make further testing simple I zipped the 0.1M and 1M train and test datasets after being 1-hot encoded and exported to parquet format with your code and you can get it all in 1 from here: https://s3.amazonaws.com/benchm-ml--spark2/spark-0.1m%2B1m-1hot-parquet.tgz

As for the training/AUC part I made some little changes and my code is here: https://github.com/szilard/benchm-ml/blob/master/z-other-tools/5xb-spark-trainpred.txt
The changes are mainly:

val trainDataPath = "spark1hot-train-1m.parquet"
val testDataPath = "spark1hot-test-1m.parquet"

that is file names (in current dir) and

val impurity = "entropy"    
val maxDepth = 20        
val maxBins = 100

to match the "absolute minimal benchmark".

So with these I get:

n=0.1M: AUC 71.1
n=1M: AUC 71.4

as I mentioned above.

I think now it would be easier for you to take a look. There should be a higher increase in AUC from n=0.1M to 1M.

[jkbradley]

For 3: Yes, that was my guess too. One more question: I re-run the logistic regression https://github.com/szilard/benchm-ml/blob/master/1-linear/5-spark.txt with 1.5.0 as well and got same training time as with 1.4.0. While my guess was that RF will be the same, I thought LR uses the DF as underlying and it will be faster with Tungsten. No?

No, unfortunately. It's using a different optimizer (OWLQN) for some models, but it's still using the RDD API.

I'll take a look at the AUC issue.

[szilard]

Thanks for answer to 3 and thanks for looking into 4 :)

[jkbradley]

I didn't find anything obvious yet for issue 4, but there are still some items I want to investigate. (I haven't yet scanned the sklearn implementation carefully for comparison.) In the meantime, would you be able to update the blog post to include the soft predictions? It'd be nice now that it's easily available in Spark 1.5. Thanks!

[szilard]

Thanks @jkbradley for working on this. I already updated the github README a few days ago and I'll update the http://datascience.la post as well soon.

[ehiggs]

Hi there. It's been a few months and I'm curious if any progress has been made here? Is there a JIRA on the apache page tracking the work in Spark for this?

Thanks

[szilard]

@ehiggs thanks for asking, though I defer this to @jkbradley and Team

[ehiggs]

I made an issue on the Apache Jira here.

[szilard]

Great, thanks.

The memory and speed are one issue, but there is also the issue of accuracy, see point numbered as 4 earlier in this thread.

In short the AUC does not increase from 100K records to 1M as it does for all other tools:
https://raw.githubusercontent.com/szilard/benchm-ml/master/2-rf/x-plot-auc.png

It can be an implementation bug or an "architecture bug" (by using some approximation that does not work well). I'm curious if anyone is using RF in production for large datasets and if they are really validating the results by comparing the accuracy to other tools.