Reproduce reported results

[trannhuthuat96]

Hi @hugochan,

Could you please public the actual 20 news data that you used to train model and get the results reported in paper?

I downloaded data from the link http://qwone.com/~jason/20Newsgroups/ (filename is 20news-bydate.tar.gz) and used your code but cannot reproduce your reported results. (I also used your processed data but didn't work as well)

Thanks,

[hugochan]

Hi @hugochan,

Could you please public the actual 20 news data that you used to train model and get the results reported in paper?

I downloaded data from the link http://qwone.com/~jason/20Newsgroups/ (filename is 20news-bydate.tar.gz) and used your code but cannot reproduce your reported results. (I also used your processed data but didn't work as well)

Thanks,

Hi @trannhuthuat96 , that is exactly the data we used in our experiments. You should be able to reproduce the results following the instructions in README. Can you report your results here? Also I am not sure if the issue is related to different TF versions, so please check #24 as well.

[trannhuthuat96]

Sorry @hugochan for late response.

I checked issued #23 and #24 as well and changed TF version to 1.2.1 but still didn't reproduce your results.

For document classification on 20 news data, I got 62% accuracy with 128 topics with KATE. Moreover, using your processed data mentioned in #23, the results is 73% instead of (74.4%) as reported.

Do you have any ideas/comments to reproduce result?

[hugochan]

Sorry @hugochan for late response.

I checked issued #23 and #24 as well and changed TF version to 1.2.1 but still didn't reproduce your results.

For document classification on 20 news data, I got 62% accuracy with 128 topics with KATE. Moreover, using your processed data mentioned in #23, the results is 73% instead of (74.4%) as reported.

Do you have any ideas/comments to reproduce result?

@trannhuthuat96 I am not sure what will be other reasons besides package (e.g., TF, Keras) versions. BTW, did you use cpu or gpu version of TF? I used cpu version when doing the experiments. Not sure if this is the cause.

You might also try running the experiments multiple times with different random seeds, and report the avg/std scores.

[trannhuthuat96]

Thanks @hugochan for your suggestions.
I used both GPU and CPU for training but cannot reproduce your reported results.

May be it is needed further exploration to reproduce the results.