Replies: 3 comments 16 replies
-
|
Hi, I'm going to have a look, this tutorial was based on (unreleased) code with Ecole that was definitely working. Perhaps something changed in the latest version that broke the code reproduction? I'll check and get back to you. |
Beta Was this translation helpful? Give feedback.
-
|
My evaluation result summary for data-size
|
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to open source your unreleased code with ecole? |
Beta Was this translation helpful? Give feedback.
-
|
I am also trying to reproduce the paper results. It would be great if you can open source your unreleased code with ecole |
Beta Was this translation helpful? Give feedback.
-
|
Yes I think this is the best option. I'll first check whether that code still works with the latest version of Ecole, and after fixing things if necessary, I'll create a public repo. |
Beta Was this translation helpful? Give feedback.
-
|
It would be great! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @DingXiangYuanZhiXing and @benjadb, I spent some time cleaning up the internal code, and re-running experiments. Because I didn't have much time, I only ran setcover with seed 0, to get a general idea. To also figure out if the problems were coming from the instance generation or Ecole/your code, I also decided to first start by running the code on the old instances that had been used for the original paper experiments, which I still had. The results were as follows: Although this was only for one seed and on one dataset, these results seem consistent with those reported in the original paper. So it seems like this code is working, that Ecole didn't break the GNN, and that the issues are either with the instance generation, or your code implementations. My next steps would be to write code for the instance generation, and re-run this code on the newly generated instances. Then we will know if the issue is your code or the Ecole instance generation. In the meanwhile, I created a new repo with the cleaned-up code. You can find it here. It is missing the instance generation code, which I still need to write, run, and debug. If eventually I manage to make it work, I'll add the missing instance generation code and I will publicize the repo on the original learn2branch repo as an official reimplementation. In the meanwhile, feel free to use the code with the instances generated by the previous repo, or on other instances of your choice. |
Beta Was this translation helpful? Give feedback.
-
This is trained on small instances, evaluated on small (different test instances), medium and big. I'm not sure what you mean by "since the nodes on big are less than small" - on the small instances the trees have 134 nodes on average, while on big they have 64,842 nodes on average?
Yes, this is a small subtlety not well explained in the original paper. To save training time, at every epoch we shuffle the 100k samples but only do a pass on the first 10k. Since the data is shuffled between every epoch, these end up being different 10k at every epoch. You can probably train on the full 100k at every epoch if you wish, with no negative consequences, I suppose?
Yes, I think this should be fine.
Well, again, at this point there is still the possibility that this is coming from the Ecole instance generators, so I wouldn't automatically assume something is up with your code. We'll see. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. I mean in small instance evaluation, gnn node num 134 is larger than scip default node num 51 but still gets time benefit, which is weird in my opinion. Hence, I assume you trained on big instances, which is wrong. |
Beta Was this translation helpful? Give feedback.
-
|
Oh I see. Yeah, the same thing was happening in the original paper as well (Table 2 on page 7): on setcover, on 5 seeds, SCIP default got 54 nodes on average while the GNN had 134. In fact, this held for all instances classes: even though the model is trained on small instances, ultimately reliability pseudocost remains better on small in nb nodes. ¯\_(ツ)_/¯ I will say, however, that in many ways the number of nodes reported by SCIP for reliability pseudocost (and non-vanilla strong branching) are arguably underestimated: there are side-effects and some nodes are "created" but automatically fathomed from the tree when branching. Whether you should consider these nodes has having been opened (hence should be counted in the tree size) is a philosophical question. On this topic you might find Gamrath and Schubert (2018) interesting, as well as the discussion in Section 4.1 of Zarpellon et al. (2020), which post-date our paper. Gamrath and Schubert propose a "fair node count" that could be used to correct that deficiency, but unfortunately it requires deactivating primal heuristics. As far as I know nobody has proposed yet a general-purpose "fair node count" that could be used in regular solving. I had opened an issue about this a while back. Gamrath, G., & Schubert, C. (2018). Measuring the impact of branching rules for mixed-integer programming. In Operations Research Proceedings 2017 (pp. 165-170). Springer |
Beta Was this translation helpful? Give feedback.
-
|
BTW, I found a weird phenomenon that the training/validation accuracy of combinatorial auction won't increase. Can you reproduce the problem? Other tasks work well. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, you're not giving me enough details. Is this from the new repo I created? Where did the instances come from? |
Beta Was this translation helpful? Give feedback.
-
|
The training accuracy doesn't increase as you can see in the picture. This situation only happens for combinatorial auction, not other 3 tasks.
----------
该邮件从移动设备发送
…--------------原始邮件--------------
发件人："Didier Chételat ***@***.***>;
发送时间：2021年10月22日(星期五) 晚上9:34
收件人："ds4dm/ecole" ***@***.***>;
抄送："dxyzx ***@***.***>;"Mention ***@***.***>;
主题：Re: [ds4dm/ecole] Cannot reproduce the experiment result in the paper (Discussion #262)
-----------------------------------
Sorry, you're not giving me enough details. Is this from the new repo I created? Where did the instances come from?
What do you mean by "the other tasks work well"?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, I understand. But I cannot help you if you do not answer my questions.
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, I reproduced this issue on my side as well - the combinatorial auctions instances are different (at minimum, harder) than what were used in the original paper. So this seems to be an issue about instance generators. I'll look into it. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
My SCIP version:
7.0.3.I use the notebook
branching-imitation.ipynbto reproduce the results in paper and try to changeMAX_SAMPLESfrom1000to120000and350000.Except for
Capacitated Facility Location, which works well on all 3 categories, we can only train model oneasyinstances( and generalize them to othereasyinstances) but cannot generalize tomediumandhard.There are some steps I have tried and problems I have encountered:
LEARNING_RATEfrom0.0001to0.01,0.001and0.00001andemb_sizefrom64to128, but none of them increase the training/valid accuracy. But the model has little positive effect when generalized toeasyinstances.easyinstances but not onmediumandhardinstances. Foreasyinstances, they all have little positive effect but onmediumandhardinstances, they all have negative effect.Combinatorial Auction:
Independent set:
Set cover:
Any advice on reproducing the result?
Beta Was this translation helpful? Give feedback.
All reactions