Outputting WandB performance metrics #903
Replies: 10 comments 20 replies
-
Hi Lorenzo, please provide some details of your code and error messages so that we can assist you. An example of using wandb with CNN.train is given here: https://opensoundscape.org/en/latest/tutorials/train_cnn.html#Set-up-WandB-model-logging Make sure you have created a wandb account and run It will also be worth checking if wandb logging works when you use the tutorial notebook from that link |
Beta Was this translation helpful? Give feedback.
-
Yeah unfortunately it occurs every time I start a new wandb_session.
Lorenzo Scala
Senior Marine Scientist
Seiche Ltd
T:
+44(0)1409 404050
E: ***@***.***
W: https://www.seiche.com
Bradworthy Industrial Estate, Langdon Road, Bradworthy, Holsworthy, Devon, EX22 7SF, United Kingdom
Seiche Ltd is registered in England & Wales. Registered number: 1468514. Registered Office: Bradworthy Industrial Estate, Langdon Road, Bradworthy, Holsworthy, Devon, EX22 7SF, United Kingdom
The information contained in this e-mail transmission, and any documents, files or previous e-mail messages attached to it, is privileged and confidential, and solely intended for the use of the individual(s) to whom it is addressed. If you are not the intended recipient or a person responsible for delivering it to the intended recipient you should not read, copy, distribute or otherwise use the information, and you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is strictly prohibited. If you have received this message in error, please notify sender immediately and delete this e-mail and attached documents.
From: sammlapp ***@***.***>
Sent: Wednesday, November 1, 2023 9:27 PM
To: kitzeslab/opensoundscape ***@***.***>
Cc: Lorenzo Scala ***@***.***>; Author ***@***.***>
Subject: Re: [kitzeslab/opensoundscape] Outputting WandB performance metrics (Discussion #903)
CAUTION: This email originated from outside of SWTG.
It seems like other wandb users have experiences a similar issue: wandb/wandb#1869<wandb/wandb#1869> (even since that issue was "closed")
Are you perhaps able to run wandb.finish() then start a new wandb session with wandb.init() ? Does the error occur again?
—
Reply to this email directly, view it on GitHub<#903 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC4F3MGRJKVCLXLX6PCLFTTYCK5D7AVCNFSM6AAAAAA6TEFXH6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TINJQGA2DS>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
I have updated to wandb version 0.16.0 but the issue persists. I am scratching my head a bit with it 😊
Lorenzo Scala
Senior Marine Scientist
Seiche Ltd
T:
+44(0)1409 404050
E: ***@***.***
W: https://www.seiche.com
Bradworthy Industrial Estate, Langdon Road, Bradworthy, Holsworthy, Devon, EX22 7SF, United Kingdom
Seiche Ltd is registered in England & Wales. Registered number: 1468514. Registered Office: Bradworthy Industrial Estate, Langdon Road, Bradworthy, Holsworthy, Devon, EX22 7SF, United Kingdom
The information contained in this e-mail transmission, and any documents, files or previous e-mail messages attached to it, is privileged and confidential, and solely intended for the use of the individual(s) to whom it is addressed. If you are not the intended recipient or a person responsible for delivering it to the intended recipient you should not read, copy, distribute or otherwise use the information, and you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is strictly prohibited. If you have received this message in error, please notify sender immediately and delete this e-mail and attached documents.
From: sammlapp ***@***.***>
Sent: Wednesday, November 8, 2023 4:04 AM
To: kitzeslab/opensoundscape ***@***.***>
Cc: Lorenzo Scala ***@***.***>; Author ***@***.***>
Subject: Re: [kitzeslab/opensoundscape] Outputting WandB performance metrics (Discussion #903)
CAUTION: This email originated from outside of SWTG.
can you make sure you have the latest version of wandb installed in your environment?
—
Reply to this email directly, view it on GitHub<#903 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC4F3MEQFFHULDZ4O3XRH73YDMAE3AVCNFSM6AAAAAA6TEFXH6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TKMBVHA2DC>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
Hi Sam,
Thanks for the tip. I’ve tried this but unfortunately no luck so far.
Lorenzo Scala
Senior Marine Scientist
Seiche Ltd
T:
+44(0)1409 404050
E: ***@***.***
W: https://www.seiche.com
Bradworthy Industrial Estate, Langdon Road, Bradworthy, Holsworthy, Devon, EX22 7SF, United Kingdom
Seiche Ltd is registered in England & Wales. Registered number: 1468514. Registered Office: Bradworthy Industrial Estate, Langdon Road, Bradworthy, Holsworthy, Devon, EX22 7SF, United Kingdom
The information contained in this e-mail transmission, and any documents, files or previous e-mail messages attached to it, is privileged and confidential, and solely intended for the use of the individual(s) to whom it is addressed. If you are not the intended recipient or a person responsible for delivering it to the intended recipient you should not read, copy, distribute or otherwise use the information, and you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is strictly prohibited. If you have received this message in error, please notify sender immediately and delete this e-mail and attached documents.
From: sammlapp ***@***.***>
Sent: Wednesday, November 15, 2023 2:45 PM
To: kitzeslab/opensoundscape ***@***.***>
Cc: Lorenzo Scala ***@***.***>; Mention ***@***.***>
Subject: Re: [kitzeslab/opensoundscape] Outputting WandB performance metrics (Discussion #903)
CAUTION: This email originated from outside of SWTG.
@canihaveabravo<https://github.com/canihaveabravo> I encountered a similar issue recently. I resolved it by deleting all folders in the temp folder that had wandb in the name and killing all wandb processes (can check for them using htop or a similar tool).
—
Reply to this email directly, view it on GitHub<#903 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC4F3MDH4J5SCVUJVFB3KCTYETIN5AVCNFSM6AAAAAA6TEFXH6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TKNZXHA3TC>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
Hi Sam. Yes I have no yet found a solution to this and have therefore taken another direction. However, happy to return to it and tackle it together. |
Beta Was this translation helpful? Give feedback.
-
If you found a solution, please let me know. I am having the same issue. |
Beta Was this translation helpful? Give feedback.
-
@canihaveabravo and @fascimare , wandb seems to have several issues that could be causing this, but its worth checking if updating your wandb package to the most recent version fixes it. I still haven't been able to reproduce the behavior myself, unfortunately, so its difficult to debug. If you do try with the newest wandb version, I would love to hear if it works or not (please also post the version you install) |
Beta Was this translation helpful? Give feedback.
-
Hi All, I have previously trained a model with opensoundscape and logged the metrics in real time with wandb. However, when I did that, I was using a Linux machine. I am currently trying to reproduce this work using a Windows machine. Originally when I tried this, I got an error messge similar to @canihaveabravo. When I updated my wandb package and made sure to activate my conda environment, I no longer got the error message. I simply typed: conda activate environment_name Unfortunately, although my code is no longer throwing an error message caused by wandb not finding the correct temp file, I am still unable to visualize the metrics using the wandb interface. The run does not crash but no metrics get logged or plotted. So, I didn't solve the problem but maybe got a little bit closer. I hope this helps. |
Beta Was this translation helpful? Give feedback.
-
it would also be helpful to know what operating system (and version) everyone is on @m1alksne @canihaveabravo @fascimare our current hypothesis is that this is a Windows-specific problem |
Beta Was this translation helpful? Give feedback.
-
Following @louisfh 's suggestion, I created a branch of opensoundscape where some special characters are removed from wandb logging calls (especially the "/" character). The branch is called To try it, you'll need to clone the opensoundscape github, checkout the branch, and pip install it. Here is an example of command line operations:
now create a new conda environment (assuming you use conda for environment management)
you might have to enter 'y' here to confirm you want to create the new conda environment You can install the package of the directory you're currently in with
Then check if you get the same errors when using this environment. Fingers crossed that this helps! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
I'm having ongoing difficulty outputting wandb logs into useful performance metrics. Can you provide any tips for this? The wandb_session argument from the CNN training tutorial doesnt seem to run with my code.
Thanks very much,
Lorenzo
Beta Was this translation helpful? Give feedback.
All reactions