name '<operator>' is not defined and TypeError: 'Pool' object is not iterable during dataset filtering #18
Description
Hi there,
I'm trying to follow the dataset generation instructions to generate a train and validation set but I'm running into many name '<operator>' is not defined messages and a TypeError: 'Pool' object is not iterable at the end of the filtering.
Could you please tell me how to generate a validation set of size N_val and a filtered train set of size (N_train - N_val) and possibly provide an end-to-end one-click script to do so?
Logs:
create_datasets.sh
# Validation set
python3 scripts/data_creation/dataset_creation.py --number_of_equations 200 --no-debug
# Train set
python3 scripts/data_creation/dataset_creation.py --number_of_equations 10000 --no-debug
# Remove validation expressions from train set
python3 scripts/data_creation/filter_from_already_existing.py --data_path data/raw_datasets/200 --csv_path "None"Output
$ ./create_datasets.sh
There are 6 equations per block. The progress bar will have this resolution
17%|████████████████████████████████▏ | 34/200 [00:00<00:00, 231.14it/s]
Expression generation took 0.262542724609375 seconds
There are 312 equations per block. The progress bar will have this resolution
0%|▌ | 33/10000 [00:03<17:02, 9.74it/s]
Expression generation took 3.5042946338653564 seconds
Loading metadata
Creating image for validation set
Starting finding out index of equations present in the validation set or wih numerical problems
0%| | 0/200 [00:00<?, ?it/s]name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'asin' is not defined
name 'asin' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'Abs' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'asin' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
name 'ln' is not defined
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 494.10it/s]
Traceback (most recent call last):
File "scripts/data_creation/filter_from_already_existing.py", line 130, in <module>
main()
File "/home/psaegert/miniconda3/envs/nsrts/lib/python3.7/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/psaegert/miniconda3/envs/nsrts/lib/python3.7/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/psaegert/miniconda3/envs/nsrts/lib/python3.7/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/psaegert/miniconda3/envs/nsrts/lib/python3.7/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "scripts/data_creation/filter_from_already_existing.py", line 126, in main
print(f"Total number of good equations {len([x for x in p if x[1]])}")
TypeError: 'Pool' object is not iterable
Environment
$ pip freeze
aiohttp==3.8.6
aiosignal==1.3.1
antlr4-python3-runtime==4.8
async-timeout==4.0.3
asynctest==0.13.0
attrs==24.2.0
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1702571698061/work
certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1720457958366/work/certifi
charset-normalizer==3.3.2
click==8.1.7
cycler==0.11.0
dataclass-dict-convert==1.7.3
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1649586340600/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
docker-pycreds==0.4.0
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2023.1.0
gitdb==4.0.11
GitPython==3.1.43
h5py==3.8.0
hydra-core==1.0.0
idna==3.7
importlib-metadata==6.7.0
importlib-resources==5.12.0
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1666723258080/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1651240553635/work
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
jsons==1.6.3
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1673615989977/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1658332345782/work
kiwisolver==1.4.5
lightning-utilities==0.10.1
matplotlib==3.5.3
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1713250518406/work
mpmath==1.3.0
multidict==6.0.5
nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work
-e git+https://github.com/SymposiumOrganization/NeuralSymbolicRegressionThatScales.git@92d7c46c0417aeb76ecebcac982b8ccf1a3f8860#egg=nesymres&subdirectory=src
numexpr==2.8.6
numpy==1.21.6
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
omegaconf==2.1.2
ordered-set==4.1.0
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1696202382185/work
pandas==1.3.5
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1712320355065/work
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow==9.5.0
platformdirs==4.0.0
prompt_toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1718047967974/work
protobuf==4.24.4
psutil==5.9.8
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1700607939962/work
pyparsing==3.1.2
python-dateutil==2.9.0.post0
pytorch-lightning==1.9.5
pytz==2024.1
PyYAML==6.0.1
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1652965483789/work
requests==2.31.0
scipy==1.7.3
sentry-sdk==2.1.1
setproctitle==1.3.3
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
smmap==5.0.1
stringcase==1.2.0
sympy==1.10.1
torch==1.13.1
torchmetrics==0.11.4
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1656937818679/work
tqdm==4.66.4
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work
typing_extensions==4.7.1
typish==1.9.3
urllib3==2.0.7
wandb==0.17.0
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1699959196938/work
yarl==1.9.4
zipp==3.15.0