pywb v-2.8.3: ModuleNotFoundError: No module named 'pyamf' #919
Description
Describe the bug
pyamf is not listed in requirements.txt but requires during runtime. I've checked with GitHub search https://github.com/search?q=repo%3Awebrecorder%2Fpywb+pyamf&type=code
grep -r pyamf target/
target/pywb/rewrite/rewrite_amf.py: from pyamf import remoting
target/pywb/warcserver/amf.py:from pyamf.remoting import Envelope, Request
target/pywb/warcserver/amf.py:from pyamf.flex.messaging import RemotingMessage
target/pywb/warcserver/inputrequest.py:from pyamf.remoting import decode
target/pywb/warcserver/test/test_amf.py:import pyamf
target/pywb/warcserver/test/test_amf.py:from pyamf.remoting import Envelope, Request, encode, decode
target/pywb/warcserver/test/test_amf.py:from pyamf.flex.messaging import RemotingMessage
target/pywb/warcserver/test/test_amf.py:pyamf.register_class(CustomObject, "custom.object")
target/pywb/warcserver/test/test_amf.py: ev = Envelope(pyamf.AMF3)
target/pywb/warcserver/test/test_inputreq.py:from pyamf import AMF3
target/pywb/warcserver/test/test_inputreq.py:from pyamf.remoting import Request, Envelope, encode1
After inspecting setup.py:
https://github.com/webrecorder/pywb/blob/97fffe3a345b10d1fb92f3a124aacfa010de9be7/setup.py#L62C1-L69C24
def load_requirements(filename):
with open(filename, 'rt') as fh:
requirements = fh.read().rstrip().split('\n')
if sys.version_info > (3, 0):
requirements.append("py3AMF")
else:
requirements.append("pyAMF")
return requirementsSteps to reproduce the bug
git clone https://github.com/webrecorder/pywb
cd pywb
git checkout v-2.8.3
mkdir target
# Remove test requirements
rm -rf test_requirements.txt
# Merge all unique entries into one file.
sort -u *requirements* > r-all_tmp
rm -rf *requirements*
mv r-all_tmp requirements.txt
# Remove uwsgi wich comes as default and add gunicorn as an app server.
echo gunicorn >> requirements.txt
sed -i '/uwsgi/d' requirements.txt
# Install all dependencies into target
python3.11 -m pip install -r requirements.txt -t target
cp -R pywb target/
# Let Python find all modules
export PYTHONPATH=$PYTHONPATH:$PWD/target
# run app
./target/bin/gunicorn pywb.apps.wayback
Expected behavior
Application lists all requirements needed for runtime and starts up successfully when everything is installed.
Screenshots
./target/bin/gunicorn pywb.apps.wayback
[2024-11-05 10:57:07 +0000] [65409] [INFO] Starting gunicorn 23.0.0
[2024-11-05 10:57:07 +0000] [65409] [INFO] Listening at: http://127.0.0.1:8000 (65409)
[2024-11-05 10:57:07 +0000] [65409] [INFO] Using worker: sync
[2024-11-05 10:57:07 +0000] [65410] [INFO] Booting worker with pid: 65410
[2024-11-05 10:57:08 +0000] [65410] [ERROR] Exception in worker process
Traceback (most recent call last):
File "target/gunicorn/arbiter.py", line 608, in spawn_worker
worker.init_process()
File "target/gunicorn/workers/base.py", line 135, in init_process
self.load_wsgi()
File "target/gunicorn/workers/base.py", line 147, in load_wsgi
self.wsgi = self.app.wsgi()
^^^^^^^^^^^^^^^
File "target/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
^^^^^^^^^^^
File "target/gunicorn/app/wsgiapp.py", line 57, in load
return self.load_wsgiapp()
^^^^^^^^^^^^^^^^^^^
File "target/gunicorn/app/wsgiapp.py", line 47, in load_wsgiapp
return util.import_app(self.app_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "target/gunicorn/util.py", line 370, in import_app
mod = importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "target/pywb/apps/wayback.py", line 2, in <module>
from pywb.apps.frontendapp import FrontEndApp
File "target/pywb/apps/frontendapp.py", line 12, in <module>
from pywb.recorder.recorderapp import RecorderApp
File "target/pywb/recorder/recorderapp.py", line 15, in <module>
from pywb.warcserver.inputrequest import DirectWSGIInputRequest
File "target/pywb/warcserver/inputrequest.py", line 3, in <module>
from pywb.warcserver.amf import Amf
File "target/pywb/warcserver/amf.py", line 3, in <module>
from pyamf.remoting import Envelope, Request
ModuleNotFoundError: No module named 'pyamf'
[2024-11-05 10:57:08 +0000] [65410] [INFO] Worker exiting (pid: 65410)
[2024-11-05 10:57:08 +0000] [65409] [ERROR] Worker (pid:65410) exited with code 3
[2024-11-05 10:57:08 +0000] [65409] [ERROR] Shutting down: Master
[2024-11-05 10:57:08 +0000] [65409] [ERROR] Reason: Worker failed to boot.
Environment
- OS: Amazon Linux 2023.6.20241010
- Browser N/A
python3.11 -m pip freeze
babel==2.16.0
boto3==1.35.54
botocore==1.35.54
brotlipy==0.7.0
certauth==1.3.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
cryptography==43.0.3
fakeredis==0.16.0
filelock==3.16.1
gevent==22.10.2
greenlet==2.0.2
gunicorn==23.0.0
idna==3.10
Jinja2==3.1.4
jmespath==1.0.1
lxml==5.3.0
MarkupSafe==3.0.2
packaging==24.1
portalocker==2.10.1
pycparser==2.22
pyOpenSSL==24.2.1
PySocks==1.7.1
python-dateutil==2.9.0.post0
PyYAML==6.0.2
redis==2.10.6
requests==2.32.3
requests-file==2.1.0
s3transfer==0.10.3
six==1.16.0
surt==0.3.1
tldextract==5.1.3
translate-toolkit==3.14.1
ua-parser==0.18.0
ujson==5.10.0
urllib3==2.2.3
warcio==1.7.4
wcwidth==0.2.13
webassets==2.0
webencodings==0.5.1
Werkzeug==2.2.3
wsgiprox==1.5.2
youtube-dl==2021.12.17
zope.event==5.0
zope.interface==7.1.1