Hit "failed to rsync up" to test test_self_hosted_huggingface_instructor_embedding_documents()

[HenryTangMain]

Describe the bug
Test with langchain function self_hosted_huggingface_instructor_embedding_documents(), it transfers small files from client to server, the client hits the following error during the process:

INFO | 2023-08-01 21:57:49,547 | Setting up Function on cluster.
INFO | 2023-08-01 21:57:49,547 | Copying folder from file:///root/t to: rh-cls
sky.exceptions.CommandError: Command rsync -Pavz --filter='dir-merge,- .gitignore' -e "ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ConnectTimeout=30s -o ForwardAgent=yes -o ControlMaster=auto -o ControlPath=/tmp/skypilot_ssh_root/3651d5b8ee/%C -o ControlPersist=300s" '/root/t/' root@172.17.0.2:'~/t/' failed with return code 2.
Failed to rsync up: /root/t/ -> ~/t/. Ensure that the network is stable, then retry.

Then, single the command out and launch:

#rsync -Pavz --filter='dir-merge,- .gitignore' -e "ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ConnectTimeout=30s -o ForwardAgent=yes -o ControlMaster=auto -o ControlPath=/tmp/skypilot_ssh_root/3651d5b8ee/%C -o ControlPersist=300s" '/root/t/' root@172.17.0.2:'~/t/'
protocol version mismatch -- is your shell clean?
(see the rsync manpage for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(622) [sender=3.2.7]

If relevant, include the steps or code snippet to reproduce the error.

Versions
Please run the following and paste the output below.

wget https://raw.githubusercontent.com/run-house/runhouse/main/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

python collect_env.py [50/1206]

Python Platform: Linux-5.15.0-60-lowlatency-x86_64-with-glibc2.35
Python Version: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]

Relevant packages:
boto3==1.28.17
fastapi==0.99.0
fsspec==2023.5.0
pyarrow==12.0.1
pycryptodome==3.12.0
rich==13.5.2
runhouse==0.0.9
skypilot==0.3.3
sshfs==2023.7.0
sshtunnel==0.4.0
typer==0.9.0
uvicorn==0.23.2
wheel==0.38.4

Additional context

1. ray start --head
2. runhouse login
3. python -m runhouse.servers.http.http_server

[HenryTangMain]

After did some research online, I have a work around for this issue according to this url: https://serverfault.com/questions/304125/rsync-seems-incompatible-with-bashrc-causes-is-your-shell-clean
mv .bashrc to otherfile after 1.2.3 steps are done.

but please fix this issue. thanks.

[dongreenberg]

Wow, that's a great tip, thanks, Henry! We'd like to reduce our use of rsync to avoid these issues. By the way, we're putting together some manual setup instructions in case any of the typical channels of the cluster are blocked, if you're interested. They'll make their way into the docs shortly but drafted here: https://docs.google.com/document/d/1ZHHxaGUcPhBP9GzdxQFdcnOOOvkj47mtZBQS9GTd1jo/edit?usp=sharing

[HenryTangMain]

Thanks @dongreenberg , this is very helpful, I would expect that we can put the doc contents into the setup. runhouse definitely needs more detail setup instructions.

[dongreenberg]

Currently in progress by @carolineechen !