TWS cannot recognize username or password after restart twice

[fuzihaofzh]

Describe the bug

I use tws and it will fail after the 2nd restart. When auto restart at the end of each day, it will have the error that cannot input the password at the second day. This can also be simulated with click the restart button in file menu. When I login with xfreerdp, The error are as shown:

When I close the dialog, it will show as follows:

It seems it fail to input the password here.

To Reproduce

docker compose pull
docker compose up

Click the Restart as shown in the figure:

After It restart successfully, click the Restart button again. It will show the error above.

I have tried bothe the latest and stable. Both of them have the same problem. Taking the stable as example:

Please provide the output of docker compose config MANDATORY ⚠️⚠️

name: algo-trader
services:
  tws:
    build:
      context: ./stable
      dockerfile: Dockerfile.tws
      tags:
      - ghcr.io/gnzsnz/tws-rdesktop:stable
    devices:
    - /dev/dri:/dev/dri
    environment:
      ALLOW_BLIND_TRADING: "no"
      AUTO_LOGOFF_TIME: ""
      AUTO_RESTART_TIME: 09:20 AM
      BYPASS_WARNING: ""
      CUSTOM_CONFIG: "NO"
      EXISTING_SESSION_DETECTED_ACTION: primary
      JAVA_HEAP_SIZE: ""
      PASSWD: ""
      PGID: "1000"
      PUID: "1000"
      READ_ONLY_API: "no"
      RELOGIN_AFTER_TWOFA_TIMEOUT: "yes"
      SAVE_TWS_SETTINGS: ""
      SSH_ALIVE_COUNT: ""
      SSH_ALIVE_INTERVAL: ""
      SSH_OPTIONS: ""
      SSH_PASSPHRASE: ""
      SSH_RDP_PORT: ""
      SSH_REMOTE_PORT: ""
      SSH_RESTART: ""
      SSH_TUNNEL: ""
      SSH_USER_TUNNEL: ""
      TIME_ZONE: America/New_York
      TRADING_MODE: paper
      TWOFA_DEVICE: ""
      TWOFA_EXIT_INTERVAL: "60"
      TWOFA_TIMEOUT_ACTION: restart
      TWS_ACCEPT_INCOMING: manual
      TWS_COLD_RESTART: ""
      TWS_PASSWORD: mypassword
      TWS_PASSWORD_PAPER: ""
      TWS_SETTINGS_PATH: /config/tws_settings
      TWS_USERID: maple123
      TWS_USERID_PAPER: ""
      TZ: America/New_York
    image: ghcr.io/gnzsnz/tws-rdesktop:stable
    networks:
      default: null
    ports:
    - mode: ingress
      host_ip: 127.0.0.1
      target: 7498
      published: "4001"
      protocol: tcp
    - mode: ingress
      host_ip: 127.0.0.1
      target: 7499
      published: "4002"
      protocol: tcp
    - mode: ingress
      host_ip: 127.0.0.1
      target: 3389
      published: "3370"
      protocol: tcp
    restart: unless-stopped
    security_opt:
    - seccomp:unconfined
    shm_size: "1073741824"
    volumes:
    - type: bind
      source: /home/maple123/workbench/explore/Quant/configs/tws/config
      target: /config
      bind:
        create_host_path: true
    - type: volume
      source: tws_tmp
      target: /tmp
      volume: {}
networks:
  default:
    name: algo-trader_default
volumes:
  tws_tmp:
    name: algo-trader_tws_tmp

Expected

Expect it can restart more than twice.

Container logs

If applicable, add the container logs docker logs <CONTAINER> or
docker compose logs to help explain your problem. MANDATORY ⚠️⚠️
See the attached file
a.log

Versions

Please complete the following information:

• OS: Linux mmaching 5.4.0-150-generic IB Gateway Docker Deployment Timeout Error on EC2 Instance #167~18.04.1-Ubuntu SMP Wed May 24 00:51:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Docker version: Docker version 24.0.2, build cb74dfc
• Image Tag: stable (I also tried latest, the same)
• Image Digest (docker images --digests): [e.g.
  sha256:60d9d54009b1b66908bbca1ebf5b8a03a39fe0cb35c2ab4023f6e41b55d17894]
  MANDATORY ⚠️⚠️

REPOSITORY                                TAG                               DIGEST                                                                    IMAGE ID       CREATED         SIZE
ghcr.io/gnzsnz/tws-rdesktop               latest                            sha256:93df11c3c71a2fffb8e1a081cc067b8a7c98bd61bb9f1be21b162865fabc9b11   6ae32e17877a   3 days ago      2.42GB
ghcr.io/gnzsnz/tws-rdesktop               <none>                            sha256:10899b53adf43bd142ae22fd47b7976ad9c17cda149df20662aa00af9ac4a0cd   963433be1c66   4 weeks ago     2.42GB
ghcr.io/gnzsnz/tws-rdesktop               stable                            sha256:225f1e4df2095ec47774a615da0a85350dd56ff93798cbed46aeeb63a6d3fe94   d7efe7b29c23   4 weeks ago     2.76GB
ghcr.io/gnzsnz/ib-gateway                 10.31.1m                          sha256:599ea5ad7b0a304aea2fc5a2c9ce565362ecd792ac400f39be73f251946b3cca   5e37e7fe496d   5 months ago    786MB
ghcr.io/gnzsnz/ib-gateway                 latest                            sha256:599ea5ad7b0a304aea2fc5a2c9ce565362ecd792ac400f39be73f251946b3cca   5e37e7fe496d   5 months ago    786MB
ghcr.io/gnzsnz/tws-rdesktop               <none>                            sha256:d824d3c06f61a567c4f1c1fb75742d0c616fac94630994ea6804b241198af1fb   340d23a604a7   9 months ago    1.59GB
ghcr.io/gnzsnz/tws-rdesktop               <none>                            sha256:0b3ef3ab32bec38848662d6d38f1663306ac0117739db1d9e0ddece7d4b7a384   969c5249330f   9 months ago    1.68GB
ghcr.io/gnzsnz/ib-gateway                 stable                            sha256:5b45bc84ad3e2ff363723592cb5df118d17ace67dcb676ffc6a3a9405da08968   1810ba66d45f   15 months ago   757MB
hello-world                               latest                            sha256:ac69084025c660510933cca701f615283cdbb3aa0963188770b54c31c8962493   d2c94e258dcb   23 months ago   13.3kB
epiai_cls                                 latest                            <none>                                                                    ae2d4d8dde1e   3 years ago     2.06GB
<none>                                    <none>                            <none>                                                                    bb86136a64a5   3 years ago     2.01GB
<none>                                    <none>                            <none>                                                                    acb8ac16171c   3 years ago     574MB
label-studio-entity-linking               latest                            <none>                                                                    e8b7fca3ef6d   4 years ago     731MB
python                                    3.6-slim                          sha256:a2bb7394b38cb394e90853a07b728b3bcd7b21521e58a07ed1158a748fe5a38c   1fbc6fae5833   4 years ago     112MB
doccano/doccano                           latest                            sha256:c75080c6e35d8765c66ac04bc010d7b24766b0be011b998ed95af44225fe474b   03b909624c37   4 years ago     619MB
heartexlabs/label-studio                  latest                            sha256:60664f2fffea8d0a7b636cedbba3e4e8c9dc25400e5e3fa8f8f781bf8e76f54b   8906a5820135   4 years ago     685MB
camltl-hdr-demo                           latest                            <none>                                                                    a40f1ef0f34d   4 years ago     2GB
ubuntu                                    focal                             sha256:703218c0465075f4425e58fac086e09e1de5c340b12976ab9eb8ad26615c3715   f63181f19b2f   4 years ago     72.9MB
nlugroup-21                               latest                            <none>                                                                    9bd115f99fac   4 years ago     8.48GB
<none>                                    <none>                            <none>                                                                    73176c8806ec   4 years ago     8.46GB
<none>                                    <none>                            <none>                                                                    4f6c2fa784b9   4 years ago     8.46GB
<none>                                    <none>                            <none>                                                                    2fcde5718b50   4 years ago     8.48GB
<none>                                    <none>                            <none>                                                                    862d231aefd0   4 years ago     8.48GB
<none>                                    <none>                            <none>                                                                    0568bc89165d   4 years ago     6.98GB
mengzaiqiao/glances                       v3.1.5-24d51a99-gpu               sha256:d9c0ca064a8de705faeb71153a1d147f8f2cb79c305232e705206a90752abb3c   3c56d868d87a   4 years ago     7.58GB
nvidia/cuda                               10.2-cudnn7-devel-ubuntu18.04     sha256:50a61a748219dc835f5a80373f9fb1eb73efa846b18b3c50945ba7c44e88d107   a4bdb1190443   4 years ago     3.82GB
python                                    3-slim-buster                     sha256:a2f5f2912364672a93f850bb219a3e687a15c365fd122b7bb35b138c22cb76dd   9fc39a5d1a9b   4 years ago     114MB
ubuntu                                    18.04                             sha256:fd25e706f3dea2a5ff705dbc3353cf37f08307798f3e360a13e9385840f73fb3   2c047404e52d   4 years ago     63.3MB
continuumio/conda-ci-linux-64-python3.7   latest                            sha256:b19c00844c25229075a06cc1c402e99d75cb22e79273c429f40c24faf9480fab   d40995f79507   4 years ago     1.19GB
nvidia/cuda                               10.1-cudnn8-runtime-ubuntu18.04   sha256:2bb0bf6204b01562975a3a031ce59c69d16db3190bc6a952cca2d4d8e8eb74e0   8975780f131b   4 years ago     2.2GB
scan/3xsutils                             latest                            <none>                                                                    0a5ee58e55f7   6 years ago     2.66GB

Additional context

It seems after 2nd restart, it fail to input the password.

I guess it may be caused that my user id is not 1000. Therefor, the config file save on the host machine is the user id 1000 which is not me. Then, it may have some trouble modify that file to store the password?

Then, I set the uid as a parameter in the cimpose file. Howwver, anorher problem happens. The docker begin loop to open a lot of tws windows and cannot login.

Any thoughts about this? Hope this can help to solve this bug. Previously, I didn't see this bug. So, it may be introduced in recent updates.

[gnzsnz]

Please test with a clean /config and clean /tmp mount

By clean I mean a dir that's empty.

If you're having user id mapping issues this might help to finger point the issue

[fuzihaofzh]

@gnzsnz Thanks for your kind help. I removed the tmp and config folder and do it again. the problem still exists.
Just wonder what do you mean by "having user id mapping issues", how can I check with this?
Thanks again for your help.

[gnzsnz]

you mentioned the user id issue.

run command id and ll /home/maple123/workbench/explore/Quant/configs/tws/config (based on your docker compose) check if they match

[fuzihaofzh]

Thank you for the feedback. My user ID is 1005, but the configuration appears to belong to the first user (ID 1000), which is not me. I believe this is because the tool enforces the user ID to be 1000, causing a mismatch with the file. When I set the PUID and GUID to 1005, the program enters a loop and continuously opens IB windows without stopping. I’m not sure what’s causing this behavior.

I just wanted to check—have you experienced this issue, particularly when using a paper account?

[gnzsnz]

the default id is 911, so the fact that shows 1000 means that the uid/gid update is working.

Everything points to docker with uid 1000 trying to write to your file-system with uid 1005 and having problems.

[fuzihaofzh]

For the second restart failing problem, can this problem be replicated in your machine? I have tried in different VPS in different locations and different paper accounts. The problem always exists. But when I try live account, it seems good. Any difference in handling paper/live accounts?

For the uid problem. I don't know whether current way of dumping files with a fixed user id (1000) is good. It may have some problems when reading the files that not belong to me. I tried to set the uid to mine (1005). Then, the program enters a dead loop and continuously opens more and more IB windows without stopping. I’m not sure what’s causing this behavior.

[gnzsnz]

please start TWS on paper, do the double restart, and send the logs.

the logs that you sent are for 40 minutes of execution, is difficult to undertand what's going on. most of what i see in the log looks OK, except a JAVA crash, that does not look good. but is unlikely to be a problem with the container. do you have enough RAM?

[fuzihaofzh]

I can confirm I have enough ram. I tried to migrate this on a new machine and see the following errors:

tws-1  | E: [pulseaudio] core-util.c: XDG_RUNTIME_DIR (/config/.XDG) is not owned by us (uid 1000), but by uid 0! (This could e.g. happen if you try to connect to a non-root PulseAudio as a root user, over the native protocol. Don't do that.)
tws-1  | E: [autospawn] core-util.c: XDG_RUNTIME_DIR (/config/.XDG) is not owned by us (uid 1000), but by uid 0! (This could e.g. happen if you try to connect to a non-root PulseAudio as a root user, over the native protocol. Don't do that.)
tws-1  | W: [autospawn] lock-autospawn.c: Cannot access autospawn lock.
tws-1  | E: [pulseaudio] main.c: Failed to acquire autospawn lock

I still suspect the way of saving the file have the permission problem. Did you see this error before?

[gnzsnz]

this explains your problem

https://forums.docker.com/t/user-ids-not-lining-up-and-permission-woes/122515

Lets remove docker for a second from the equation amd assume you installed an application using a package from your distros repository, which installs an application, creates a restricted user and runs the application as such restricted user. Furthermore lets assume you want to use a folder of your choosing, but the restricted user that runs the process is not allowed to write into that folder.

To resolve this situation you would allign either the ownership of the folder with the user that runs the process or less likely change the users uid/gid of the process to match the folder.

A container is nothing more than an isolated process on the host, so the solution looks pretty much the same. Make sure you align the uid/gid of the (host) folder owner to the uid/gid that runs the process.

and this one explains how linux server docker images (the base for TWS image) manages UID and GID

https://docs.linuxserver.io/general/understanding-puid-and-pgid/

that should solve your problem.

[Rjdrenth]

EDIT: This might be an upstream issue: IbcAlpha/IBC#267
On the off-chance that this issue is something that might be solved from the docker side of things, below you can find my original message.

In any case, I can reproduce this problem in 2 ways - for paper trading. I have not tested live trading.

• when running TWS and I restart from the menu the first time it works properly, but the second time it fails.
• when I kept running the Gateway it automatically restarted and failed. I mapped no volumes, so there should not be an issue with either UID or GID.

These are my Gateway logs:

_ib_gateway-ib-gateway-1_logs.txt

Note that at the end of the logs it says Click button: Re-login 2 times. Thos occurred after clicking the Ok button on the Unrecognised username of password window.

[gnzsnz]

I can't reproduce the issue with TWS 10.35.1g (latest), nor with 10.30.1u (stable). based on IbcAlpha/IBC#267 this seems to happen ramdomly, and seems to be comming from IBKR. so it's not something that can solved on the container.

moving this to discussion. as it's not something that can be fixed here