Increase database's shared memory size in Docker Compose configuration

[Amaimersion]

I'm self-hosting RudderStack backend using Docker Compose. My docker-compose.yml is similar to official one:

version: "3.7"

services:
  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=rudder
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=jobsdb
    ports:
      - "6432:5432"
  backend:
    depends_on:
      - db
      - metrics-exporter
      - d-transformer
    image: rudderlabs/rudder-server:latest # latest version that matches 0.*.*
    entrypoint: sh -c '/wait-for db:5432 -- /rudder-server'
    ports:
      - "8080:8080"
    environment:
      - JOBS_DB_HOST=db
      - JOBS_DB_USER=rudder
      - JOBS_DB_PORT=5432
      - JOBS_DB_DB_NAME=jobsdb
      - JOBS_DB_PASSWORD=password
      - DEST_TRANSFORM_URL=http://d-transformer:9090
      - CONFIG_BACKEND_URL=https://api.rudderstack.com
      - WORKSPACE_TOKEN=<your_workspace_token> # Must be written without quotes e.g. WORKSPACE_TOKEN=20TokEO34NWitou2t3quO8teS7
      - STATSD_SERVER_URL=metrics-exporter:9125
      # DO NOT REMOVE - Mandatory env for Shopify
      - RSERVER_GATEWAY_WEBHOOK_SOURCE_LIST_FOR_PARSING_PARAMS=Shopify 
      # - RSERVER_BACKEND_CONFIG_CONFIG_FROM_FILE=true
      # - RSERVER_BACKEND_CONFIG_CONFIG_JSONPATH=<workspace_config_filepath_in_container> # For ex., /etc/rudderstack/workspaceConfig.json
    # Uncomment the following lines to mount workspaceConfig file
    # volumes:
    #   - <absolute_path_to_workspace_config>:<workspace_config_filepath_in_container> # Value for <workspace_config_filepath_in_container> should be same as the value provided for RSERVER_BACKEND_CONFIG_CONFIG_JSONPATH
  d-transformer:
    depends_on:
      - metrics-exporter
    image: rudderstack/rudder-transformer:latest
    ports:
      - "9090:9090"
    environment:
      - STATSD_SERVER_HOST=metrics-exporter
      - STATSD_SERVER_PORT="9125"
  # minio:
  #   image: minio/minio
  #   ports:
  #     - "9000:9000"
  #   command: server /data
  metrics-exporter:
    image: prom/statsd-exporter:v0.22.4
    ports:
      - "9102:9102"

Recently I noticed that sometimes the container crashes with following error:

rudderstack-backend-1  | 2025-02-03T10:17:00.400Z	ERROR	processor	processor/processor.go:3518	Failed to get unprocessed jobs from DB. Error: pq: could not resize shared memory segment "/PostgreSQL.1265551680" to 8388608 bytes: No space left on device
rudderstack-backend-1  | 2025-02-03T10:17:00.400Z	ERROR	runner.panic	crash/logger.go:33	Panic detected. Application will crash.	{"stack": "goroutine 1119 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\ngithub.com/rudderlabs/rudder-server/utils/crash.(*panicLogger).Notify.func1.1()\n\t/rudder-server/utils/crash/logger.go:34 +0x58\nsync.(*Once).doSlow(0x43bb25?, 0xc001409880?)\n\t/usr/local/go/src/sync/once.go:76 +0xb4\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:67\ngithub.com/rudderlabs/rudder-server/utils/crash.(*panicLogger).Notify.func1()\n\t/rudder-server/utils/crash/logger.go:32 +0x9c\npanic({0x41c7000?, 0xc00237e6c0?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/rudderlabs/rudder-server/processor.(*Handle).getJobs(0xc000c90008, {0xc00522a6c0, 0x1b})\n\t/rudder-server/processor/processor.go:3519 +0x9ca\ngithub.com/rudderlabs/rudder-server/processor.(*worker).Work(0xc003655300)\n\t/rudder-server/processor/worker.go:141 +0xc9\ngithub.com/rudderlabs/rudder-server/utils/workerpool.(*internalWorker).start.func1()\n\t/rudder-server/utils/workerpool/internal_worker.go:64 +0x2a7\ngithub.com/rudderlabs/rudder-server/rruntime.Go.func1()\n\t/rudder-server/rruntime/goroutine-factory.go:23 +0x57\ncreated by github.com/rudderlabs/rudder-server/rruntime.Go in goroutine 929\n\t/rudder-server/rruntime/goroutine-factory.go:21 +0x4f\n", "panic": "pq: could not resize shared memory segment \"/PostgreSQL.1265551680\" to 8388608 bytes: No space left on device", "team": "Core", "goRoutines": 834, "version": "1.41.0", "releaseStage": "development", "appType": "rudder-server-EMBEDDED"}
rudderstack-backend-1  | 2025-02-03T10:17:00.400Z	ERROR	runner.panic	crash/logger.go:33	goroutine 1119 [running]:
rudderstack-backend-1  | github.com/rudderlabs/rudder-go-kit/logger.(*logger).Fataln(0xc000aa8ea0, {0x48ddf45, 0x27}, {0xc0000e2008, 0x7, 0xc00182e300?})
rudderstack-backend-1  | 	/go/pkg/mod/github.com/rudderlabs/rudder-go-kit@v0.46.1/logger/logger.go:381 +0x314
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/utils/crash.(*panicLogger).Notify.func1.1()
rudderstack-backend-1  | 	/rudder-server/utils/crash/logger.go:33 +0x604
rudderstack-backend-1  | sync.(*Once).doSlow(0x43bb25?, 0xc001409880?)
rudderstack-backend-1  | 	/usr/local/go/src/sync/once.go:76 +0xb4
rudderstack-backend-1  | sync.(*Once).Do(...)
rudderstack-backend-1  | 	/usr/local/go/src/sync/once.go:67
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/utils/crash.(*panicLogger).Notify.func1()
rudderstack-backend-1  | 	/rudder-server/utils/crash/logger.go:32 +0x9c
rudderstack-backend-1  | panic({0x41c7000?, 0xc00237e6c0?})
rudderstack-backend-1  | 	/usr/local/go/src/runtime/panic.go:785 +0x132
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/processor.(*Handle).getJobs(0xc000c90008, {0xc00522a6c0, 0x1b})
rudderstack-backend-1  | 	/rudder-server/processor/processor.go:3519 +0x9ca
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/processor.(*worker).Work(0xc003655300)
rudderstack-backend-1  | 	/rudder-server/processor/worker.go:141 +0xc9
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/utils/workerpool.(*internalWorker).start.func1()
rudderstack-backend-1  | 	/rudder-server/utils/workerpool/internal_worker.go:64 +0x2a7
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/rruntime.Go.func1()
rudderstack-backend-1  | 	/rudder-server/rruntime/goroutine-factory.go:23 +0x57
rudderstack-backend-1  | created by github.com/rudderlabs/rudder-server/rruntime.Go in goroutine 929
rudderstack-backend-1  | 	/rudder-server/rruntime/goroutine-factory.go:21 +0x4f
rudderstack-backend-1  | 	{"stack": "goroutine 1119 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\ngithub.com/rudderlabs/rudder-server/utils/crash.(*panicLogger).Notify.func1.1()\n\t/rudder-server/utils/crash/logger.go:34 +0x58\nsync.(*Once).doSlow(0x43bb25?, 0xc001409880?)\n\t/usr/local/go/src/sync/once.go:76 +0xb4\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:67\ngithub.com/rudderlabs/rudder-server/utils/crash.(*panicLogger).Notify.func1()\n\t/rudder-server/utils/crash/logger.go:32 +0x9c\npanic({0x41c7000?, 0xc00237e6c0?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/rudderlabs/rudder-server/processor.(*Handle).getJobs(0xc000c90008, {0xc00522a6c0, 0x1b})\n\t/rudder-server/processor/processor.go:3519 +0x9ca\ngithub.com/rudderlabs/rudder-server/processor.(*worker).Work(0xc003655300)\n\t/rudder-server/processor/worker.go:141 +0xc9\ngithub.com/rudderlabs/rudder-server/utils/workerpool.(*internalWorker).start.func1()\n\t/rudder-server/utils/workerpool/internal_worker.go:64 +0x2a7\ngithub.com/rudderlabs/rudder-server/rruntime.Go.func1()\n\t/rudder-server/rruntime/goroutine-factory.go:23 +0x57\ncreated by github.com/rudderlabs/rudder-server/rruntime.Go in goroutine 929\n\t/rudder-server/rruntime/goroutine-factory.go:21 +0x4f\n", "panic": "pq: could not resize shared memory segment \"/PostgreSQL.1265551680\" to 8388608 bytes: No space left on device", "team": "Core", "goRoutines": 834, "version": "1.41.0", "releaseStage": "development", "appType": "rudder-server-EMBEDDED"}
rudderstack-backend-1  | panic: pq: could not resize shared memory segment "/PostgreSQL.1265551680" to 8388608 bytes: No space left on device [recovered]
rudderstack-backend-1  | 	panic: pq: could not resize shared memory segment "/PostgreSQL.1265551680" to 8388608 bytes: No space left on device
rudderstack-backend-1  | 
rudderstack-backend-1  | goroutine 1119 [running]:
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/utils/crash.(*panicLogger).Notify.func1()
rudderstack-backend-1  | 	/rudder-server/utils/crash/logger.go:43 +0x85
rudderstack-backend-1  | panic({0x41c7000?, 0xc00237e6c0?})
rudderstack-backend-1  | 	/usr/local/go/src/runtime/panic.go:785 +0x132
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/processor.(*Handle).getJobs(0xc000c90008, {0xc00522a6c0, 0x1b})
rudderstack-backend-1  | 	/rudder-server/processor/processor.go:3519 +0x9ca
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/processor.(*worker).Work(0xc003655300)
rudderstack-backend-1  | 	/rudder-server/processor/worker.go:141 +0xc9
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/utils/workerpool.(*internalWorker).start.func1()
rudderstack-backend-1  | 	/rudder-server/utils/workerpool/internal_worker.go:64 +0x2a7
rudderstack-backend-1  | github.com/rudderlabs/rudder-server/rruntime.Go.func1()
rudderstack-backend-1  | 	/rudder-server/rruntime/goroutine-factory.go:23 +0x57
rudderstack-backend-1  | created by github.com/rudderlabs/rudder-server/rruntime.Go in goroutine 929
rudderstack-backend-1  | 	/rudder-server/rruntime/goroutine-factory.go:21 +0x4f
rudderstack-backend-1 exited with code 0

The key message is pq: could not resize shared memory segment. No space left on device. Quick googling indicates that the problem is related to default low size of shared memory in a Docker container, which is 64MB. I increased it to 1GB by setting shm_size: 1g and the problem has gone.

Noteworthy, official Postgres Docker configuration increases shared memory right from the beginning by setting it to 128MB:

# Use postgres/example user/password credentials
version: '3.9'

services:

  db:
    image: postgres
    restart: always
    # set shared memory limit when using docker-compose
    shm_size: 128mb
    # or set shared memory limit when deploy via swarm stack
    #volumes:
    #  - type: tmpfs
    #    target: /dev/shm
    #    tmpfs:
    #      size: 134217728 # 128*2^20 bytes = 128Mb
    environment:
      POSTGRES_PASSWORD: example

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

I'm proposing that RudderStack default Docker Compose configuration will ship with increased shared memory for database. Default may be kept low enough. The key thing is to indicate that this setting can be (and maybe worth be) increased.

So instead of:

db:
  image: postgres:15-alpine
  environment:
    - POSTGRES_USER=rudder
    - POSTGRES_PASSWORD=password
    - POSTGRES_DB=jobsdb
  ports:
    - "6432:5432"

use:

db:
  image: postgres:15-alpine
  environment:
    - POSTGRES_USER=rudder
    - POSTGRES_PASSWORD=password
    - POSTGRES_DB=jobsdb
  ports:
    - "6432:5432"
  shm_size: 128mb

Also consider if 128MB is good enough for default value. I'd rather to use 256MB just to make sure that there will be enough memory for the shared cache.

[gitcommitshow]

Thank you for the detailed suggestion @Amaimersion. Let me brainstorm this with the team and get back to you.
To me, this looks like a useful change. Feel free to raise a PR for the same.

[Amaimersion]

If no one wants to take over this one-line PR, then I can do it 🙂 The main task here is to decide what size of shm_size is appropriate by default. For me Postgres official default 128MB makes sense.

[gitcommitshow]

Yes, you're right. I confirmed with the team, we internally use shm_size 256MB in many tests.
Let's set it to 128 MB for now and then we'll see if anyone else want to set the default higher.
If it is not too much trouble for you, can you raise the PR @Amaimersion ? This will be better for collaboration.

[Amaimersion]

Sure. Let me handle it in near time

[Amaimersion]

@gitcommitshow Done #5541

[Amaimersion]

PR #5541 merged, so I'm closing this. Thanks!

[gitcommitshow]

Thank you @Amaimersion for your contribution 🙏

[gitcommitshow]

Btw, do join our RudderStack community on Slack, in case you're not there already. It is always good to have another data engineering expert in the community and also we organize exclusive RudderStack contributor meet-ups there.