Need help with heavily modified epic dev stack

[vonkoff]

Hello,

I heavily modified the epic dev stack.... I would really appreciate help in any form... I just don't know what to do and my current prod for my live site if failing. Until I get this fixed I can't replace it and I don't know how to find out what the issue is.

Any guidance on what is needed would be greatly appreciated it.

This is the error below:

/usr/bin/docker run --name c9a4a5cd27d3e6b879447bb7bcc56bfffbc3a4_b9eb21 --label c9a4a5 --workdir /github/workspace --rm -e "FLY_API_TOKEN" -e "INPUT_ARGS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/remix-guj/remix-guj":"/github/workspace" c9a4a5:cd27d3e6b879447bb7bcc56bfffbc3a4 deploy --remote-only --build-arg COMMIT_SHA=cb9d304e340b793558d3e16b96a88c498b306d9a --app remix-staging-guj
==> Verifying app config
--> Verified app config
Validating /github/workspace/fly.toml
Platform: nomad
✓ Configuration is valid
==> Building image
Waiting for remote builder fly-builder-quiet-sound-4473...
Remote builder fly-builder-quiet-sound-4473 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
#1 [internal] load remote build context
Sending build context to Docker daemon  6.823MB

#1 DONE 0.0s

#2 copy /context /
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/node:18-bullseye-slim
#3 DONE 0.4s

#4 [base 1/2] FROM docker.io/library/node:18-bullseye-slim@sha256:1ba1ddfc61b385b6436fd0fa0d1d42d322a0cd03c1ff110fa39e828511152aef
#4 DONE 0.0s

#9 [deps 2/3] ADD package.json package-lock.json .npmrc ./
#9 CACHED

#15 [build 3/6] ADD prisma .
#15 CACHED

#10 [deps 3/3] RUN npm install --include=dev
#10 CACHED

#5 [base 2/2] RUN apt-get update && apt-get install -y fuse3 openssl sqlite3 ca-certificates
#5 CACHED

#11 [production-deps 2/4] COPY --from=deps /myapp/node_modules /myapp/node_modules
#11 CACHED

#8 [production-deps 1/4] WORKDIR /myapp
#8 CACHED

#16 [build 4/6] RUN npx prisma generate
#16 CACHED

#17 [build 5/6] ADD . .
#17 DONE 0.1s

#18 [build 6/6] RUN npm run build
#18 1.089 
#18 1.089 > build
#18 1.089 > run-s build:*
#18 1.089 
#18 1.721 
#18 1.721 > build:remix
#18 1.721 > remix build --sourcemap
#18 1.721 
#18 3.539 [info] building... (NODE_ENV=production)
#18 3.540 [warn] 🚨  source maps enabled in production
#18 3.540 ┃ You are using `--sourcemap` to enable source maps in production,
#18 3.540 ┃ making your server-side code publicly visible in the browser.
#18 3.540 ┃ This is highly discouraged!
#18 3.540 ┃ If you insist, ensure that you are using environment variables for secrets
#18 3.540 ┃ and are not hard-coding them in your source.
#18 3.540 ┗
#18 8.016 [info] built (4.4s)
#18 9.068 
#18 9.068 > build:server
#18 9.068 > tsx ./other/build-server.ts
#18 9.068 
#18 9.623 
#18 9.623 building...
#18 9.634 
#18 9.634   server-build/index.js      4.9kb
#18 9.634   server-build/index.js.map  9.8kb
#18 9.634 
#18 9.634 ⚡ Done in 4ms
#18 DONE 9.7s

#7 [stage-4  2/10] WORKDIR /myapp
#7 CACHED

#13 [production-deps 4/4] RUN npm prune --omit=dev
#13 CACHED

#19 [stage-4  4/10] COPY --from=build /myapp/node_modules/.prisma /myapp/node_modules/.prisma
#19 CACHED

#12 [production-deps 3/4] ADD package.json package-lock.json .npmrc ./
#12 CACHED

#14 [stage-4  3/10] COPY --from=production-deps /myapp/node_modules /myapp/node_modules
#14 CACHED

#6 [stage-4  1/10] RUN echo "#!/bin/sh\nset -x\nsqlite3 $DATABASE_URL" > /usr/local/bin/database-cli && chmod +x /usr/local/bin/database-cli
#6 CACHED

#20 [stage-4  5/10] COPY --from=build /myapp/server-build /myapp/server-build
#20 CACHED

#21 [stage-4  6/10] COPY --from=build /myapp/build /myapp/build
#21 DONE 0.0s

#22 [stage-4  7/10] COPY --from=build /myapp/public /myapp/public
#22 DONE 0.0s

#23 [stage-4  8/10] COPY --from=build /myapp/package.json /myapp/package.json
#23 DONE 0.0s

#24 [stage-4  9/10] COPY --from=build /myapp/prisma /myapp/prisma
#24 DONE 0.0s

#25 [stage-4 10/10] ADD . .
#25 DONE 0.1s

#26 exporting to image
#26 exporting layers 0.1s done
#26 writing image sha256:590bc28689263d04736e1[30](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:31)5159527c58134f927aee1a99510e6813f3205f2ba done
#26 naming to registry.fly.io/remix-staging-guj:deployment-01H5K6EFG5S2ZE9S8B36K3QY8K done
#26 DONE 0.1s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/remix-staging-guj]
71c982a4ca04: Preparing
cf7e48afc7cd: Preparing
fcb755295e56: Preparing
de76304a3ea8: Preparing
96e7ccee0a6a: Preparing
19003ab37af0: Preparing
33db105fe198: Preparing
bf7a68793954: Preparing
1a1a4a88d85e: Preparing
adcfa5956c62: Preparing
1a1f46636e5f: Preparing
6381d2[31](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:32)3ce4: Preparing
bce5f4fa6540: Preparing
9fde10389920: Preparing
de28499355cf: Preparing
4b3ba104e9a8: Preparing
1a1a4a88d85e: Waiting
adcfa5956c62: Waiting
19003ab37af0: Waiting
[33](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:34)db105fe198: Waiting
bf7a68793954: Waiting
1a1f46636e5f: Waiting
9fde10389920: Waiting
6381d2313ce4: Waiting
bce5f4fa6540: Waiting
de28499355cf: Waiting
4b3ba104e9a8: Waiting
fcb755295e56: Pushed
96e7ccee0a6a: Pushed
cf7e48afc7cd: Pushed
19003ab37af0: Layer already exists
33db105fe198: Layer already exists
bf7a68793954: Layer already exists
1a1a4a88d85e: Layer already exists
adcfa5956c62: Layer already exists
1a1f46636e5f: Layer already exists
6381d2313ce4: Layer already exists
bce5f4fa6540: Layer already exists
9fde10389920: Layer already exists
de28499355cf: Layer already exists
4b3ba104e9a8: Layer already exists
de76304a3ea8: Pushed
71c982a4ca04: Pushed
deployment-01H5K6EFG5S2ZE9S8B36K3QY8K: digest: sha256:3ba2f688bfee7161c9b10a1d9a9f72a1f537de7d4a09ec976284415d3b16619a size: 3671
--> Pushing image done
image: registry.fly.io/remix-staging-guj:deployment-01H5K6EFG5S2ZE9S8B36K3QY8K
image size: 452 MB
==> Creating release

Watch your app at https://fly.io/apps/remix-staging-guj/monitoring

--> release v108 created

--> You can detach the terminal anytime without stopping the deployment
==> Release command detected: node ./other/sentry-create-release

--> This release will not be available until the release command succeeds.
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Unpacking image
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Configuring firecracker
	 Configuring firecracker
	  INFO Preparing to run: `docker-entrypoint.sh node ./other/sentry-create-release` as root
	 Configuring firecracker
	  INFO Preparing to run: `docker-entrypoint.sh node ./other/sentry-create-release` as root
	  INFO Preparing to run: `docker-entrypoint.sh node ./other/sentry-create-release` as root
	 Configuring firecracker
	  INFO Preparing to run: `docker-entrypoint.sh node ./other/sentry-create-release` as root
	 2023/07/18 00:30:33 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 2023/07/18 00:30:33 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 2023/07/18 00:30:33 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 2023/07/18 00:30:33 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	  INFO Main child exited normally with code: 0
	  WARN hallpass exited, pid: 246, status: signal: 15 (SIGTERM)
	  INFO Main child exited normally with code: 0
	  WARN hallpass exited, pid: 246, status: signal: 15 (SIGTERM)
	  INFO Main child exited normally with code: 0
	  WARN hallpass exited, pid: 246, status: signal: 15 (SIGTERM)
	 Missing Sentry environment variables, skipping sourcemap upload.
	  INFO Main child exited normally with code: 0
	  WARN hallpass exited, pid: 246, status: signal: 15 (SIGTERM)
	 2023/07/18 00:30:[34](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:35) listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 2023/07/18 00:30:34 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 2023/07/18 00:30:34 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 2023/07/18 00:30:34 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Preparing kernel init
	 Configuring firecracker
	 Starting virtual machine
	  INFO Starting init (commit: 1d1821d)...
	  INFO Preparing to run: `docker-entrypoint.sh node ./other/sentry-create-release` as root
	 2023/07/18 00:30:33 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 Missing Sentry environment variables, skipping sourcemap upload.
	  INFO Main child exited normally with code: 0
	  INFO Starting clean up.
	  WARN hallpass exited, pid: 246, status: signal: 15 (SIGTERM)
	 2023/07/18 00:30:34 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Preparing kernel init
	 Configuring firecracker
	 Starting virtual machine
	  INFO Starting init (commit: 1d1821d)...
	  INFO Preparing to run: `docker-entrypoint.sh node ./other/sentry-create-release` as root
	 2023/07/18 00:30:33 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
	 Missing Sentry environment variables, skipping sourcemap upload.
	  INFO Main child exited normally with code: 0
	  INFO Starting clean up.
	  WARN hallpass exited, pid: 246, status: signal: 15 (SIGTERM)
	 2023/07/18 00:30:34 listening on [fdaa:0:8864:a7b:28df:5c16:a5d5:2]:22 (DNS: [fdaa::3]:53)
Logs: https://fly.io/apps/remix-staging-guj/monitoring
==> Monitoring deployment

v108 is being deployed
812cc[37](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:38)4: lhr2 running healthy
812cc374: lhr2 running unhealthy [health checks: 2 total, 2 warning]
812cc374: lhr2 running unhealthy [health checks: 2 total, 1 warning, 1 critical]
812cc374: lhr2 running unhealthy [health checks: 2 total, 2 critical]
Failed Instances

Instance
Failure #1
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 	HEALTH CHECKS      	RESTARTS	CREATED   
812cc374	app    	108    	lhr2  	run    	running	2 total, 2 critical	0       	4m59s ago	

Recent Events
TIMESTAMP           	TYPE      	MESSAGE                 
2023-07-18T00:30:[42](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:43)Z	Received  	Task received by client	
2023-07-18T00:30:42Z	Task Setup	Building Task Directory	
2023-07-18T00:31:07Z	Started   	Task started by client 	

2023-07-18T00:30:44Z   [info]Starting instance
2023-07-18T00:30:44Z   [info]Configuring virtual machine
2023-07-18T00:30:44Z   [info]Pulling container image
2023-07-18T00:30:54Z   [info]Unpacking image
2023-07-18T00:31:06Z   [info]Preparing kernel init
2023-07-18T00:31:07Z   [info]Configuring firecracker
2023-07-18T00:31:07Z   [info]Starting virtual machine
2023-07-18T00:31:07Z   [info] INFO Starting init (commit: 1d1821d)...
2023-07-18T00:31:07Z   [info] INFO Preparing to run: `docker-entrypoint.sh npm start` as root
2023-07-18T00:31:07Z   [info]2023/07/18 00:31:07 listening on [fdaa:0:8864:a7b:a98:812c:c374:2]:22 (DNS: [fdaa::3]:53)
2023-07-18T00:31:08Z   [info]> start
2023-07-18T00:31:08Z   [info]> cross-env NODE_ENV=production node .
2023-07-18T00:31:09Z   [info]🚀  We have liftoff!
2023-07-18T00:31:09Z   [info]Local:            http://localhost:8081
2023-07-18T00:31:09Z   [info]On Your Network:  http://172.19.64.58:8081
2023-07-18T00:31:09Z   [info]Press Ctrl+C to stop
2023-07-18T00:31:13Z   [error]Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 4[43](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:44)] will have intermittent failures until the health check passes.
2023-07-18T00:31:17Z   [error]Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, [44](https://github.com/goodunionjobs/remix-guj/actions/runs/5581948126/jobs/10200676770#step:6:45)3] will have intermittent failures until the health check passes.

--> v108 failed - Failed due to unhealthy allocations - not rolling back to stable job version 108 as current job has same specification and deploying as v109 

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error: abort

[kentcdodds]

I'm afraid I can't find anything in there that explains why it may be not working. It looks like it starts up without issue. I'm guessing you've not done anything to change the LiteFS proxy server that handles the 8080 traffic and forwards it to port 8081 where your app is listening. That's the only thing I can think of.

[vonkoff]

So since I am not using LiteFS I changed the Port to 8080 in the dockerfile ENV PORT="8080". Thanks for that.

I'm getting closer...at the moment now in dev. I didn't notice but at some point with my commits I guess HMR went away somehow and with that I can't see my Algolia hits. I can load in data because I am using prisma to connect to my planetscale DB... I guess the prod site is gonna have to be down until I try this out again.

Think it best to go back through my commits and see at which point I lost HMR and the Algolia hits from showing up.

[vonkoff]

I am not sure still. I got the app running somewhat https://remix-staging-guj.fly.dev/

But, getting a plethora of errors in console:

react-dom.production.min.js:131 Uncaught Error: Minified React error #418; visit https://reactjs.org/docs/error-decoder.html?invariant=418 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at ki (react-dom.production.min.js:131:278)
    at Ba (react-dom.production.min.js:293:379)
    at Ua (react-dom.production.min.js:280:389)
    at Af (react-dom.production.min.js:280:362)
    at Fa (react-dom.production.min.js:268:300)
    at Sl (scheduler.production.min.js:13:203)
    at MessagePort.hl (scheduler.production.min.js:14:128)
react-dom.production.min.js:293 Uncaught Error: Minified React error #423; visit https://reactjs.org/docs/error-decoder.html?invariant=423 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at Ba (react-dom.production.min.js:293:39)
    at Ua (react-dom.production.min.js:280:389)
    at Vf (react-dom.production.min.js:280:320)
    at qr (react-dom.production.min.js:280:180)
    at Ii (react-dom.production.min.js:271:88)
    at Fa (react-dom.production.min.js:268:429)
    at Sl (scheduler.production.min.js:13:203)
    at MessagePort.hl (scheduler.production.min.js:14:128)
    ```
    
Agina, this all works with `npm run dev`

[vonkoff]

I'm not sure if any of these cause issues as well.

[vonkoff]

			directives: {
				'connect-src': [
					MODE === 'development' ? 'ws:' : null,
					process.env.SENTRY_DSN ? '*.ingest.sentry.io' : null,
					"'self'",
					'*.algolia.net',
				].filter(Boolean),

This was the answer ^^. Always a bit simple. Anyways had a fun time debugging. Also, switched over to Sqlite3 from Planetscale Mysql database. Kind of went back and did this whole project how it was meant to be used and everything is working out GREAT! Even though the whole issue could have been fixed with one line lol.