-
|
This is something that has been bugging me for some time now, and I'm finally taking the time to understand it. The end-goal is to have a single artefact for the entire Changelog app that goes out into production. Currently, this is a container image that is the end result of This is what we do currently:
The longest and most resource-hungry step is 2. Step 1 feels somewhat redundant. I would like to think that I am missing the trick that pulls all dependencies at once for all environments. Do I really need to run it 3 times, one for each: I think that 4 & 5 are outdated by now and should be replaced by esbuild: https://github.com/phoenixframework/esbuild . I have not spent enough time with. this, but the whole node.js & webpack dependency always felt too heavy-weight. At step 6, I expect you to tell me that I should be using mix releases, as shown here: https://github.com/gerhard/noted/blob/5dd6efcc2a28d5775381f82da1651647b42f8a3b/Dockerfile#L41 . That's what I think I know, but I am sure that I am missing a few things. I am very curious how you would approach this @lawik & @akoutmos . If you know anyone that could help, tag them. I am very keen on capturing the canonical implementation that you think is a good default for Phoenix/Elixir apps. Yes, there is a bigger picture here, but for now I would very much like to solve this in the context of changelog.com. Thank you! cc @nickjj because I know that he is keen on this type of stuff 😉 |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 5 replies
-
|
Interesting question. So esbuild is a neat thing. We can set that aside I think as that's a tool transition. And you'll need some kind of way to wrangle that stuff regardless. Compiling can certainly take a fair bit of time. Unless you use dialyzer for static analysis or have really slow tests the compile step is reasonably the slowest. Not sure about the value of compiling all deps in parallel as I think the compiler is already quite parallel and I wonder if that might starve other resources. Memory, IO or so. Top of mind I don't see any reason to build for the dev env. I don't believe my similar workflows require that. I don't see how step 4 requires it. A mix release would package all you need for running the app, either obviating the need for a container or letting you make the container rather small. It can be a bare debian slim, ubuntu or alpine one. Doesn't need Elixir or Erlang. I imagine you ship those runtimes if you aren't using a mix release. Haven't honestly tried without mix release in ages. Wouldn't be a huge difference regardless I imagine but slightly more mainstream to use a release. I've shipped those bare or in a container. There is the option of burrito if you want a single binary built containing your release. Haven't tried it but seems cool. An evolution of the bakeware project. I typically make a variant of this from the Phoenix docs: https://hexdocs.pm/phoenix/releases.html#containers It's a two-stage dockerfile so the release comes into paly for creating a second smaller image. It also has decent ordering for good caching I think. |
Beta Was this translation helpful? Give feedback.
-
Is there a command that gets & compiles all deps in parallel, across all environments? My understanding is that I need to run MIX_ENV=dev MIX_ENV=test MIX_ENV=prod
mix deps.get mix deps.get mix deps.get
| | |
v v v
MIX_ENV=dev MIX_ENV=test MIX_ENV=prod
mix do deps.compile, compile mix do deps.compile, compile mix do deps.compile,compile
# all the above run in parallel, each mix instance spinning it's own beam.smp processI now realise that this is sufficient: mix deps.get -------------------\
| |
v v
MIX_ENV=dev MIX_ENV=test
mix do deps.compile, compile mix do deps.compile, compileAnd now I am wondering if this exists (or if it should exist) mix --all deps.get
|
v
mix --all do deps.compile, compile
You are absolutely right, I had a disconnect with some volumes that were not being mounted correctly, and as a result the build cache was not getting populated correctly. Thank you!
Yes, I was meant to improve this for years, and will do so as soon as I shift some of the current context. There is a lot of good stuff coming in this area any week now 😉
Interesting! Will check it out, thanks!
Another excellent reference, thank you, will check it out too. 🙌🏻 |
Beta Was this translation helpful? Give feedback.
-
|
I imagine you might be able to copy the built test deps into the prod build dir but you probably shouldn't. You almost certainly have test-only deps. There may also be parts of your deps that rely on compile-time config, macros and other fun things that might lead to a different end product between environments. It probably shouldn't differ beyond the test-only deps but I wouldn't put money on it. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for CC'ing me.
Step 5 of digesting the files will still be a requirement with esbuild. This is the step that adds the md5 digest to the file name of your assets -- these assets could come from Webpack, esbuild, etc.. I do have an example Dockerfile that handles releases at https://github.com/nickjj/docker-phoenix-example/blob/main/Dockerfile. The final production image copies in the I'm not sure if my example addresses your concerns but feel free to pick anything you want out of it. One thing I tried to accomplish was if |
Beta Was this translation helpful? Give feedback.
-
Got it!
That is really helpful, thank you! |
Beta Was this translation helpful? Give feedback.
-
|
It will be interesting to see what you come up with. A common flow is to build things with your test libraries available, run your tests but now after the tests pass you end up re-building a new image in prod mode and then pushing that. This pattern has always felt dirty to me because you're technically shipping a different image than what you tested and you do need to build things twice (in Do you have a plan moving forward to reduce this down to 1 build while still being able to run tests and ship a production image? |
Beta Was this translation helpful? Give feedback.
-
|
I understand what you mean, there is a lot more to improve. This is already miles better than what we had before: #395 I am curious to explore what a canonical Dagger package for Elixir/Phoenix apps would look like. I intend to pull multiple people in that discussion, including Chris McCord & José Valim. But that is one for next year, for sure. For now, I am looking forward to finishing Ship It episode 33 for you all 🎄 |
Beta Was this translation helpful? Give feedback.
Interesting question.
So esbuild is a neat thing. We can set that aside I think as that's a tool transition. And you'll need some kind of way to wrangle that stuff regardless.
Compiling can certainly take a fair bit of time. Unless you use dialyzer for static analysis or have really slow tests the compile step is reasonably the slowest.
Not sure about the value of compiling all deps in parallel as I think the compiler is already quite parallel and I wonder if that might starve other resources. Memory, IO or so.
Top of mind I don't see any reason to build for the dev env. I don't believe my similar workflows require that. I don't see how step 4 requires it.
A mix release would package all …