Add possibility to have "SAFE" environnment variables ?

[kjorand]

First of ... love your work @harrisonpim !
Thanks a lot !
It was exactly what I was looking into, planning to write jsut a simple script ... but then that whole package makes it simpler to use / reuse !

Then ... In a project we have quiet a few variables that are non-secret (more like defining how it should behave/almost config variables). I'd like those to still be pushed ... I was thinking of a specific syntax where preceding a line with #SAFE (could be #public) on the previous line would stop stripping it. We have already some comments describing function and working of them ... so adding that specific comment shouldn't be a big hassle ...

Also, taking the approach where you have to mark the ones that are safe to push ... seems pretty reasonable to me security wise (unlike having to prefix the secret ones where ... you ... could forget to mark a secret) ... But I get it that ... then it's a more "lightenend" filtering ...

Also it's so easy anyway to comment out a variable and it's actually a secret ... [not really sure how handle that security risk however since ... we usually definitely want the comments in the stripped out version ... ]

I'm planning to do it anyway on our repo ... but wanted to check on you if that'd be of any interest to you (in which case I'd follow the "CONTRIBUTING" guide ... else not necessarily).

I'm also thinking it could be possible to have the two different filters in one "package" and having the ability to install any of the two and so on ... (could be of interest to you / the community) ... but that I don't have the knowledge (yet) nor am willing to do...

So yeah in any case ... let me know !

[harrisonpim]

Hi @kjorand! Thanks for posting this here ☺️ I'm really happy to hear that you've found the tool useful/interesting!

This sounds like a sensible feature to add, and I appreciate the thought you've put into possible implementations :) I totally agree about the need to explicitly mark some variables as safe, rather than the other way around.

I can see a clear path forward using your suggestion, but I'm also interested in exploring some alternative ways we could achieve the same results. Personally, I feel a bit uncomfortable about marking safe/unsafe variables with comments (mostly because the way that people use comments varies a lot from user-to-user, project-to-project, and I don't want to mandate the style that people follow within their files any more than I am already!).

I'd feel more comfortable separating safe/unsafe variables into totally separate .env files, and marking the safe file with some kind of extra config file... That way, people could write comments across files in whatever way they want to, without worrying about whether it will affect the secret stripping.

For example, that might look something like:

my_project
├── src
│   └── ...
├── .dotenv-stripout-ignore
├── safe.env
└── unsafe.env

the .dotenv-stripout-ignore file might look like

safe.env

unsafe.env might contain

# here are some comments
# i can add comments however i choose
THIS_VARIABLE_SHOULD_BE_SECRET=
SO_SHOULD_THIS_ONE=

and safe.env might contain

THIS_VARIABLE_IS=okay_to_share
AND_THIS_ONE_IS=too
# also able to add comments in this file, in whatever format I want!

If the tool knows that it should look for a .dotenv-stripout-ignore file and leave any files mentioned within it unstripped, we should be able to bypass the filter and users would be able to share the contents of those .envs like any other file :) For every file which isn't mentioned in .dotenv-stripout-ignore (or for all files if a .dotenv-stripout-ignore doesn't exist), their contents would be stripped as usual.

What do you think? Would an approach like this work for you?

[harrisonpim]

Hi @kjorand, sorry for the slow response. I've been thinking about the problem carefully, and reading some more about best practices for local storage of env/config variables (eg https://12factor.net/config). I almost have a clear plan, but I'm not quite there yet...

It would be great to know whether you've found any examples of tools which don't allow for the use of multiple .env files. Everything I've seen allows for multiple files, but knowing how the edge cases work would be really helpful.

[kjorand]

Hi @harrisonpim
Sure, no worries, no pressure ! It was just to know how the project would evolve ... I'm having some env files here on a project that I need to (partially) push ... so it was just to decide which solution to set up ... =) ...

So I also did a little research ... found none of the tools I work with currently have a limitation with multiple files ...

Other random findings :

• Most of the discussion / documentation I read was only talking about multiple files for multiple environnements (.env.prod, .env.dev, .dev.staging ... etc)
• Found none that'd recommend using several in a single run
• Found one discouraging use of multiple .env.* : https://github.com/motdotla/dotenv#should-i-have-multiple-env-files
• Found one that, quiet opposed to all that by default loads all .env.* files in the root directory (??)

The more I think of your solution, the more it makes sense to me (but I admit it still seems a little more annoying to have to setup a second env file and have to setup a config-file for this ... especially in a relatively small environnement [limited number of variables] )

As a final word (for tonight at least) ... in the meantime I've had the following thoughs regarding security : from the point of view of a newcomer on a project ... I would fear that : (at least it's a risk to consider)

• They'd have a new secret key to add
• Find an env file ...
• (maybe it turns out it contains already most of the SAFE settings for the part he's working on ... )
• Adds it there ... only to discover after commit/pushing his work that the has been pushed (and once it has been pushed on a public repo [or even a private with some specific relation to a public one, as I recently learnt] ... it is publicly available forever, on github)

Yeah I know ... not very likely ... but ... definitely a case to consider (at least to prepare against => give a proper introduction to newcomers ... )

Also that made me realise that in my suggestion : # SAFE also isn't great in that context ... security wise, it'd be better to have something more explicit as # PUBLIC ...

In any case, thanks for the great job, kudos for all the effort you put in !!

[harrisonpim]

It looks like we've been reading a lot of the same resources!

Thanks again for all of the thought that you've put into this - I really appreciate it 😁

I've decided to move ahead with the .dotenv-stripout-ignore option, and I've had a go at implementing a solution in this PR - please take a look and let me know what you think!

This discussion has also given me the motivation to make the test suite (and documentation!) more robust and exhaustive, but I'll leave that work for a separate PR. As you said, I think it's especially important that users who want to commit safe variables understand the expected behaviour of the tool, and better docs will help a lot with that.

[kjorand]

Looking great !
I'm not fluent enough in python to assert its perfection ... but looking pretty good !
I have to admit I'm looking forward to test it !

PS : bad news is I haden't actually tested when doing my research ... and it's not exactly very well documented, but next doesn't actually allow you to choose arbitrary env file names ... =(

Found it out while testing and my env files weren't loading anymore after separating and renaming then in preparation for the .dotenv-stripout-ignore ... Also found a better doc explaining it ... and ended up diving deeper found out it is also explained on the next website ... casually sitting at the almost very end of the documentation page ... very (UN-)conveniently after refering dozen of times to them as .env.* ... leading me to wrongly believe (cf previous comment) that they were all loaded ...

So ... now I'm not sure if I'm gonna "hijack" the next convention or ... deal with it diffrently ...

[kjorand]

Actually ... I really (REALLY!!) dislike that "hijacking" idea (again for security reasons ... ).

It'd be way too easier to forget about that ... and just follow the next conventions (which really make sense with those names) and end up leaking secrets ...

Don't know ... either I'll keep doing it by hand (staging sensitive variables only up to the "=" ... or ... don't know yet ... I'll take some time to think about it).

Currently my mind's very divided ... they're convention makes a ton of sense to me and allows to make tings easier ... but at the same time the lack of flexibility / personnalisation possibility annoys me a bit ... I'd love to be able to have a .env.config .env.production .env.production.public ... or something ...

Same goes for the NEXT_PUBLIC_ convention ... in essence I like it : simple, efficient, PUBLICITY right in the name ... but damn the NEXT_ makes it way too proprietary for me to fall in love with that convention !