How to create a reproducible setup in an local development environment

[simon-retsch]

For local development we currently use the Novu docker solution. Since multiple developers work in our team we want to be able to reproduce the same local Novu setup for every local environment automatically.
The API already provides great possibilities to automatically create things like integrations, templates and subscribers with some lines of code. But we haven't found a possibility for:

• Creating a default admin/managing account
• Set an own API key to the Novu instance

Does any one know a solution to for one of these?

[jainpawan21]

Hi @simon-retsch

1. Right now all users have admin access, role based access control is in. our roadmap.
2. Can you please share a bit more about this?

[simon-retsch]

Thanks for the response. We have a web app where we use Django as backend and React as frontend. When developing, everything runs locally in docker containers. Now we want to add Notifications in the app using Novu. Especially the feature to run Novu locally on a docker container was a major reason for our decision to use Novu. We know about the possibilities of the environments, but we would prefer to develop on a docker container. In order to have the same default setup for every developer we want to establish that a prepared/build Novu instance already has a default user and a known/default (and therefore probably self-defined) API key.

[adschwartz]

@simon-retsch I started a thread about this topic in the discord. Would you be interested in collaborating on this problem?
https://discord.com/channels/895029566685462578/984071377701183518/1110212582897369208

[simon-retsch]

Thanks for the opportunity, @cedric-fauth resolved the task for us and is therefore also more qualified to collaborate on this topic.

[adschwartz]

@simon-retsch great! Was it the code to reverse engineer the novu web api that resolved it for you or something else? If so, is that code available, I'd love to be using the same solution and not having to reinvent that particular wheel. :)

[leonardbinet]

Actually, I have the same need for all environments.

In our case, we have dev, staging, demo, production and ci environments, in all those case we expect our state to be programatically generated (no manual intervention).

From what I see, it seems to be doable via idempotent operations via rest API, ie for templates it would look like this:

• list current templates
• create those missing
• deleting unintended ones

It seems fine, but maybe there could be a "declarative way" to create those, like a terraform provider?

[leonardbinet]

I see 2 potential approaches, not sure yet which would be the best, @simon-retsch how did it work out in your case?

• "isolated" approach: have isolated environments, ie no parentId specifying that an environment depends on another (maybe this could be added in the documentation), each environment state is programatically generated (approach of previous comment), the source of truth is the applicative code that generate the environment state at deployment:
  • benefits:
    • can be generated programatically on demand
  • drawbacks:
    • cannot use GUI (changes made via GUI would be then deleted at each deploy)
• "incremental" approach: dev is the root environment, staging depends on it (via parentId), demo depends on staging, prod depends on staging, it seems to best fit the way novu was designed. In this case on deployments the CI would promote environment changes (when deploying staging, CI would promote dev changes to staging environments, etc):
  • benefits:
    • can let some product folks use GUI to configure notifications
  • drawbacks:
    • I fear that a manual error couldn't be easily rollbacked

Note: for now I'm thinking based on documentation, not based on experimentation, I might have misunderstood some parts.

[adschwartz]

@leonardbinet It sounds like you have a use case where you'd like non technical people to interact/mutate the (Novu?) environment via a GUI. In the "incremental" approach: are you also promoting the data from environment to environment? If so, does this mean that test data from lower environments are promoted to production and is that ok for your use case?

Regarding the "isolated" approach: What if you had full symmetry in relation to data, meaning what if you could export the data programmatically as well as load it programmatically? In this case you don't have to lose the data that is generated inside environment. Would that help alleviate the drawback you mentioned or not?

[simon-retsch]

I see 2 potential approaches, not sure yet which would be the best, simon-retsch how did it work out in your case?

I think our approach differ a little from the one you need:

• local development: We use a local docker instance. The settings are created with an incremental and reproducible way (migrations) with realistic settings (mostly templates) that we also use on the other stages
• staging: The incremental solution also sets/updates the settings for the staging instance
• prod: Currently this needs to be promoted manually (from staging)

[cedric-fauth]

For local development we currently use the Novu docker solution. Since multiple developers work in our team we want to be able to reproduce the same local Novu setup for every local environment automatically. The API already provides great possibilities to automatically create things like integrations, templates and subscribers with some lines of code. But we haven't found a possibility for:

• Creating a default admin/managing account
• Set an own API key to the Novu instance

Does any one know a solution to for one of these?

We solved this now by reverse engineering the rest api of novu web (monitoring all network requests made when interacting with novu web). This uses bearer auth so we wrote a script that first creates a user, logs and saves the token from the response. Now we were able to make all kind of user requests like fetching the api key.

[cedric-fauth]

Sorry the code is not open source but for the self hosted version of novu we did the following steps to track and modify the state:

Sign up user and create organization:

1. sign up: POST /v1/auth/register with json payload: first/lastname, email, password
Save Auth Token in response["data"]["token"]

2. create org: POST /v1/organizations with payload name=orgname and Header "Authorization": "Bearer AUTHTOKEN"
Save org_id from response['data']['id']

3. Switch to this org (this needs to be done when signing up and it returns a new auth token)
POST /v1/auth/organizations/:org_id/switch with old auth header

Save the new auth token if you want which is in response["data"]. You'll need this if you want to retreive the api key or do further requests from now on because the old token is not valid anymore.

Logging in an existing user:

1. POST /v1/auth/login with email and password payload
You get an auth token under login_data["data"]["token"]

After sign up or login you get a new auth token which can be used in the header to make requests for the user.
E.g. retreive the api key:

Note: The Header "Authorization": "Bearer AUTHTOKEN" needs to be send with every request

1. get all environments: GET /v1/environments/

Now to get api key and app id you can do something like:
for env in response["data"]:
            if env["name"] == "Development":
                dev_app_id = env["identifier"]
                dev_api_key = env["apiKeys"][0]["key"]

Other user requests follow the same schema. For example you can switch the environment to production if you want to make changes there. This functionality is not provided by the application api. But after retrieving the api key via the users api we continued using the app api to create resources like templates.

For all other state related requests you can just monitor the network requests made by your browser then using the web app and making the same requests in your code.

[adschwartz]

@simon-retsch That's great, do you also have a way to export that state? Let's say that you have users or applications that have changed the state and you want to persist it so you can potentially import it again (using the code you already have)?

[simon-retsch]

No, we do not really have anything to export data or settings. We rather sync our data with Novu instead of storing it mainly there. For the settings like templates or feeds we user Django migrations to make Novu changes. With that we have the possibility to restore them by re-running the migrations.

[adschwartz]

@simon-retsch Got it. So your use case is mostly about applying your settings into Novu and not about extracting (exporting) the data again. Is that right?

[cedric-fauth]

Yes, but you can retreive all template data over the api if you want. But reimporting these data is a pain, because the objects you get back are not the exact same objects you would need for import into novu. You can modify these objects and delete attributes like _id etc. but there is no api documentation of how these objects need to look like and what fields they have. It's just trail and error.

@jainpawan21 It would be great if there was better documentation on how to create templates. API methods for creating are available but the field description is non-existent.

Much worse here I think.