How do you best handle normalizing data from nested MST models fetched from the API

[derekyau]

When fetching data from an external REST API, its fairly typical to have nested models in the response. To set up the context for my question I'll define a few things here

GET /projects/:id might return something like

{ 
 id: 1,
 name: "Project XYZ",
 tasks: [
    {id: 1, name: "Task 1", status: "done},
    {id: 2, name: "Task 2", status: "in progress"}
  ]
}

If we have a MST models defined as such:
ProjectModel

...props({
  id: types.identifier,
  name: types.maybeNull(types.string),
  tasks: types.array(types.reference(TaskModel))
})

TaskModel

...props({
  id: types.identifier,
  name: types.maybeNull(types.string),
  status: types.maybeNull(types.string)
})

In the above example, fetching a project from the API also fetches nested task objects. Say I have two additional stores that each contain a MST map (where we store the data from the API), something like:

projectMap: types.map(ProjectModel),
taskMap: types.map(TaskModel),

Later, when we use a flow function to fetch the data from the API, we need to remember to do something like:

  const response = <make API call here>
  let projectData = response.data
  // loop through response to store all tasks
  projectData.tasks.forEach((task) => self.taskMap.put(task))

  // transform the projectData response to include identifiers in the array instead of the full object
  projectData.tasks = projectData.tasks.map((task) => task.id))
  self.projectMap.put(projectData)

We can make the above code a bit better by using preProcessor and other mixins that take care of merging / updating data, but in general this becomes pretty tedious especially when there are multiple nested objects within one API call. Keeping the data up to date in these cases gets pretty gnarly as we have to mergeUpdate many models and their related models.

This works, but seems highly inefficient (and causes significant lag) as various models are transformed / updated / and put into their various maps. So ultimately my question is... is there a better way / generally accepted pattern to do this?

I've been seeing the popular useQuery hook that seems to solve this problem under the hood. I'm also aware there is a mst-query library that seems? like might help with something like this?

Outside of switching our codebase to use libraries like that, I'm curious - how are people handling this right now? Or is the way we are fetching / updating data in our stores flawed?

[EmilTholin]

Hi @derekyau!

The process you outline in your last code snippet is generally how I go about doing it. It's a bit of tedious manual work, but it makes it obvious what's happening if you are working with references.

I know it's just an example in your question, but I'm wondering if your tasks are strongly tied to a project if you need to model the tasks as references? Maybe it would be easier to just have the TaskModel objects nested directly under the ProjectModel objects? This way you can just instantiate the ProjectModel with the API data as is.

const ProjectModel = types.model('ProjectModel', {
  id: types.identifier,
  name: types.maybeNull(types.string),
  tasks: types.array(TaskModel)
});

[derekyau]

Thanks for the response @EmilTholin - I think through various projects, my team and I have always gone back and forth on the best way to model the data in MST-stores.

Pros of using types.array is that we can do (as you suggested just apply the object as is).

Some challenges we have had though:

• Sometimes it is useful to have access to the task object in the above example without the parent project in a specific view, or vice versa, so having access to it anytime not coupled with the project is useful (say for example if there was a get /tasks endpoint and we only fetch related project object on demand if needed
• Arrays have some drawbacks since for example to find and update a specific item in the array we'd have to use array operations array.find((task) => task.id == id)) or something to find and update one specific task if we needed to do that. In general, MST maps with the put / delete / etc. interface with first order support for MST identifiers makes for a much easier time fetching and manipulating the data.

[EmilTholin]

Yeah, there is no one-size-fits-all way to model your data across projects, that's for sure.

Sometimes it is useful to have access to the task object in the above example without the parent project in a specific view

Yes, you're right. This is a real fork in the road for how to model your data, but many times I find you can figure out what project it belongs to, but not always of course.

// Example with React Router on url "/projects/:projectId/tasks/:taskId/"
const TaskDetailView = observer(() => {
  const { projectId, taskId } = useParams();
  const { projects } = useRootStore();

  const project = projects.find(p => p.id === projectId);
  if (!project) return null;

  const task = project.tasks.find(t => t.id === taskId);
  if (!task) return null;

  return <div>{/* Render task info */}</div>;
});

Arrays have some drawbacks since for example to find and update a specific item in the array we'd have to use array operations

Yes, definitely. If we are in control of the backend I would return data that can be used directly for a map instead, or transform the data a bit in the frontend if that is not possible.

projectData.tasks = projectData.tasks.reduce((acc, task) => {
  acc[task.id] = task;  
  return acc;
}, {});

[derekyau]

Thanks for your thoughts @EmilTholin :) - it struck me the other day that another situation that is awkward to model is when there are either:

1. many to many relationships (in the above example, for example if project and task were many to many)
2. other models that also contain references to the child object (for example project has many tasks but say... portfolio can also has many of the same tasks).

Alright the example is clearly starting to not make sense and break down :p but in real world practice, we could potentially get into these scenarios.

In these cases it makes nesting the object under the parent more awkward since in the ideal case we would want one "source of truth" for where the store of the data is (essentially normalizing the data) so that say for example tasks are always stored in tasksMap within the store. That way many projects can reference the same task object and the task will live in only one place on the tree. If we included it as an array on project I guess in this situation we would run into a duplicate identifier when we load the second project that references this?

Re: your suggestion of transforming the data - yeah that is what we have resorted to doing for now, but always wondered whether there was a better way

[EmilTholin]

Yes, if there is a many-to-many relationship having a standalone map and references is the way to go to normalize it properly, I agree.

[jamonholmgren]

One thing I'd like to recommend is mst-reference-pool. This allows you to normalize these instances easier and handle references and garbage collection. Read the README carefully -- this could be very useful for you.

https://github.com/infinitered/mst-reference-pool

Note that it does not support types.map yet (wouldn't be that hard to add), but you can actually get objects in a types.array by identifier just as fast as a types.map because MST has an identifier cache under the hood and exposes the resolveIdentifier helper function to get elements from any MST tree very quickly. It uses a map to do this, in fact.

import { resolveIdentifier } from "mobx-state-tree"

...

const task = resolveIdentifier(Task, store, taskId)

// could also be a more specific MST node than just the generic `store`
const task = resolveIdentifier(Task, project.tasks, taskId)

Between these tools, you might get closer to what you're looking to do.

Beyond that, I think mst-query is very cool, and potentially a good solution for this too, but I don't have a lot of experience with it yet.

[jamonholmgren]

@ConrabOpto might be able to give more ideas on this as well.

[derekyau]

awesome, thanks @jamonholmgren - I'll take a look!

[k-ode]

Here's roughly how you'd setup mst-query for this:

const RootStore = createRootStore({
    projectStore: types.optional(createModelStore({ projects: types.map(ProjectModel) }), {}),
    taskStore: types.optional(createModelStore({ tasks: types.map(TaskModel) }), {}),
});

// in the next version of mst-query this will be less verbose
const ProjectQuery = createQuery('ProjectQuery', {
    data: MstQueryRef(ProjectModel),
    request: types.model({ projectId: types.string })
}).actions((self) => ({
    run: flow(function* () {
        const next = yield* self.query(getProjectFn);
        const { result, error, data } = next();
    }),
}));

// And then in your component:
const { data, error, isLoading } = useQuery(ProjectQuery, {
   request: { projectId: id }
});

// Models can then be accessed from the store as normalized entities
const { projectStore } = useRootStore();
const project = projectStore.projects.get(projectId);

mst-query will automatically infer any level of nested identifier models and normalize these into the root store. If a model already exists it only updates the data actually fetched from the server, so partial updates work great.

[derekyau]

This does look super cool. I will have to give it a try in a full project one day !

The auto infer / normalization is great. Thanks for the snippet

[Anoninz]

There is a very useful normalize tool

Normalizr

you can try using it with mst