Merging, Derivation Groups, Activities, and Scheduling Goals

[pranav-super]

What's the deal with Merging?

The topic of this discussion post concerns a comprehensive look at plan branching and merging behavior as it pertains to derivation groups (specifically their association with plans), scheduling goals, and activities scheduled from those goals. The approach described in this post, following extensive discussion with @mattdailis, @JoelCourtney, @DanDelaney, and @JosephVolosin, is ultimately one of greater simplicity, with a focus on providing the user with maximal visibility and manual control over the operation to prevent any obfuscation.

Motivation

The initial problem at hand was that, to date, branching/merging behavior for the associations that plans had with derivation groups was yet undefined. This means that if Plan A has an association to Derivation Group A, it is not yet defined what associations a branch off of it should have. Similarly, if Plan A has an association to Derivation Group A, and a branch off of that plan, Branch B, had instead an association to a different Derivation Group B, no behavior was defined on merge.

Defining merge behavior for derivation group associations isn't particularly complicated. Since derivation groups provide a set of events separate from the plan, defining as a default that all derivation group associations across all branches should be maintained, with deference to the user to select which ones should be excluded, seems very appropriate. The result would be a
plan that has visibility of all groups that each of the individual branches would have had, unless any are manually excluded.

With that being said, there existed one logical issue with this simple approach.

Imagine we have in Plan A some Goal A that can maximally only schedule 2 activities based on an associated derivation group, and it schedules activities 2 and 3 based on the plan's association to some Derivation Group A. The branch off of this plan, Branch B, could also have this same goal and an association to a different Derivation Group B, and place activities 4 and 5.

When merging these two plans, our current merge would put all 4 activities in the merged result. But, if that goal was run on this plan without the presence of activities 2-5, it might only place activities 2 and 4 based on the associations to Derivation Group A and Derivation Group B. 3 and 5 have no clear justification for being present in the final plan. This means that the invariant of
the goal, which requires maximally 2 activities based on external events, has been violated, as now 4 such activities are present.

It's worth noting that this problem exists even without the introduction of derivation groups. Consider the same type of goal that places maximally 2 activities based on the value of a resource. If that resource that takes on completely different profiles across Plan A, Branch B, and the merged result, the very same issue would arise.

Discussion

The discussion following this started off by questioning the last statement in the above summary, which seems to imply that goals are invariants. If a goal is understood as an invariant, that would mean that it suggests some property about the plan that must always be true.

In this case, Goal A's invariant would be that only 2 external-events-based activities should be present. Merging behavior as currently defined clearly would not line up with this.

Several solutions were devised based on possible answers to the question of whether goals should be treated as invariants, but an answer was reached that ultimately rejected the premise.

The conclusion was reached that goals are not to be treated as invariants, but rather as idempotent and nothing more, as this is how they have been defined in previous discussions. This simply means that if a goal is run n times, the plan should look the same as it did after the first run.

Idempotency is clearly very different from invariance. There are a lot of possible solutions to our 'invariant problem' that could obey idempotency. But what is important here is that because we reject that question entirely, any merge solution could be valid, even if any invariants suggested by goals are disobeyed. Approaches that keep all activities or remove any activities scheduled by
goals are all valid here.

A Solution

The suggested solution, then, is to preserve all activities during the merge process. In addition to that, users would be provided with tools that can be used prior to the merge, such as the ability to clear out, select, or copy and paste all activities place by a given goal. By default, we transfer over all derivation group associations to plan branches and include all associations in the merged result, though we allow users to manually exclude any.

This solution clearly obeys the idempotency issue, as post merge, goals wouldn't place anything new and all subsequent runs would do the same. But the same is true if we included no activities after the merge, and that solution would also obey the invariant rule more closely. We chose specifically this approach for two reasons:

• understandability: a merge that very transparently preserves everything and leaves it to the user to manually clean up the plan before merge is very understandable and traceable, and seems preferable to a merge process that may somewhat opaquely remove activities. The case that we would like to avoid is a merge automatically resolving differences between plans, leaving a planner confused about the final result.
• invariance is a commitment: it also seems reasonable to not commit to an approach that clearly gains its edge by providing an invariance guarantee, as that might imply that invariance is a fundamental tenet of scheduling. That has not previously been true in the development of the feature and supporting that could require a fair amount of redesign, which we cannot easily justify.

Derivation Groups

Furthermore, because we no longer consider invariance, our simple solution to derivation group association merging (an additive approach that by default includes all associations in the final result, but allows users to optionally exclude any derivation group associations) also seems valid and appropriate here. Given that we can allow users to exclude associations from the final, merged
result, we also return as much control to the user from the merge as possible.

Scheduling Specifications

There is one additional aspect of this solution not yet mentioned, which concerns scheduling goals themselves. Currently, scheduling specifications are not a part of the merge process, but since we are now moving to include plan peripheries (in this case, plan associations to derivation groups) in the merge process, it seems reasonable to include the scheduling specification in this process too. However, the verdict is still out on exactly what this would look like.

Summary

We went into this discussion thinking we would develop a 'clever' merge that handles the problem described above very explicitly, but left with a simple and rudimentary merge that handles very little, instead pushing the onus of control to the planner. Our commitment to simplicity here implies another philosophy in our approach to merging:

We would defer to solutions providing greater manual control in the merge process, as that provides the greatest amount of transparency to the user.

Application

To make this concrete, here is what this solution means in terms of the original problem. We have our same two branches. The activities 2-5, in their respective plans, can be deleted up front, before the merge. If none is performed, as is the case in the diagram, all activities persist into the result. There are no merge semantics defined yet for scheduling specifications, so the
assumption is that each of these branches had the goal added manually. Finally, all derivation group associations by default persist into the result, though again, the user could manually exclude either from the final result.

Tangible Actions

We proposed introducing several things to the merge:

• additive inclusion of derivation groups, with user control over exclusion
• tools to clean up activities prior to merge, relating to the scheduling goal that placed them
• merging of scheduling specifications

We only want to focus on one of these today, and that's the derivation group functionality. This is clearly specified as part of the SSMO team's current batch of work, while the rest is definitely out of scope for this immediate batch. The other tasks (the clean-up tools, the merging of scheduling specifications) will not be implemented now, but they can and should be deliberated in this discussion now!

Open Questions

Following all of this, there remains one final open question that wasn't resolved in any prior discussion, and that's the question of whether there is a use case for non-idempotent goals.