Generalized Assets as Entities

[ElliottjPierce]

For assets as entities (#11266), we need an asset to live in an entity and be identified by that entity. However, for nested assets, this gets complicated. Say an asset type has sub-assets that it references by the asset handle. If that handle/id becomes an entity, how does the loader reserve the Entity without &Entities. This is possible (See #18380), but it's costly.

Clarifications

To be perfectly clear, I'm going to define some ideas up front:

• Entity: A collection of components.
• Component: Any data.
• Asset Entity: An entity that holds asset components.
• Asset Component: A component that is (not holds) asset data.
• Asset: Any data that drives/informs systems rather than participating in them. For example on the player entity, health_left is a normal component, health_stats is a link to a asset entity with asset components like max_health and regeneration, etc.
• Asset Location: The file or other path within an asset source (common sources: directory, http, etc.).
• Asset Loader: The systems that takes an asset location in a source and converts it into a hierarchy of loaded asset entities (including sub-assets).
• Asset Saver: Exactly the inverse of an asset loader.

Maybe you disagreed with how I called max_health an asset earlier. That's my point. With assets as entities, there's a very blurry line between serializable components and assets. So lets split them up entirely.

Proposal: Loadable Components

Let's generalize asset loading and saving to work with any serializable component.

pub struct SerializingLocation<T: Component> {
    marker: PhantomData<T>,
    path: AssetPath,
    // If `T` hasn't changes since it was added, we can skip saving it.
    added: Tick,
}

We can stick this on any entity with T, and just like that, we know where to save and load it. To load or save one of these entity's components, we can use bevy's existing saving and loading system almost verbatim.

This does two really exciting things:

First, users can make their own save system super easily.

world.add_observer(|
    to_save: Trigger<OnRemove, T>, 
    paths: Query<(&SerializingLocation<T>, &T)>,  
    // Any other components we want to save here. Ex sub-asset queries.
    mut saver: ResMut<SaveServer>
| {
    if let Some((path, value)) = paths.get(to_save.entity()) {
        saver.save(path, value);
    }
});

I'm skipping some details, but you get the point. Bevy's existing save/load infrastructure goes into ResMut<SaveServer>.

Second, assets, including nested assets, are just scenes. Nothing more, nothing less. That means they can soak up all the benefits of bsn. (Though they don't have to. Custom asset loaders are still trivially possible. But if a component is serializable, it can be in a bsn asset loader, so that's a good default. Plus, it means no more having to do the boiler plate of creating another asset loader for your new asset type.)

I'm glossing over some fairly obvious details here, for example, if T is removed but not replaced, we should get rid of the SerializingLocation, etc.

If you're wondering how Arc'd assets (#15813) fits in here, I think that's tangential to this. It's very much compatible here, but with this generalization, not all serializable components should be put in an arc.

Proposal: Garbage collected components

One of the big features of bevy's asset system is that when one is no longer needed, it is freed automatically. We can't afford to drop this feature. (Just look at the ecosystem of Unity plugins that try to figure out which assets can be unloaded.)

Instead of a custom Handle system for exclusively assets, let's generalize this too.

pub struct EntityComponentHandle {
    arc: Arc<()>,
    entity: Entity,
}

pub struct GarbageCollected<T: Component> {
    base: Arc<()>,
    //  We can give a grace period before garbage collection to allow systems to respond to the new entity and get a handle
    extra_life: u32,
}

Now, we simply periodically check the garbage collected components for if their base is shared, and if not, we remove the component. If there are no other GarbageCollected components, we can despawn the entity.

Putting it all together

With all of this implemented, we can do the following:

• Load an asset by its path. We can create the root entity now, and insert it's loaded components and children later via scenes.
• Create custom asset loaders besides bsn. Just load the asset like we do now, throw it in a scene, and map entities when we spawn the scene into the world.
• Load the same path twice (The second load would just return the Entity of the first load.)
• Add a GarbageCollected to the entity for automated release. (If a user wants to do this manually, they don't have to add it, and we can automate adding it otherwise.)
• Get handles to that GarbageCollected.
• Clone the serialized component to a different entity to use the asset as data separately from its source. (Very useful and Godot supports this). We can even add GarbageCollected here too (or anywhere).
• Modify the original entity's component or it's "sub-assets"/children (We'd probably use a new relationship here.)
• Drop all handles to the original entity's component. The GarbageCollected will remove T, and the SerializingLocation can be used with an observer to start saving the modified data.
• Add a SerializingLocation to the other entity (the one we cloned the component onto earlier).
• Let that other entity be garbage collected.
• The SerializingLocation and associated observer will save the new component to it's new location.

Now, the asset system is more than just images and meshes. We can use it for save data, settings/player preferences, player/item/etc stats, and so much more. Plus we can do things like, when a Mesh is added to an entity, we use hooks to calculate its AABB on the fly. Or, for unique, procedurally generated meshes, we don't need an asset entity. Just throw the Mesh, Mesh3d, MyMaterial, (and the rest) on the same entity.

Extension Assets

One final possibility is extension assets.

Instead of loading as their own scene, these load and prepare a closure that modifies an existing scene. (Or something like that). For example, an Item may be an asset, and Price may be an extension asset on Item. We can load a Price which triggers loading the Item associated with it. When they're both done, they would be merged. In this case, the result would be an entity with Item, Price, SerializingLocation<Item>, SerializingLocation<Price>.

This is invaluable for modding, but not much use otherwise. (Why not just include Price in the item's bsn file?)

Conclusion

This design offers extreme ergonomics that I find really appealing. It also simplifies assets as entities and breaks them down into very approachable chunks for implementation. What makes a traditional asset special is that it can be saved, loaded, and garbage collected. This proposal splits those up into their own features allowing the ecs to benefit from these asset patterns and allowing the asset system to benefit from the ecs.

It also doesn't hurt that this solves a lot of design questions about how to use an Entity has an asset handle without &Entities.

[andriyDev]

Ok I've thought about this for a bit and I've been teetering back and forth.

First, I think this idea is interesting. It trades off all the complexities of remote entity allocation with entity mapping. It's neat! Even more, there's no need to depend on bsn here. We could just do this with Scene the way it's implemented today more or less.

However there are two problems. One small and one big. The small issue is that it means we can no longer load assets from a thread. Since this solution depends on the fact that we can entity-map later, we can't have adhoc asset loading (e.g., a worker thread that is queuing up entities to be spawned using CommandQueue). Probably not too big a deal, and could be worth giving up.

The second and bigger problem is that we can't entity-map through Arcs. If your asset internally holds Arcs which themselves hold asset handles, those asset handles cannot be entity-mapped later. MeshletMesh is an example of an asset that stores Arcs, and it also seems possible that you might even want your asset to be a wrapper around Arc<RwLock> or something (even without #15813). This seems like a very scary footgun for users.

Some extra thoughts: I have no qualms about GarbageCollected, though maybe calling it RefCounted would be more accurate. Maybe someone can use it for something else.

SerializingLocation on the other hand is less convincing to me. Save files are unlikely to be a single component. I'd expect you to have a system that collects all sorts of components, bundle it into an asset, and then write that with one save call. There's no reason to keep that asset in memory after saving, so making it an entity is a little weird to me (as in, you need to spawn an entity for the asset, save it, then despawn it, or otherwise let it be RC-ed). For loaded assets, having a component with the path of the asset is desirable, but it's not clear to me it would be useful outside of that.

[ElliottjPierce]

Ok I've thought about this for a bit and I've been teetering back and forth.

First, I think this idea is interesting. It trades off all the complexities of remote entity allocation with entity mapping. It's neat! Even more, there's no need to depend on bsn here. We could just do this with Scene the way it's implemented today more or less.

Absolutely! The bsn mention is just me thinking ahead; this doesn't depend on bsn.

However there are two problems. One small and one big. The small issue is that it means we can no longer load assets from a thread. Since this solution depends on the fact that we can entity-map later, we can't have adhoc asset loading (e.g., a worker thread that is queuing up entities to be spawned using CommandQueue). Probably not too big a deal, and could be worth giving up.

The second and bigger problem is that we can't entity-map through Arcs. If your asset internally holds Arcs which themselves hold asset handles, those asset handles cannot be entity-mapped later. MeshletMesh is an example of an asset that stores Arcs, and it also seems possible that you might even want your asset to be a wrapper around Arc<RwLock> or something (even without #15813). This seems like a very scary footgun for users.

Ah ok. So we still need remote entity reservation. Thanks for such a clear explanation. That makes a ton of sense.

Some extra thoughts: I have no qualms about GarbageCollected, though maybe calling it RefCounted would be more accurate. Maybe someone can use it for something else.

That's a better name!

SerializingLocation on the other hand is less convincing to me. Save files are unlikely to be a single component. I'd expect you to have a system that collects all sorts of components, bundle it into an asset, and then write that with one save call.

I wrote my post when I was kinda tired. Let me try to explain what I mean here because I don't think we're on the same page.

Say we have an item asset in a bsn scene (end game):

(
    Item {
        name: "Wooden Pickaxe",
        kind: ItemKind::Tool,
    },
    InventoryDisplay {
        image: "some/path/to/image.png", // gets turned into an Image handle
        tool_tip: "breaks ston",
    },
) [
    // Child 1
    (
        ItemAction(OnLeftClickHold),
        BreakBlock {
             max_hardness: 2,
             durration: 0.2,
        },
        Animation("path/to/animation/clip"),
    ),

    // Child 2
    (
        ItemAction(OnLeftClick)
        Attack {
             damage: 4,
             range: 5,
        }
        Animation("path/to/animation/clip")
    ),
]

Here's what happens when we say aset_server.load::<Item>("file/path.item.bsn"): First, it gives us the root entity. Then it loads the bsn file, deserializing the asset types via loaders (which can do things like pull out the animation clip and image handles into their own loading process, etc.). In the end we have an entity with SerializingLocation<Item>, Item, InventoryDisplay, etc, and the children in the scene become child entities under the root. When it is no longer used, it is "garbage collected". If it's changed, this starts a save operation. (It wouldn't change in this case, but it would for player data or something). Anyway, when it's despawned, an observer can, at the user's opt-in choice, kick off a save process with the SerializingLocation<Item> file path. This won't just save Item but will eat through it's children and siblings, packaging everything up into a neat Scene. That scene is then saved to disk. SerializingLocation<T> is really only generic over T so we can know which asset saver/hot reloader to use. For example, the Image asset loader would not use bsn, but we can make a bsn loader that works with types that implement Serde serialization. This also means we could have loaded the asset as a InventoryDisplay instead. I don't really see the use in that, but it's a thing.

There's no reason to keep that asset in memory after saving, so making it an entity is a little weird to me (as in, you need to spawn an entity for the asset, save it, then despawn it, or otherwise let it be RC-ed).

To save an asset you don't need to spawn an entity. We can keep 99% of our current save infrastructure. It's just that if a entity is being despawned with changes not reflected on the file it was loaded from, we can save it here automatically. This just makes it easier for users. Ex: if you modify your player settings and close the window, we can save things with a simple query for SerializingLocation<PlayerSettings>, which would get it's children "sensitivity", "jump_key", "volume", etc. Or, if the player puts an item in a chest and walks far away, unloading the chest entity, as it despawns, we can save it automatically. This is basically cache coherency where the ecs acts as a cache for the file system. That's one way of looking at it anyway.

For loaded assets, having a component with the path of the asset is desirable, but it's not clear to me it would be useful outside of that.

I agree that that's a main use, but think about my unloading chest example above. What if the chest is brand new, meaning the inventory "asset" is not from disk? We can just slap a SerializingLocation<Inventory> on it, and let the save system tackle it automatically.

---

Hopefully that clarifies my thoughts here. Even though I agree now that remote reservation is a must, I still think these abstractions are really worth it. But that's just my opinion.

[andriyDev]

SerializingLocation is really only generic over T so we can know which asset saver/hot reloader to use.

Ahhh ok this is what I was missing. Thank you! That is more convincing.

To save an asset you don't need to spawn an entity.

Ok I think I understand a little more. I agree we can automate asset saving in that case. Though I think at that point we should just make the component AutoSaveAsset<L: AssetLoader>, and this should be opt on. Asset saving today is an academic debate though, since we kinda can't save assets without #15813 (unless we require all assets be clone, which is also not desirable). The bigger issue is which components we have to extract off the entity...

However, I personally don't think we should "break up" assets. That is, I think an asset's data should remain a single component rather than being split apart. It makes managing assets much easier (e.g., the asset is loaded if it has the Asset component), keeps things simple, and asset saving can be much less ambiguous (how much of an entity do you need to take to save it?). To be clear, the asset metadata will still be separate components, but the asset data would be one component. But that's just my take.

Thank you for your clarification! This was helpful!

[ElliottjPierce]

Good name ideas. What it does > what it is.

The bigger issue is which components we have to extract off the entity...

100% agree here. My answer "magic"! That is to say, I have no clue how we'd do this. Maybe the asset type specifies which sibling and child components to look for? IDK

However, I personally don't think we should "break up" assets. That is, I think an asset's data should remain a single component rather than being split apart.

I can kinda see what you mean here. But only kinda. My problem with this is that it makes it harder to use the ecs to get data easily. Take the player settings for example. If all settings are in one massive asset, it's going to get messy really fast. If each setting is in its own file, that gets messy too. I'd rather have each setting be a single asset/component but all be in the same file. When it's loaded, all the settings get their own entity under the Settings root entity. That means, for example, you can build the settings menu by querying for each of the settings types. Ex: Query<(&SettingName, &KeyBindingSetting)>. That's much nicer than having to set up the ui with settings.jump, settings.forward, settings.left, settings.right, settings.back, settings.attack, .... This example's a little contrived, but you can see where I'm coming from.

Obviously, with something like image assets, this is completely ridiculous. 🤓 ☝️ "One component per pixel please!" Lol.

Asset saving today is an academic debate though, since we kinda can't save assets without #15813 (unless we require all assets be clone, which is also not desirable).

I agree here, but that's kinda tangential to this design. If the entity is being garbage collected anyway, we shouldn't need to clone. Just remove the component and kick of the saving process. No clone or arc needed. (In theory). We'd need to add support for this in the ecs, but that shouldn't be too hard.

---

With needing remote reservation anyway, I don't see this design as pressing. But I do still really like it. You know a lot more about the asset system than I do, so I'll leave this up to you and others in the long run. I just wanted to stress the ergonomics of being able to split asset data across components and having them saved as they are no longer needed.

Regardless of how we organize this (1 or many components per asset), it sounds like assets as entities would still need GarbageCollected or RefCounted or whatever to work. Is that accurate? If so, that can be something I work on while we sort out other details.

[andriyDev]

Many have expressed a desire to have a bunch of components make up the asset, so you're not alone! This is worth investigating, but I'd like assets-as-entities v2 (if you count Cart's original as v1) to be a fairly minimal port.

I personally think GarbageCollected/RefCounted could be helpful. I'm also tinking with assets-as-entities on the side (using a fake for remote reservation), and implementing a nice RC pattern for entities can be a little annoying (e.g., having a weak on the entity so you can fetch its handle is useful too). Working that out would be nice! The only sad part is I don't know if there will be a motivating example that isn't assets-as-entities...

[ElliottjPierce]

That's all fair. I'm happy to wait on this for a while. I have plenty of ECS stuff to keep me busy. But I do think it's an ergonomic end game. I expect the idea will evolve as the first assets as entities idea rolls out though. And I agree that jumping strait to this is not ideal. (You've convinced me). Let's get a working assets-as-entities implementation first, and then try to improve it. (Maybe with this as a reference.)

But the principal of re-exposing the robust asset saving and loading system and re-using it for saving game state is really appealing to me. Pretty much every engine I know of has a separate asset system and lets users make their own save system. But having just the one asset system and making it robust and reusable is more appealing to me. (Users could always make a custom one anyway.)