-
Notifications
You must be signed in to change notification settings - Fork 549
Write support for blob handles with pending payload #24458
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 13 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
34eac3e
Port core-interfaces portion
ChumpChief f715d3a
Port runtime-utils portion
ChumpChief 6f7beca
Port BlobManager changes
ChumpChief 6510b24
Fixup tests
ChumpChief 5e8872d
Add end-to-end-tests
ChumpChief 854d288
Merge branch 'main' into PendingPayloadWrite
ChumpChief aff9689
Update packages/runtime/container-runtime/src/test/blobManager.spec.ts
ChumpChief 9770df6
Comment clarification
ChumpChief 142aebb
Switch handles over to new event emitter
ChumpChief b280f92
Convert BlobManager to new event emitter
ChumpChief dff0849
Defer instantiation of event emitter
ChumpChief b100d89
Merge remote-tracking branch 'upstream/main' into PendingPayloadWrite
ChumpChief 29a334c
Fix tests
ChumpChief 9d24881
Merge remote-tracking branch 'upstream/main' into PendingPayloadWrite
ChumpChief a93261f
Merge remote-tracking branch 'upstream/main' into PendingPayloadWrite
ChumpChief 516f2d7
Split local payload state out from remote payload state
ChumpChief 0c78871
Merge remote-tracking branch 'upstream/main' into PendingPayloadWrite
ChumpChief 7021f99
Settle on final names where possible
ChumpChief File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'm trying to think through the usage here. what are the supported state transitions? how do users reason about them? What is the experience for local client vs remote clients? are they symmetrical? should they be, or do client need custom code for local and remote.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think this would be easier to review if the implementation were split from api, as the api will likely have significantly more comments than the internals. getting the internals in would also unblock re-enabling testing with staging mode.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think we should remove 'local' at this layer, and only have
"persisted" | "pending" | "failed"
, anything related to local should be moved to a blob handle type such that local information is only available on the strongly typed local object returned from the call to upload.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For visibility - had a chat with Tony on the side, we have a bit better understanding of each other now.
Re: the question above, the states are not symmetrical for local/remote nor should they be, since the states are wholly different between the two (both in what can be detected and what reaction should be taken), with the exception of "shared" state which is where they reach parity. I've updated the documentation a bit to clarify as well.
In our conversation Tony suggested splitting the interface instead, such that local/remote have different types. I'm thinking more on this option or if there's any preferable alternative here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also just realized that you can use mermaid diagrams in github comments, so including the current state diagram here too:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i understand the remote case a bit better now, but i'm still not sure the api as is ergonomic for the remote case
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re: splitting the interface, I'm struggling to find a case where it improves the customer experience.
When creating a BlobHandle via uploadBlob(), the customer immediately gets the handle back in local state (and it will not exit this state until they elect to attach the handle, giving them plenty of time to register listeners). They need the full interface except "pending" state, since they likely want to observe all of the possible transitions and states through the upload process. It's true we have type knowledge here that the handle will never enter "pending" state, but at the same time the customer doesn't need to do anything special to avoid the "pending" state; it doesn't get in their way.
To acquire a RemoteFluidObjectHandle, the customer is likely pulling it from a location where it might be mixed in with BlobHandles in normal app operation. For example, if I get an IFluidHandle out of a SharedMap it is probably dangerous to assume whether that IFluidHandle is actually a BlobHandle or RemoteFluidObjectHandle. The customer can't ignore that both are possibilities with whatever logic they're applying. However, I don't think this is a burden for them. If we assume they are doing their local failure watching at the point of calling uploadBlob as suggested above, then this portion is probably just looking to run the remote cleanup heuristic.
These are obviously just examples, but I don't see where having a narrower interface necessarily simplifies. If there are other better examples would be glad to take a look though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I just realized I had missed your tsplayground - responses inline here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
based on the responses i feel pretty strongly we should split the local and remote experiences. its very confusing that only some states and events only apply local or remote. I can see customer writing code where they wait for failure on remote handles which is impossible, but nothing in the interface or code will prevent it. I'm of the opinion that when possible, it should be clear from the code itself how to use an api, and it seems quite achievable here with little change.
for the local api, i would still push for a combined event which covers both shared and failed states to avoid the need to write code that crosses event handles, as that type of code gets very complex, especially when you take in further requirements like tracking close/dispose to give up.
I'm also curious what @yann-achard-MS and @znewton think about this, as they will helping partners to onboard these apis and their extensions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would also consider punting on the remote half of the api, since its not hooked up yet, so no need to commit to that api yet anyway.