Add indirect pointer to barrier support in hl.signal & hl.wait (as_ptrs) #261
Conversation
joydddd
force-pushed
the
joydddd/stack/13
branch
from
dd98a26 to
4c4b3ae
Compare
facebook-github-bot
added
the
CLA Signed
joydddd
force-pushed
the
joydddd/stack/13
branch
2 times, most recently
from
4eb10c4 to
dac9b8f
Compare
joydddd
force-pushed
the
joydddd/stack/13
branch
2 times, most recently
from
053923a to
1a74563
Compare
joydddd
force-pushed
the
joydddd/stack/15
branch
from
bc6ea11 to
5348847
Compare
joydddd
force-pushed
the
joydddd/stack/13
branch
from
1a74563 to
a82b355
Compare
joydddd
force-pushed
the
joydddd/stack/13
branch
from
a82b355 to
9f46f56
Compare
| ) | ||
| N = signal_pad_ptrs.size(0) | ||
| for i in hl.grid(pad_shape): | ||
| offset = i * 4 # number of btypes of each pointer (torch.int32) |
There was a problem hiding this comment.
Calculating offset to pointers is kinda ugly here and it breaks the no-index calculation / pointer-handling abstraction. The tensor metadata (i.e. size, stride etc.) is lost when passed in as Tensor of data_ptrs, therefore offset calculation need to be explicit here for indirect barriers.
joydddd
force-pushed
the
joydddd/stack/13
branch
from
9f46f56 to
e646848
Compare
| signal_pad_list = [ | ||
| torch.ones(4, device=DEVICE, dtype=torch.int32) for _ in range(4) | ||
| ] | ||
| signal_pad_ptrs = torch.as_tensor( | ||
| [p.data_ptr() for p in signal_pad_list], device=DEVICE, dtype=torch.uint64 | ||
| ) |
There was a problem hiding this comment.
What is the motivation for multiple signal pads here?
Could we have a multi-dimensional pad instead like:
signal_pad_list = torch.ones([4, 4], device=DEVICE, dtype=torch.int32)
then make pass in an index like (multicast_tile, i)?
There was a problem hiding this comment.
The motivation is to implementation cross rank sync i.e. each barrier lives on a difference card, and we signal our peers and then wait for signal from our peers to make sure all cards have reached this kernel.
There was a problem hiding this comment.
See test/test_distributed in #245
There was a problem hiding this comment.
Yeah, I guess the tensor of pointers is required in this case -- let's figure out a higher level abstraction to make this cleaner.
stack-info: PR: #261, branch: joydddd/stack/13
joydddd
force-pushed
the
joydddd/stack/13
branch
from
e646848 to
ab85dfe
Compare
Stacked PRs:
Add indirect pointer to barrier support in hl.signal & hl.wait (as_ptrs)