Skip to content

[Common] Multicast Fixes#2847

Draft
phu0ngng wants to merge 6 commits intoNVIDIA:mainfrom
phu0ngng:multicast
Draft

[Common] Multicast Fixes#2847
phu0ngng wants to merge 6 commits intoNVIDIA:mainfrom
phu0ngng:multicast

Conversation

@phu0ngng
Copy link
Copy Markdown
Collaborator

@phu0ngng phu0ngng commented Apr 8, 2026

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

phu0ngng and others added 5 commits April 6, 2026 18:47
@phu0ngng
fix uninit comm->mac_max_size usage
cf5cfa4 
Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
Fix cuMulticastAddDevice parameter type
1dad574 
Pass CUdevice instead of incorrectly casting device index to CUdeviceptr.
Use cuDeviceGet to properly obtain the device handle before adding it
to the multicast team.

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
added sm_arch check to supports_multicast()
0d1cc81 
Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
Fix peer-to-peer access check to verify all GPU pairs
3c7d728 
Previously the check would succeed if ANY peer was accessible, but later
code attempts IPC operations with ALL peers. This caused illegal memory
access errors on systems where some GPU pairs lack peer access.

Now verify that ALL required GPU pairs have peer-to-peer access before
attempting IPC memory handle operations.

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
3ed8d18 
for more information, see https://pre-commit.ci
@phu0ngng
Copy link
Copy Markdown
Collaborator Author

phu0ngng commented Apr 8, 2026

Pipeline 47971845

@phu0ngng
Merge branch 'main' into multicast
a5014fe 
Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
Copy link
Copy Markdown
Collaborator Author

phu0ngng commented Apr 9, 2026

/te-ci JAX L1

@phu0ngng
Copy link
Copy Markdown
Collaborator Author

phu0ngng commented Apr 9, 2026

Pipeline 48162981 with the nightly image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@phu0ngng