fix the error of illegal memory access caused by cub #10
+1
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix an issue caused by cub.
From my own test, the problem happened when you call cub related functions, such as InclusiveSum Here https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L281.
The problem is that the copy process does not go all the way. From my test, the copy somehow stopped at around 80K. After I tried to bypass that function by using std::inclusive_scan. However, illegal memory access happened somewhere in the backward pass.
Afterward, I found that a similar issue kind of happened in custom-built Pytorch functions (NVIDIA/thrust#1341) and found the solution here pytorch/pytorch#52663 (comment).
After the test, I used the modified code to train Gaussian Splatting on the Blender Lego dataset, and it worked like a charm.
Related issues:
Potentially related issue: