[GraphBolt][CUDA] Overlap feature fetcher

[mfbalin]

Description

UVA feature fetching is overlapped with other operations. The picture below shows that IndexSelectMultiKernelAligned executes simultaneously as the GNN model forward/backward operations. We may need to tune the 6144 default value in a later PR for optimal performance (Might also want to implement autotuning logic). If it is too high, then too much unnecessary GPU resources will be used for the UVA copy, if it is too little, we can't fully utilize the PCI-e connection. However, it is safer for overall performance to set it to a higher value to ensure the PCI-e can be fully utilized in all different systems.

NOTE: The memory leak was caused by multiple CUDA streams being created when we created multiple dataloaders. Making uva_stream a global variable resolved the issue.

Checklist

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [$CATEGORY] (such as [NN], [Model], [Doc], [Feature]])
• I've leverage the tools to beautify the python and c++ code.
• The PR is complete and small, read the Google eng practice (CL equals to PR) to understand more about small PR. In DGL, we consider PRs with less than 200 lines of core code change are small (example, test and documentation could be exempted).
• All changes have test coverage
• Code is well-documented
• To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
• Related issue is referred in this PR
• If the PR is for a new model/paper, I've updated the example index here.

Changes

[dgl-bot]

To trigger regression tests:

• @dgl-bot run [instance-type] [which tests] [compare-with-branch];
  For example: @dgl-bot run g4dn.4xlarge all dmlc/master or @dgl-bot run c5.9xlarge kernel,api dmlc/master

[dgl-bot]

Commit ID: bdf9d81

Build ID: 1

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: bdf9d81

Build ID: 2

Status: ❌ CI test failed in Stage [Lint Check].

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 0cf33a5

Build ID: 3

Status: ❌ CI test failed in Stage [Lint Check].

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: cd42c36

Build ID: 4

Status: ❌ CI test failed in Stage [Lint Check].

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 8d68ace

Build ID: 5

Status: ❌ CI test failed in Stage [Lint Check].

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 3e81d6e

Build ID: 6

Status: ❌ CI test failed in Stage [Lint Check].

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 9a205e3

Build ID: 7

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 239f1c3

Build ID: 8

Status: ❌ CI test failed in Stage [Torch GPU Unit test].

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: e600b19a7f88285d3018b9618bfac169569aeb66

Build ID: 9

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 80a8a80

Build ID: 10

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 0bd41de

Build ID: 11

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 2c8863d2b7439a53854f0072cd93ce5f365e3bc0

Build ID: 12

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 51e494f

Build ID: 27

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 7aecffc

Build ID: 28

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: ba52ef2

Build ID: 29

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 16151dc

Build ID: 30

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 2e00869

Build ID: 31

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 5d30466

Build ID: 32

Status: ❌ CI test failed in Stage [Torch GPU Unit test].

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: b552ddb

Build ID: 33

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 6c7305c

Build ID: 34

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 0d6a9ca

Build ID: 35

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

[dgl-bot]

Commit ID: 5e62667

Build ID: 36

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link