NVIDIA cuObject lets applications read and write S3 compatible object storage directly into GPU memory over RDMA. The concept has strong efficiency promise. Here is my overview of architecture and some gaps that I see.
1. The concept
a. Normal object storage access moves data from storage to the CPU and then to the GPU, even though the CPU does nothing with that data along the way.
b. cuObject splits an S3 request into a control plane that stays ordinary HTTP and a data plane that moves the actual bytes by RDMA between the storage node and GPU or host memory.
c. The payload travels over RDMA rather than in the HTTP body, and only a small metadata token travels with the HTTP request.
2. There are two components
a. cuObjClient runs inside the application or data loader and issues the GET or PUT request through a normal S3 SDK.
b. cuObjServer runs inside the object storage vendor's own server software rather than as a separate proxy in front of it. Meaning it requires deep integration.
c. The client adds a header called x-amz-rdma-token to the S3 request, and the server replies with x-amz-rdma-reply when it can offload the transfer to RDMA.
d. The transport used is Dynamic Connection over InfiniBand or RoCEv2, which avoids pre-establishing a connection for every client and server pair, so it scales to a large fabric.
e. A client that does not send the RDMA token is served over ordinary HTTP, which is what allows a vendor to add cuObject support without breaking existing S3 clients.
3. Dependencies
a. It requires RDMA capable networking between client and server, such as InfiniBand or RoCEv2, since it does not work over plain TCP or standard Ethernet.
b. It requires GPUDirect RDMA support on the GPU and the NIC.
c. It requires the object storage vendor to implement cuObjServer inside their own stack.
d. It requires client side integration through an S3 SDK that understands the RDMA token.
4. Some gaps that I see
a. The maximum size is 1 GiB per single GET or PUT call, with up to 10 scatter gather entries per operation, so a large checkpoint shard has to move through UPLOAD_PART or multiple RANGE_GET calls.
b. Vendor support is still limited. I have not seen public cloud object storage such as S3 itself on the list.
c. Not sure how it would work with custom implementations like EFA.