MmapStorage

class paddle. MmapStorage ( filename: str, nbytes: int ) [source]

This class will use mmap to load a file.

Parameters

• filename (str) – the name of .safetensors file.

• nbytes (int) – number of bytes to map into memory.

Examples

>>> import paddle
>>> shape = [4,5]
>>> dtype = paddle.float32
>>> a = paddle.arange(4*5).reshape(shape).astype(dtype)
>>> a.numpy().tofile("test.pp")
>>> size = a.size * a.element_size()
>>> t = paddle.MmapStorage("test.pp", size)
>>> t.get_slice(dtype = dtype, start = 0, stop = a.size).reshape(shape)
Tensor(shape=[4, 5], dtype=float32, place=Place(cpu), stop_gradient=True,
       [[0. , 1. , 2. , 3. , 4. ],
        [5. , 6. , 7. , 8. , 9. ],
        [10., 11., 12., 13., 14.],
        [15., 16., 17., 18., 19.]])

get_slice ( dtype: DTypeLike | None = 'uint8', start: int = 0, stop: int = -1, step: int = 1 ) → paddle.Tensor

get_slice¶

Slice the tensor from the mmapped file. :param dtype: The data type of the output tensor. Default: “uint8”. :type dtype: DTypeLike | None :param start: The start index of the slice. Default: 0. :type start: int :param stop: The end index of the slice. Default: -1. :type stop: int :param step: The step size of the slice. Default: 1. :type step: int

Returns

The sliced tensor.

Return type

Tensor