3.1 Release Note

New features

• New APIs: paddle.msort, paddle.ravel, paddle.nn.functional.dropout1d, paddle.Tensor.type_as, paddle.Tensor.requires_grad, paddle.view_as_complex, paddle.view_as_real, paddle.nn.Parameter, paddle.broadcast_shapes, paddle.range, paddle.as_tensor, paddle.scatter_reduce/scatter_reduce_, paddle.scatter_add, paddle.tensor, paddle.softmax, paddle.Tensor.softmax, paddle.rand_like, paddle.is_autocast_enabled, paddle.get_autocast_gpu_dtype, paddle.Tensor.repeat, paddle.permute. #74421,#74439,#74444,#74454,#74459,[#74491]( https://github.com/PaddlePaddle/Paddle/pull/74491 ）[# 74466]（ https://github.com/PaddlePaddle/Paddle/pull/74466 ),#74438,#74594,#74542,#74694,#74564,#74540,#74586,#74651,#74807,#74632,#74834,#74952,#74772,#74441,#74561,#74525

• Added a series of APIs under paddle.compat.* to support common usage in the industry and facilitate code migration, including paddle.compat.median, paddle.compat.nanmedian, paddle.compat.softmax, paddle.compat.sort, paddle.compat.split, paddle.compat.min/max, and paddle.compat.Unfold. #74865, #74874

• Added a series of initialization APIs to support commonly used parameter initialization methods in the industry, including paddle.nn.init.kaiming_uniform_, paddle.nn.init.xavier_uniform_, paddle.nn.init.uniform_, paddle.nn.init.kaiming_normal_, paddle.nn.init.xavier_normal_, paddle.nn.init.normal_, paddle.nn.init.calculate_gain, paddle.nn.init.constant_, paddle.nn.init.dirac_, paddle.nn.init.eye_, paddle.nn.init.ones_, paddle.nn.init.orthogonal_, paddle.nn.init.trunc_normal_, and paddle.nn.init.zeros_. #74478

• Added usage of parameter aliases in API, allowing for more flexible input options such as x or input. This includes functions like paddle.maximum, paddle.minimum, paddle.sqrt, paddle.topk, paddle.polar, paddle.stack, paddle.cos, paddle.floor, paddle.log, paddle.pow, paddle.rsqrt, paddle.sign, paddle.sin, paddle.multiply, and paddle.where. #74683, #74795, #74887, #74592

• paddle.Tensor now supports multiple initialization methods, enabling flexible Tensor creation. #74619, #75022, #75065

• The API has added some proprietary parameters to enhance existing functions, including paddle.nn.functional.gelu, paddle.divide/div/div_, paddle.add, paddle.Tensor.copy_, paddle.norm, paddle.linalg.norm, paddle.nn.functional.silu, and paddle.repeat_interleave. #74485, #74562, #74420, #74768, #74855, #74903, #74788, #74631, #74947

• The API has added some common parameters: out, device, dtype, requires_grad, pin_memory, and bias, enhancing the existing functionality. These include paddle.zeros, paddle.zeros_like, paddle.ones, paddle.ones_like, paddle.arange, paddle.eye, paddle.empty, paddle.empty_like, paddle.full, paddle.full_like, paddle.randn, paddle.Tensor.new_full, paddle.Tensor.new_empty, paddle.Tensor.new_ones, paddle.Tensor.new_zeros, paddle.tril/triu, paddle.bmm, paddle.nn.Conv1D/Conv2D/Conv3D/Embedding, paddle.diff, paddle.cumsum, paddle.var, paddle.multinomial, and paddle.mean. #74477,#74526,#74711,#74582,#74624,#74849,#74612,#74875,#74641,#74949,#74918,#74914,#74934,#74920,#74955,#74226,#74946

• Added aliases to APIs to support more calling methods. These include paddle.Tensor.mul_/mul, paddle.autograd.Function, paddle.argwhere, paddle.cat, paddle.clamp, paddle.ger, paddle.take_along_dim, paddle.linalg.matmul, paddle.special.logsumexp, paddle.concatenate, paddle.eq/gt, paddle.Tensor.take_along_dim, and paddle.nn.Conv1d/Conv2d/Conv3d, etc. #74493, #74569, #74870

Bug fixes

• Fixed the precision issue of paddle.nanmedian in PaddlePaddle. #74263

• Fixed the issue of paddle.distributed.fleet.utils.hybrid_parallel_util.fused_allreduce_gradients in 0-D scenarios. #74957

• Fixed the issue of paddle.matmul in distributed mode. #74989

Enhanced functionality

• For scenarios involving the return of multiple Tensor objects, the experience has been optimized through encapsulation using the Paddle data structure, including paddle.topk.#74931

• Create a class API to support the usage of variable-sized parameters. #74494

Documents

• Added or fixed documentation. #74453, #74846, #74982

Other

• Optimization related to code style. #74654,#74655,#74665,#74660,#74667,#74664,#74662,#74661,#74658,#74657,#74666,#74659,#74663,#74656,#74673,#74672,#74671,#74674,#74675,#74670,#74669,#74677,#74709,#74714,#74712,#74713,#74704,#74746,#74748,#74743,#74742,#74744,#74745,#74747,#74794,#74789,#74793,#74786,#74791,#74787,#74827,#74608,#74288,#74287,#74385,#74395,#74475,#74647

• Optimization related to MKLDNN/ONEDNN. #74299,#74244,#74230,#74314,#74327,#74325,#74326,#74315,#74399,#74398,#74393,#74392,#74367,#74391,#74423,#74424,#74436,#74417,#74410,#74473,#74458,#74501,#74487,#74502,#74513,#74518,#74516,#74507,#74504,#74505,#74509,#74535,#74536,#74517,#74503,#74557,#74550,#74575,#74587,#74576,#74588,#74549,#74581,#74583,#74628,#74630,#74635,#74679,#74648,#74127,#74636,#74552,#74551,#74678,#74680,#74730,#74751,#74895,#74821,#74897,#74734

• Optimizations related to code implementation, variable and file renaming. #74309, #74597, #74613, #74376, #74479, #74960, #74968, #74977

• Optimizations related to unit tests, and bug fixes for unit test issues. #74595

• Compilation-related optimizations and CI issue fixes. #74356, #74936

• Optimize debugging and printing information, and optimize error reporting information. #74765, #74381, #74384, #74386, #74387, #74383, #74519, #74520, #74468

• Optimizations related to custom operators. #74402

• Distributed FlexCheckpoint support. #74966, #74593, #74785, #74814

New features

• Support for dynamic graphs. #74484

• Support for safetensors. #74642, #74609, #75049

• Added offloader to optimize computation efficiency. #74837

• Added API support for forward computation of conv_transpose. #74431

• Added offloader to optimize computation efficiency. #74837

• The inference deployment has added w4afp8 quantization inference, supporting w4afp8 quantization weight pure permutation and all2all communication #74270

Bug fixes

• Core framework and infrastructure optimization. #74336, #74554, #74634

• Calculation accuracy and type handling. #74278, #74222, #74830

• Optimization of dynamic dimension check logic. #74633, #74650

• Memory and illegal access fixes. #74347, #73443, #74953

• Fixed printing of error/warning messages. #74474, #74533, #74685, #74721, #74754

• Code quality and documentation correction. #74378, #74828

• Fixed the processing logic of the flashmask API. #74928

• Fixed the issue where splitting CudaGraph subgraphs did not take effect in dynamic-to-static mode. (#74749)

Enhanced functionality

• C++ extension development. #74338

• Optimization of FlexCP function. #74752, #74981

• Optimize memory allocation. #74463

Deprecated

• Clean up old IR-related unit tests for dynamic, static, and transition scenarios. #74698, #74715, #74718, #74782, #74962

Other

• Update patch version. #74940

Parallel strategy

In version 3.2, we have made multiple enhancements to the pipeline parallelism feature, including implementing support for dictionary parameter passing and extending the compatibility of Pipeline Layer and SharedLayerDesc with non-pipeline parallelism. Additionally, we have fixed several critical issues, such as IPC API exceptions for large-sized tensors, evaluation batches and non-computational losses in pipeline parallelism, gradient release errors in MoE models, hang issues caused by NCCL communication reconstruction in PP scenarios, and event management errors in dual-pipeline parallelism. Furthermore, we have conducted various performance optimizations, improved the computation overlap efficiency of dual-pipeline parallelism to enhance training performance, and upgraded the clear_param_storage method to support the clearing and resetting operations of multiple color collections in sharding mode.

Automatic parallelism

Communication Library

In version 3.2, we fixed an error in DeepEP's support for sm90 compilation, added a pre-allocation function to the video memory allocation requested by DeepEP, and upgraded its intranode and internode computation kernels, further optimizing performance and stability.

New features

• API compatibility support. #74506, #74676, #74558, #74572, #74691, #74703, #74750, #74757, #74802, #74546, #74547, #74802, #74859, #74910, #74873, #74882, #74901, #74899, #74449

• Added fused_partial_rope operator. #74577

Bug fixes

• 0-size Tensor related fixes. #74295, #74305, #74323, #74354

• Major Tensor-related fixes. #74242, #74293, #74289, #74279, #74330, #74329, #74342, #74369, #74370, #74404, #74537, #74451, #74172, #74324, #74964, #74360, #74379, #74377, #74380, #74362, #74197

• API compatibility-related fixes. #74764, #74869, #74935

• [Open Source Task] Investigate and resolve precision issues in Paddle CPU/GPU Kernels. #74149, #74598, #74719, #74625, #74555

• Other important fixes. #74282, #74313, #74303, #74306, #74298, #74044, #74290, #74348, #74364, #74332, #74224, #74382, #74406, #74434, #74448, #74457, #74322, #74530, #74716, #74839, #74842, #74854, #74919, #74767, #75003

Enhanced functionality

• Improved API compatibility. #74456, #74480, #74523, #74490, #74548, #74596, #74568, #74559, #74629, #74623, #74700, #74643, #74602, #74783, #74781, #74735, #74725, #74815, #74856, #74925, #74545, #74932, #74784

• Slice/stride related optimizations. #74731, #74740, #74769, #74810, #74841, #74954, #74888, #74944, #74312, #74291, #74271, #74320, #74344, #74727, #74637

• Operator optimization and CUDA support. #74693, #74922, #74967

• Improved debugging information and compatibility enhancements. #74372, #74622

• Operator function expansion and optimization. #74790, #74979

Performance optimization

• FP8 computation optimization. #74471, #74684, #74911

• Basic operator performance optimization. #74442, #74638

• Support fa3 variable-length sequence reverse computation and optimize forward API. #73831

• Added FlashMask V2 function. #74729

Documents

• Fixed issues with English documentation and copyright year. #74737

Other

• The WITH_XPU_FFT option is enabled by default on XPU hardware. #74699

Improved CUDA-like hardware integration solution

• The CUDA-like hardware access solution supports the reuse of cuBlas kernels #74591,

• Fix known issues in the CUDA-like hardware access solution #74397, #74411, #74428, #74877, #74939

Main warehouse supports multiple hardware for single testing

• Single test supports multiple hardware #74349, #74363, #74806, #74868, #74820, #74927

New Custom Device API Support

• Added support for Custom Device API #74308, #74371, #74539

Bug fixes

• Fixed the bug in flashattent compilation cache. #74388

• Fixed the bug where site.USER_SITE was None. #74373

• Fixed the compilation bug of gtest in multi-architecture Linux systems. #74723

• Fixed multiple compilation errors in DEBUG mode when WITH_GPU=ON. #74401

• Fixed the compilation bug of CUDA12.6 under Windows. #74990

• Fixed the bug in the api-benchmark baseline pipeline. #74770

• Fixed the bug in the api-benchmark baseline pipeline. #74778

• Fixed the bug in the api-benchmark baseline pipeline. #74779

• Fixed the bug in the api-benchmark baseline pipeline. #74780

• Fixed the bug in the api-benchmark baseline pipeline. #74800

• Fixed the bug in the api-benchmark baseline pipeline. #74803

Other

• Disable the test_custom_contiguous unit test. #74337

• Support for timed triggering of baseline tasks in the slice pipeline. #74419

• Support manually specifying the pr for adding slice recording baselines. #74445

• Check if there are any issues in the code. #74460

• Support CI PaddleX tasks on XPU. #74426

• Support slice pipeline exemption mechanism. #74482

• Updated the Paddle base image. #73423

• Fixed Ninja version 1.11 for Windows. #74590

• Support adding the ability to close PRs and cancel CIs. #74604

• Support for quickly skipping all CI. #74696

• Add an api-benchmark baseline pipeline. #74690

• Update the nccl version. #74809

• Update the RD list for the approve pipeline. #74838

• Update the RD list for the approve pipeline. #74902

• Update safetensor to the mirror. #74904

• Added the compilation flag for flashatten. #74959

• Temporarily disable the win-inference pipeline. #74980

• Support for compiling phi dynamic libraries on Windows. #74950