DŽgbddlZddlZddlmZmZmZddlZgdZdZ dZ dZ dZ dZ deejeejfd dfd Zde ddfd Zdde ddfdd deejd eeejeejfdededeeej$j&deeejd dfdZ ddeejded dfdZ ddeejdeejd dfdZe ddfdeejdeejded dfdZy)N)OptionalSequenceUnion) all_reducereduce broadcast all_gatherreduce_scatterc0ttjdstjdyt }|D]W}|j ry|jsy|jsy|j}||vry|j|Yy)N_nccl_all_reducez)PyTorch is not compiled with NCCL supportFT) hasattrtorch_Cwarningswarnset is_sparse is_contiguousis_cuda get_deviceadd)tensorsdevicestensordevices W/home/mcse/projects/flask_80/flask-venv/lib/python3.12/site-packages/torch/cuda/nccl.py is_availablers 588/ 0 ABeG    ##%~~""$ W  F  ctjj}|dz }|dz dz}|dz}tjjj d}|dk(r|||fS||||fS)a Returns the version of the NCCL. This function returns a tuple containing the major, minor, and patch version numbers of the NCCL. The suffix is also included in the tuple if a version suffix exists. Returns: tuple: The version information of the NCCL. izutf-8)rr _nccl_version_nccl_version_suffixdecode)vermajorminorpatchsuffixs rversionr+#sv (( "C 2IE BY% E %KE XX * * , 3 3G tjjSN)rr_nccl_unique_idrr unique_idr08s 88 # # %%rcDtjj|||Sr-)rr_nccl_init_rank) num_ranksuidranks r init_rankr6<s 88 # #IsD 99rinputsreturnct|tjjrt|tj r t dy)Nz(Inputs should be a collection of tensors) isinstance collectionsabc ContainerrTensor TypeError)r7s r_check_sequence_typer@@s9 fkoo77 8J =BCC=rc~t|||}t|tjj|||||yr-)r@rrr r7outputsopstreamscommss rrrGs6 ! HHfgr7EBr)rCoutputrootrDrErCct||/| tdtjdtd||}nit |t jsFt |tjjr"tjdtd||}n |||n|}t jj||||||y)Nz'output' and 'outputs' can not be both specified. 'outputs' is deprecated in favor of 'output', taking in a single output tensor. The signature of reduce is: reduce(inputs, output=None, root=0, op=SUM, streams=None, comms=None).z`nccl.reduce` with an output tensor list is deprecated. Please specify a single output tensor with argument 'output' instead instead.) stacklevelz\nccl.reduce with an output tensor list is deprecated. Please specify a single output tensor.) r@ ValueErrorrr FutureWarningr:rr>r;r<rr _nccl_reduce)r7rGrHrDrErFrC_outputs rrrQs   Y  MM`   dmG  -* ((3   5   ,"(.&,f HH&'4WeDrc^t|tjj||||yr-)r@rr_nccl_broadcast)r7rHrErFs rrr|s$  HHVT7E:rctt|t|tjj||||yr-)r@rr_nccl_all_gather)r7rCrErFs rr r s,  ! HHfgw>rcvt|t|tjj|||||yr-)r@rr_nccl_reduce_scatterrBs rr r s. ! HH!!&'2wFr)rNN)NN)r;rtypingrrr torch.cudar__all__SUMrr+r0r6r>r@rintcudaStreamrrr r r/rrr]s,, P*-*&:Du||Xell5K'K!LDQUD $TCEI59  (E15(E U\\ "(E U5<<%,,)??@ A(E (E (E huzz001 2 (Ehu||, -(E (EXHL; U\\ ";*-; ;   ? U\\ "? ell #?  ?   G U\\ " G ell # G  G  Gr