ɯwgZ ddlZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddl m cmZddlmZddlmZddlmZ ddlmZe j6rdZn dd lmZ eee!eee!e jDeed e!fe#fZ$ d Z%dnde jLde!de$de#fdZ'dnde jLde#de$de#fdZ( dnde jLde!de$de#fdZ) dnde jLde!de$de#fdZ* dnde jLde#de!de$de#f dZ+ dnde jLde#de!de$de#f dZ, dndee jLde#de$de#dee jLf dZ- dndee jLde$de#dee jLfdZ. dndee jLde#dee!de$de#dee jLf dZ/d Z0 dnde jLd!eee!d"eee!de$de#de jLf d#Z1 dnde jLd!eee!d"eee!de$de#de jLf d$Z2 dnde jLd%ee!de$de#de jLf d&Z3Gd'd(e jLZ4 dnde$de#dee#ee!e!ffd)Z5dnde$de#de#fd*Z6Gd+d,e jnjpZ9de:fd-Z;de jLfd.Zd1Z?d2Z@d3ZAd4ZBd5ZCd6ZDd7ZEd8ZFd9ZGd:ZHd;ZId<ZJd=ZKd>ZLd?ZMe j6sPe jjd@dAZPePjdBe?dCePjdDeDdCePjdEeCdCePjdFeFdCePjdGe@dCePjdHeIdCePjdIeJdCePjdJeKdCePjdKeLdCePjdLeMdCePjdMeHdCePjdNe>dCePjdOeEdCe jjje jjjJje jjje jjjJe jjdPdQZXe jjdPdAZYgdRZZeje\Z]eZD]\Z^e^de^jdSZ`eaedTe`ZbeXje^e jjUeYje`ebdV^nej@dW dodXe jLdYe jLdZe:de#de!f d[Zf dpd]e jLd^e jLd_e#dZe:de!de#f d`Zge jjd\e jjdae jjdbe jjdce jjdde jjdee jjdfe jjdgiZq dqdhe jLd_e#dZe:de#fdiZr drd]e jLd^e jLde#fdjZs dsdkee jLdhe jLde#fdlZtddml muZvmwZxmyZzm{Z|m(Z}m1Z~m+Ze|efeege}ere~esezetexegevefiZy#e$r ddlmZYwxYw#e$rej@d d ZYwxYw)tN)castListOptionalTuple TYPE_CHECKINGUnion) DeviceMesh)get_proxy_mode)_functional_collectives_impl) tree_map_onlycy)z9Can't import torchdynamo in torchdeploy builds currently.Frn/home/mcse/projects/flask/flask-venv/lib/python3.12/site-packages/torch/distributed/_functional_collectives.pyis_torchdynamo_compilingrsr)is_dynamo_compilingzdUnable to import torchdynamo util `is_torchdynamo_compiling`, so won't support torchdynamo correctlycy)NFrrrrrr#srzdist.tensor.DeviceMeshcTtjjj|S)z Wait on a tensor returned by the collectives ops. Waiting follows device semantics, which means blocking on CPU and synchronizing streams on CUDA. )torchops_c10d_functional wait_tensor)tensors rrrs 99 % % 1 1& 99rselfsrcgrouptagct||}tjjj |||}t |S)a Broadcasts the tensor to all processes in the given process group. Args: src (int): Source rank group (ProcessGroup or List[int]): The process group to work on. tag (str, optional): A unique identifier for the collective. Default: empty string )_resolve_group_namerrr broadcast_maybe_wrap_tensor)rrrr group_namers rr!r!s9%UC0J YY ' ' 1 1$Z HF f %%rreduceOpct||}tjjj ||j |}t |S)a Reduces the tensor data across all machines in such a way that all get the final result. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. )r rrr all_reducelowerr")rr$rrr#rs rr&r&s@"%UC0J YY ' ' 2 249I: VF f %%r gather_dimc||jsJt||}tj|}tj j j|||}t|}|dk7rMt|tr|j}t jt j||d|}|S)a% Gather tensor data across from all machines and concatenate over ``gather_dim``. Note that it currently only supports gather_dim = 0. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. rdim) is_contiguousr c10d_get_group_size_by_namerrrall_gather_into_tensorr" isinstanceAsyncCollectiveTensorwaitcatchunkrr(rrr# group_sizerress rall_gather_tensorr8s,     $UC0J--j9J YY ' ' > > j*F V $CQ c0 1((*Cii C;L Jrclt||}tj|}tjj j |||}tj|}|dk7rMt|tr|j}tjtj||d|}|S)a< Gather tensor data across from all machines and concatenate over ``gather_dim``. Note that it currently only supports gather_dim = 0. This function is the same as all_gather_tensor but will propagate the backwards gradient across workers. See all_gather_tensor for more details on usage. rr*)r r-r.rr_c10d_functional_autogradr/_FromTorchTensorapplyr0r1r2r3r4r5s rall_gather_tensor_autogradr=s %UC0J--j9J YY 0 0 G G j*F   (CQ c0 1((*Cii C;L Jr scatter_dimct||}tj|}|j||zdk(sJd|jdd||dk7r-t j |||}t j |}tjjj||j||}t|} | S)a( Reduces the tensor data across all machines in such a way that all get the final result, then scatter the results to corresponding ranks. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. rinput dimension 0 (" must be a multiple of group_size r*) r r-r.sizerr4r3rrreduce_scatter_tensorr'r" rr$r>rrr#r6 tensor_listrr7s rrCrCs,%UC0J--j9J ++q0Z TYYq\N*LZLYZ0akk$ D yy% YY ' ' = =  F V $C Jrct||}tj|}|j||zdk(sJd|jdd||dk7r-t j |||}t j |}tjjj||j||}tj|} | S)a| Reduces the tensor data across all machines in such a way that all get the final result, then scatter the results to corresponding ranks. This function is the same as reduce_scatter_tensor but will propagate the backwards gradient across workers. Currently only the "sum" reduceOp is supported. See reduce_scatter_tensor for more details on usage. rr@rAr*) r r-r.rBrr4r3rr:rCr'r;r<rDs rreduce_scatter_tensor_autogradrG%s&%UC0J--j9J ++q0Z TYYq\N*LZLYZ0akk$ D yy% YY 0 0 F F  F   (C Jrreturnct||}tjjj ||j |}t tt|S)a  Reduces a list of tensors across all machines in such a way that all get the final result. The all tensors in the input list are left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. ) r rrrall_reduce_coalescedr'listmapr")rr$rrr#rEs rrJrJLsO&%UC0J)),,AA K & 4 55rct||}tj|}tjj j |||}ttt|S)a Gather a list of tensors across from all machines. Note that it currently only supports gather_dim = 0. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. ) r r-r.rrr all_gather_into_tensor_coalescedrKrLr")rrrr#r6rEs rrNrNhsX&%UC0J--j9J)),,MM K & 4 55rinputsc :t||}tj|}t|t|k(sJt t ||D]w\}\}} | j ||zdk(s"Jd|d| j |d|d||dk7sHtj| ||} tj| ||<ytjjj||j||} ttt | S)a, Reduces a list of tensors across all machines in such a way that all get the final result, then scatter the results to corresponding ranks. The input tensors are left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. rzinput dimension z (rAz for tensor at index r*)r r-r.len enumerateziprBrr4r3rrreduce_scatter_tensor_coalescedr'rKrLr") rOr$r>rrr#r6idxr+rrEs rrTrTs,%UC0J--j9J { s6{ ** *'K(@A1]c6 KK z )Q . @ cU"V[[%5$66XYcXddyz}y~  @ . !8++fjcBK))K0F3K 1)),,LL K & 4 55rct|tjjsJ|j}t |j dkDr6|j d}|jduxr|jj SyNr) r0r_ops OpOverload_schemarQ arguments alias_infois_write)tgtschema first_args r _is_view_oprasr c5::00 11 1 [[F 6  q $$Q' ##4/U 8L8L8U8U4UU!routput_split_sizesinput_split_sizescl|td|DsJ||td|DsJ|t||}tj|}||#||Jd|jd|zg|z}|}t j jj||||}t|S)aC Each process splits input tensor and then scatters the split list to all processes in a group. Then concatenate the received tensors from all the processes in the group and return single output tensor. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. c3\K|]$}t|ttjf&ywNr0intrSymInt.0rBs r z$all_to_all_single..% 6:Jtc5<<0 1 *,c3\K|]$}t|ttjf&ywrfrgrjs rrlz$all_to_all_single..rmrn^output_split_sizes and input_split_sizes must either be specified together or both set to Noner) allr r-r.shaperrrall_to_all_singler"rrbrcrrr#r6rs rrsrss,% >P      $ >O      %UC0J--j9J!%6%>!).?.G  5 G#jjmz9:ZG. YY ' ' 9 9  F f %%rc|td|DsJ||td|DsJ|t||}tj|}||#||Jd|jd|zg|z}|}t j jj||||}tj|S)z: Same as all_to_all_single but supports autograd. c3\K|]$}t|ttjf&ywrfrgrjs rrlz-all_to_all_single_autograd..rmrnc3\K|]$}t|ttjf&ywrfrgrjs rrlz-all_to_all_single_autograd..rmrnrpr) rqr r-r.rrrrr:rsr;r<rts rall_to_all_single_autogradrxs% >P      $ >O      %UC0J--j9J!%6%>!).?.G  5 G#jjmz9:ZG. YY 0 0 B B  F  ! !& ))rsrc_dstcZt||\}}}tj|||}dg|z}dg|z} t|D]\\} } | t j |k(r|j | | <| t j |k(sJ|j || <^t||| ||S)a" Permutes the elements of the tensor according to the given source/destination pairs. `src_dst` should be defined such that src_dst[m] == n means m sends to n. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one r) _expand_groupr-#_find_or_create_pg_by_ranks_and_tagrRdistget_ranknumelrs) rryrrtranksetr6local_pgrbrcrdsts rpermute_tensorrs"+5#6Aw 777JOHz)j(g&3S $--) )%)ZZ\ c " $--) )&*jjl s # 3 T#57H%QT UUrceZdZUdZej ed<eed<ddgZe dej fdZ dZ dZ e dZ dZd Zd ej fd Zd ZeddZdZy )r1a A Tensor wrapper subclass that is used to trigger a call to wait prior to first use of the underlying tensor. Use it inside functional collective pytorch wrappers like the following: def functional_collective(self, group, tag): tag, rankset, group_size = _expand_group(group, tag) tensor = torch.ops.c10d_functional.{collective}(self, tag, rankset, group_size) return _maybe_wrap_tensor(tensor) elem completedc tjj||j|j |j |j |j|j|j}||_ d|_ |S)N)stridesstorage_offsetdtypelayoutdevice requires_gradF) rTensor_make_wrapper_subclassrBstriderrrrrrr)clsrrs r__new__zAsyncCollectiveTensor.__new__?so LL / /  IIKKKM..0**;;;;,, 0  rc dgdfSNrrrs r__tensor_flatten__z(AsyncCollectiveTensor.__tensor_flatten__Osx~rc>|jjSrf) trigger_waittolistrs rrzAsyncCollectiveTensor.tolistRs  "))++rc*|J|d}t|Sr)r1) inner_tensorsmeta outer_size outer_striders r__tensor_unflatten__z*AsyncCollectiveTensor.__tensor_unflatten__Us!||V$$T**rc*d|jdS)NzAsyncCollectiveTensor())rrs r__repr__zAsyncCollectiveTensor.__repr__[s'(9(9(;'>rcn|jst|j}d|_|S|jSNT)rrr)routs rrz"AsyncCollectiveTensor.trigger_wait^s-~~dii(C!DNJ99 rrHc,t|jSrf)rrrs rr2zAsyncCollectiveTensor.waitfs499%%rc|jS)zOThis method enables _functional_collectives_impl to test if a tensor is an ACS)rrs r_get_acs_underlying_tensorz0AsyncCollectiveTensor._get_acs_underlying_tensoris yyrNc |tjjjjk(r&||dj |d}t |}|St| dt f fd }dtjfd}tt ||} tt ||} || i| } rttj|| } | S)Nrr ec@s|jS|jSrf)rr)r is_view_ops runwrapz8AsyncCollectiveTensor.__torch_dispatch__..unwrapxs~~''66Mrc@t|trJt|}|Srf)r0r1)rr7s rwrapz6AsyncCollectiveTensor.__torch_dispatch__..wrap~s"!!%:; ;;'*CJr) rratenviewdefaultrr1rarr ) rfunctypesargskwargsr7 wrapper_resrrunwrapped_argsunwrapped_kwargsrrs @r__torch_dispatch__z(AsyncCollectiveTensor.__torch_dispatch__ms 599>>&&.. .tAw||T!W-C/4K  &  +   ELL  ''ON7&67  dC8C rc>|jjSrf)r2numpyrs rrzAsyncCollectiveTensor.numpysyy{  ""r)rN)__name__ __module__ __qualname____doc__rr__annotations__bool __slots__ staticmethodrrrrrrr2r classmethodrrrrrr1r10s ,,O%I 5<<  ,++ ?&ell&  D#rr1c trd}d}nd}d}t|trt|dtr^||}g}d}|D]K}|j||dk7r(|t |k7rt d|dt |t |}Mn1||}t |}nt|t jr:t j|}t |}|xstj|}nt|tr5|jd k(sJd |jd\}}}t |}nt|trht |d k(rOt|dtrtttt|Srfrrrhxs rcast_listlistintz'_expand_group..cast_listlistintsT#Y+ +rc0ttt|Srfrrs r cast_listintz#_expand_group..cast_listintsS 1% %rc|Srfrrs rrz'_expand_group..cast_listlistintHrc|Srfrrs rrz#_expand_group..cast_listintrrrz$group sizes must be identical found z and r JOnly 1D mesh is supported, pass in (DeviceMesh, int) together if mesh > 1D1Invalid tuple for group must be (DeviceMesh, int)z[Invalid type for group, must be one of List, Processgroup, DeviceMesh or (DeviceMesh, int).)rr0rKextendrQ ValueErrorr} ProcessGroupget_process_group_ranksr-_get_group_tagr ndim_dim_group_infostuplerh) rrrr nested_listrr6rs_dmeshr+s rr{r{s , &  % eAh %*51KGJ! %r"# c"g(=$>zl%PSTVPWyY!W  %#5)GWJ E4,, -..u5\ /T((/ E: & JJ!O X W X  003Wa\ E5 ! J!O58Z058S)!HE(C#44S9OC!WJPQ Q i   * %%rct|tjr |jSt|tr|St|t r(|j dk(sJd|jddSt|tr[t|dk(rBt|dt r/t|dtr|d}|d}|j|dStdt|trQtstjdt dt#j$t't(t||Std t+|d |) z; Given group in RANK_TYPES, return the group name. r rrrrzThe combination of ranks + tag as process group identifier has been deprecated. Please switch to using ProcessGroup, DeviceMesh, or group name instead.) stacklevelzUnsupported group type: z, )r0r}rr#strr rrrrQrhrrKrwarningswarn FutureWarningr-$_resolve_group_name_by_ranks_and_tagrrtype)rrrr+s rr r s9 %**+ E3  E: & JJ!O X W X %%a(++ E5 ! J!O58Z058S)!HE(C))#.q1 1PQ Q E4 ') MMI  88d3i9OQTUU3DK=5'JKKrceZdZdZedej dej fdZedej dej fdZy)r;zm _FromTorchTensor allows autograd to propagate from a normal Tensor to an AsyncCollectiveTensor. inputrHct|Srf)r")ctxrs rforwardz_FromTorchTensor.forwards "%((r grad_outputc|Srfr)rrs rbackwardz_FromTorchTensor.backwardsrN) rrrrrrrrrrrrr;r; s_ )||) )) 5<<ELLrr;ctrytjjtjjj  yt duSr)rr_C_get_dispatch_mode_TorchDispatchModeKey FUNCTIONALr rrr_are_we_tracingrsJ! ##EHH$B$B$M$MN   4 ''rcvtr t|St|}ttj |Srf)rrr1rrr)rr7s rr"r",s.4   %C  c ""rcDfd}|Dcgc] }|| c}Scc}w)Ncvt|j}|dxxzcc<|j|}|SrWrKrB new_empty)shardout_size out_tensorr6s r mk_out_tensorz=_all_gather_into_tensor_coalesced_meta..mk_out_tensor4s5 % z! __X. rr)rrrr6rrs ` r&_all_gather_into_tensor_coalesced_metar3s# '+ +M!  ++ +c,tj|Srfr empty_likerrs r_broadcast_metar >   D !!rc,tj|Srfrrs r_all_reduce_metar Br rc,tj|Srfrrs r_wait_tensor_metarFr rcpt|j}|dxx|zcc<|j|SrWr)rrrr6rs r_all_gather_into_tensor_metarJs/EJJL!H QK:K ??8 $$rcpt|j}|dxx|zcc<|j|SrWr)r reduce_oprrr6rs r_reduce_scatter_tensor_metarPs/EJJL!H QKJK ??8 $$rcR|Dcgc]}tj|c}Scc}wrfr)rrrs r_all_reduce_coalesced_metarVs!)- .AE  Q  .. .s$c|Srfrinprs r_all_reduce__metarZ Jrc|Srfrrs r_broadcast__metar^rrc|Srfr)rOrs r_all_reduce_coalesced__metarbs MrcDfd}|Dcgc] }|| c}Scc}w)Ncvt|j}|dxxzcc<|j|}|SrWr)rrrr6s rrz<_reduce_scatter_tensor_coalesced_meta..mk_out_tensorgs5 % " __X. rr)rOr$rrr6rrs ` r%_reduce_scatter_tensor_coalesced_metar!fs# '- -M!  -- -rc||j|jS|D]}tj|t |j}t ||d<|j|SrW)rrBr_check_is_sizerKsum)rrbrcrrsrs r_all_to_all_single_metar&usj!uzz|,,# $A   # $ %,- x((rcpt|j}|dxx|zcc<|j|SrWr)rr6r#rrrs r'_all_gather_into_tensor_out_native_metar(/  E !H H ??5 !!rcpt|j}|dxx|zcc<|j|SrWr)rr6r#rrs r#_all_gather_into_tensor_native_metar+r)rcB|Dcgc]}t|||c}Scc}wrf)r+)rOr6r#rs r-_all_gather_into_tensor_coalesced_native_metar-s-   ,E:zJ  scpt|j}|dxx|zcc<|j|SrWr)rrr6r#rrs r"_reduce_scatter_tensor_native_metar/s/  E !HH == rc D|Dcgc]}t||||c}Scc}wrf)r/)rOrr6r#rs r,_reduce_scatter_tensor_coalesced_native_metar1s/    +3 :zR  srIMPLr&Meta all_reduce_rJall_reduce_coalesced_rall_gather_into_tensor_outr/rNrCrTrsr! broadcast_c10d_functionalDEF) zObroadcast(Tensor self, int src, str tag, int[] ranks, int group_size) -> TensorzUall_reduce(Tensor self, str reduceOp, str tag, int[] ranks, int group_size) -> Tensorzcall_reduce_coalesced(Tensor[] self, str reduceOp, str tag, int[] ranks, int group_size) -> Tensor[]z"wait_tensor(Tensor self) -> TensorzTall_gather_into_tensor(Tensor shard, str tag, int[] ranks, int group_size) -> Tensorzball_gather_into_tensor_coalesced(Tensor[] input, str tag, int[] ranks, int group_size) -> Tensor[]zareduce_scatter_tensor(Tensor input, str reduceOp, str tag, int[] ranks, int group_size) -> Tensorzpreduce_scatter_tensor_coalesced(Tensor[] inputs, str reduceOp, str tag, int[] ranks, int group_size) -> Tensor[]zall_to_all_single(Tensor input, SymInt[]? output_split_sizes, SymInt[]? input_split_sizes, str tag, int[] ranks, int group_size) -> Tensor(r)tagsCompositeImplicitAutogradzJPyTorch Distributed functional collectives do not work with torch::deploy. output_tensor input_tensorasync_opc|rJd|xstjj}|J|jt ||||SN@Can't remap async version of inplace op to functional collective)r}rWORLDcopy_r8)r=r>rr?rr(s rall_gather_tensor_inplacerEsY JIJ  %TZZ%%E      0z5RUV WWrr$outputropc |rJd|xstjj}|J|jt |||||SrA)r}rrCrDrC)rFrrGrr?r>rs rreduce_scatter_tensor_inplacerIsX JIJ  %TZZ%%E    <<-eReSQ RRravgproductminmaxbandborbxorrc|rJd|xstjj}|J|jt ||||SrA)r}rrCrDr&)rrGrr?rs rall_reduce_inplacerRsU JIJ  %TZZ%%E    << 62uc: ;;rc |rJd|xstjj}|J|jt |||||SrA)r}rrCrDrs)rFrrbrcrr?rs rall_to_all_inplacerT/sc JIJ  %TZZ%%E    <<        rrEc |rJdtfd|DsJd|xstjj}|Jt d||}g}d}|D]<}|j ||||j dz||j dz }>t||D]\} } | j| |S)NrBc3dK|]'}|jdjdk()yw)rN)rB)rkrrs rrlz%all_gather_inplace..Ts+()q V[[^#s-0z7Remapping variable size all_gather is not yet supportedr) rqr}rrCr8appendrBrSrD) rErrr?rrF output_splitsoffsetrrrs ` rall_gather_inplacerZJs JIJ -8 A@A   %TZZ%%E    vq% 5FM F VFVaffQi-?@A!&&) ]3S # r)_all_gather_base_reduce_scatter_base all_gatherr/r&rsrC))Fr^r)r$NFrr^)r$NFr^)NNNFr^)NFr^)sysrtypingrrrrrrrtorch.distributed distributedr}"torch.distributed.distributed_c10ddistributed_c10dr-torch.distributed.device_meshr "torch.fx.experimental.proxy_tensorr r^r fun_col_impltorch.utils._cxx_pytreer ImportErrortorch.utils._pytree_running_with_deployrtorch.compilerr Exceptionrrhrr RANK_TYPESrrr!r&r8r=rCrGrJrNrTrarsrxrr1r{r autogradFunctionr;rrr"rr r rrrrrrrr!r&r(r+r-r/r1libraryLibrarylib_implimplfxnodehas_side_effectrrr legacy_liblegacy_lib_implops_defsmodulesr my_moduleop_defindexop_namegetattr backend_impldefineTagpt2_compliant_tagrErIReduceOpSUMAVGPRODUCTMINMAXBANDBORBXORREDUCE_OP_TO_STRrRrTrZr[legacy_all_gather_baser\legacy_reduce_scatter_baser]legacy_all_gatherr/legacy_allgatherlegacy_allreducelegacy_all_to_all_singlelegacy_reducescattertraceable_collective_remapsrrrrs/  DD 114=:25 5 R #J IcO "C '(   :: &ELL &s &: &C &&U\\&S&&#&4 $ ,,$$ $ $V  ,,  L ' ,,'''  ' '^ $ ,,$$$  $ $PLN6 u|| 6(+64>6EH6 %,,6:=?6 u|| 6%/6696 %,,6D )6  )6)6c)6  )6 )6  %,, )6\V -& ,,-& c+-& S *-&  -& -&  \\ -&j "* ,,"* c+"* S *"*  "* "*  \\ "*R V ,,V #YV V V  \\ V>a#ELLa#H H&H&#H&uS$s)S=P7QH&V%Lz%L%LS%LPu~~..$ ( (# #,"""% % /. )" "  "u!!#}}$$%7@H MM, 0&9 MM-!2F; MM(*DfM MM)+FO MM-!2F; MM$&Mv MM*,OQWX MM*5  MM)+MvV MM)4  MM%'>G MM+7 MM, 0&9 HHMM!!%))"<"<"H"H"P"PQ HHMM!!%))"<"<"H"HI&&'8%@Jmm++,=vFO H H%IQV\\#./|q ]; &uyy'B'BCWl4OP QHMMT  X<<X,,X X X  X* S LLS <<S S  S  S S( MMuMMuMM9MMuMMuMMMMuMM   < LL< < < <(   LL << <   ell# LL >/7(0) =5K#2112  r   s#[[ [[[76[7