DŽgn$ddlmZmZmZmZmZmZmZddlZddlm Z ddl m Z m Z m Z mZmZmZddgZGddeZd d e zd ze_d ee d ee deee deee deee dee dee dee dedee efdededeedededef dZdedeeeej.eej0efeeee ffdZd ee d ee deee deee deee dee dee dee dedee efdededeedededef d Ze e! d$d ee d ee deee deee deee dee d"eedee dee dededeee fdededededef"d#Zy)%)castDictListOptionalTuple TYPE_CHECKINGUnionN)Tensor)_disable_dynamo_if_unsupported_get_scalar_dtype _maximize_doc OptimizerParamsTTensorListList Adafactor adafactorceZdZ dddddedeeefdedeeeefded ed ee d e ffd Z fd Z dZ e jddZxZS)rNF)foreachmaximizeparamslr beta2_decayepsd weight_decayrrc t|tr|jdk7r tdd|kstd|d|k\std||dd|dkstd|dd|dkstd|dd |kstd |d|kstd |t ||||||| } t ||| y) Nr zTensor lr must be 1-elementz%Learning rate should be >= 0 but is: z#beta2_decay should be <= 0 but is: rz epsilon1 should be >= 0 but is: z epsilon2 should be >= 0 but is: ?z,Clipping threshold d should be >= 1 but is: z$weight_decay should be >= 0 but is: )rrrrrrr) isinstancer numel ValueErrordictsuper__init__) selfrrrrrrrrdefaults __class__s ^/home/mcse/projects/flask_80/flask-venv/lib/python3.12/site-packages/torch/optim/_adafactor.pyr%zAdafactor.__init__s b& !bhhjAo:; ;byDRDIJ Jk!B;-PQ Q q6 cSVm?AxHI Ic!f}?AxHI IaxKA3OP Pl"CL>RS S#%  *cft|||jD]}|jdd|dD]v}|jj |g}t |dk7s.tj|drGt|d}tj|t|d<xy)Nrrrstepdtype) r$ __setstate__ param_groups setdefaultstategetlentorch is_tensorfloattensorr )r&r2grouppp_statestep_valr(s r)r/zAdafactor.__setstate__;s U#&& XE   Y -8_ X**..B/w<1$U__WV_-M$WV_5H&+ll8CTCV&WGFO  X Xr*c$|dD]}|jtj|r td|jjr td|j ||j |j|j |} t| dk(rtjdt| d<|jjdkDrt|jj} d| d <|jj| | d <t|jj} d| d <|jj| | d <n2tj|jtj | d<|j | j!d d|j | j!d d|j | j!dd|j | d y)Nrz-Adafactor does not support complex parametersz+Adafactor does not support sparse gradientsrrr-r,r row_varcol_var) memory_formatvarianceF)gradr5 is_complex RuntimeError is_sparseappendr2r4r8r dimlistshape new_zeros zeros_likepreserve_formatr3) r&r9params_with_gradgradsrow_varscol_vars variances state_stepsr:r2 row_shape col_shapes r) _init_groupzAdafactor._init_groupEsx% .Avv~""#RSSvv"#PQQ  # #A & LL JJqME5zQ!& S8I8K Lf 66::2i(z*"4t<!$ T:'# '% N U ! !s B;;C)g{Gz?g)NgMbP?rrN)__name__ __module__ __qualname__rr r7r rrboolr%r/rWr5no_gradr, __classcell__)r(s@r)rrs$(!-9!#+#'#+#+ %- #+ #+ 8E?E) * #+  #+#+$#+#+JX0dU]]_55r*aImplements Adafactor algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{(lr)}, \: \tau \text{(}\beta_2\text{ decay)}, \: \theta_0 \text{(params)}, \: f(\theta) \text{(objective)}, \\ &\hspace{15mm} \: \epsilon_1, \epsilon_2 \text{ (epsilons)}, \: d \text{(clipping threshold)}, \\ &\hspace{15mm} \: \lambda \text{(weight decay)}, \: \textit{maximize} \\ &\textbf{initialize} : \: R_0 \leftarrow 0 \text{ (second moment row factor)}, \\ &\hspace{23mm} \: C_0 \leftarrow 0 \text{ (second moment col factor)}, \\ &\hspace{23mm} \: \widehat{V}_0 \leftarrow 0 \text{ (second moment for vectors)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}G_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}G_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\widehat{\beta}_{2_t} \leftarrow 1 - t^{\tau} \\ &\hspace{5mm}\rho_t \leftarrow min(lr, \frac{1}{\sqrt{t}}) \\ &\hspace{5mm}\alpha_t \leftarrow max(\epsilon_2, \text{RMS}(\theta_{t-1}))\rho_t \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \gamma \lambda \theta_{t-1} \\ &\hspace{5mm}\textbf{if} \: \text{dim}(G_t) > 1: \\ &\hspace{10mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t) \cdot 1_m \\ &\hspace{10mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t) \\ &\hspace{10mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\widehat{V}_t \leftarrow \widehat{\beta}_{2_t}\widehat{V}_{t-1}+ (1-\widehat{\beta}_{2_t}) \cdot (G_t \odot G_t) \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ &\hspace{5mm}\widehat{U}_t \leftarrow \frac{U_t}{max(1, \frac{\text{RMS}(U_t)}{d})} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \alpha_t \widehat{U}_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost`_. a0 Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, Tensor, optional): unlike other optimizers, Adafactor does not require a learning rate, and Shazeer, Noam, and Mitchell Stern do not use lr at all. Deviating from the paper, this implementation uses lr for applying weight decay and as the maximum value for relative step size rho_t. Note that in the paper, a constant of 0.01 is used as the maximum value for relative step size, and so we set 0.01 as the default value. (default: 1e-2) beta2_decay (float, optional): the decay rate of beta2. beta2 standardly refers to the coefficient used for computing the running average of the gradient squared. (default: -0.8) eps (Tuple[float, float], optional): epsilon1 is the term added to the denominator of the update calculation to improve numerical stability. This use of epsilon1 deviates from the algorithm written in the paper! See note below for more details. epsilon2 is the term used to avoid having too small a weight update when applying parameter scaling. (default: (None, 1e-3)) d (float, optional): the clipping threshold, used to avoid larger-than-desired updates. weight_decay (float, optional): weight decay coefficient (default: 1e-2) foreach (bool, optional): whether foreach implementation of optimizer is used. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to the intermediates being a tensorlist vs just one tensor. As Adafactor is commonly used when memory is prohibitive, Adafactor will default to the slower single tensor for-loop implementation unless this flag is explicitly True. This behavior is contrary to other optimizers, which will attempt defaulting to foreach on CUDA for faster runtime. (default: None) a4 .. Note:: The implementation of Adafactor subtly differs from Shazeer, Noam, and Mitchell Stern and implementations in some other frameworks with its use of learning rate and :math:`\epsilon_1`. Regarding the learning rate hyperparameter: Shazeer, Noam, and Mitchell Stern do not use lr at all, as the stated algorithm uses :math:`\rho_t` and update clipping to affect the step size. This implementation allows `lr` to influence the maximum value for :math:`\rho_t`: .. math:: \begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(lr, \frac{1}{\sqrt{t}}) \end{aligned} This differs from Shazeer, Noam, and Mitchell Stern, who use a constant of 0.01 as the maximum value of :math:`\rho_t` .. math:: \begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(0.01, \frac{1}{\sqrt{t}}) \end{aligned} Shazeer, Noam, and Mitchell Stern do not enforce an opinion on how weight decay should be computed, and so we use the learning rate as a coefficient for decoupled weight decay, similar to what is suggested in `Decoupled Weight Decay Regularization`_. Regarding the use of :math:`\epsilon_1`: The implementation attempts to replicate the presumed intention of Shazeer, Noam, and Mitchell Stern to use :math:`\epsilon_1` as a stabilizing term when the squared gradient becomes small. This stabilization can be written as .. math:: \begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ \end{aligned} where the row and column factors of gradient squared :math:`R_t` and :math:`C_t` are left alone, and we apply :math:`\epsilon_1` at the final calculation of the variance estimate :math:`\widehat{V}_t` and for the update :math:`U_t`. This is in contrast to Shazeer, Noam, and Mitchell Stern and other frameworks which apply :math:`\epsilon_1` to both row and column factors of the squared gradient, but not in the calculations after: .. math:: \begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{1^\top_n \cdot R_t} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{\sqrt{\widehat{V}_t}} \\ \end{aligned} .. _Adafactor\: Adaptive Learning Rates with Sublinear Memory Cost: https://arxiv.org/pdf/1804.04235 .. _Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101 rrPrQrRrSrTrYrZrrrrr[r\rr]c||Jdtjjrt| tsJt |D]\}}|s||n|| }||}||}||}||}| )tj |jj} |dz }|j}|| z}t| d|dzz }t| |jdj|jdzz |z}| dk7r|jd| | zz |jdkDr||Jdtj|dd j!j#|j%d}|j'||tj|d d j!j#|j%d }|j'||||z}|j#|j)d d j+| n0|Jd ||z}|j'|||j-}|j+| | z j/}|j|td |jdj|jdz|zz } |j1|| | z y)N5Grad scaling should occur outside of optimizer.step()r ?rCrow_var and col_var should be defined when grad is multidimensionalr>TrIkeepdimr@)min0variance should be defined when grad is a vectorralpha)r5jit is_scriptingr r7 enumeratefinfor.ritemrqmaxnormr!mul_rIsquare_div_sizelerp_meanclamp_clonersqrt_add_)!rrPrQrRrSrTrYrZrrrrr[r\rr]iparamrDstep_tr?rArC step_floatone_minus_beta2_trho_trtrow_meancol_mean var_estimate grad_squaredupdatedenoms! r)_single_tensor_adafactorrFs0 y0?>?0 yy"e$$$f%215'uQxeAhYQ1+1+Q< <;;u{{+//D ! [[] & 3BZ_-.D%**Q-,,.%++-32FGH5P 1  JJq2 ,, - 88:>#(; UT U; 4R6>>@EEdiiPRmT  MM($5 6 4R6>>@EEdiiPRmT  MM($5 6"W,L   gllr4l@GGDGQ R$ BA B$$;L NN<): ;#>>+L$$$5<<> DCQ,,.6<<>S3HA2MNO 6%% 0e21r* tensorlistsreturnc2tj|}i}|jD]\\}}\}}||df}||df}t|dD]\}} | Jd| j dkDrN||vr|Dcgc]}gc}||<t t |D]} ||| j|| |!p||vr|Dcgc]}gc}||<t t |D]} ||| j|| |!|Scc}wcc}w)zGroups tensors by device, dtype, AND multidimensionality -- whether the tensor has multiple dims or just one dim (is a vector). This allows the foreach impl of Adafactor to assume that every group of params will either be factored or not.TFr zgrad should not be None)r"_group_tensors_by_device_and_dtypeitemsrwrIranger4rH) rgrouped_tensorsultra_grouped_tensorsdevicer._ matrix_key vector_keyjr8rs r)._group_tensors_by_device_dtype_and_is_multidimrsl BB;OO .=-B-B-DS))+qeT* eU+ #;q>2 SIAv% @'@ @%zz|a%::EP8Q8Q)*5s;/0SA)*5a8?? Aq@QRS%::EP8Q8Q)*5s;/0SA)*5a8?? Aq@QRS S S" ! 9R 9Rs 6 D Dc 2 t|dk(ry||Jdt||||||g}|jD]\\}}}\}}}}}}ttt |}ttt |}ttt |}| (|Jdt j|j} tr|dJ|rt j|}t jjs=|djr.t j|t jdddnt j|dg}g}g}|D]x} |j!| j#| z|j!d| j#| zz |j!t%| d| j#d zz zt'||D!"cgc]E\}!}"t)| |!j+d j#|!j-d zz |"zG}#}!}"| dk7rt j.|d| | zz |rttt |}$ttt |}%|$d|%dJd |D&cgc]}&t j*|&d d }'}&t j.|'|'t j0|'|D&cgc]}&|&j3d c}&t j.|$|t j.|'|t j|$|'~'|D&cgc]}&t j*|&dd }(}&t j.|(|(t j0|(|D&cgc]}&|&j3dc}&t j.|%|t j.|(|t j|%|(~(t'|$|%D)*cgc] \})}*|)|*z }+})}*|$D)cgc]})|)j5dd },})t j6|,| t j0|+|,~,nttt |}-|-dJdt j8||}.t j.|-|t j.|.|t j|-|.~.|-D/cgc]}/|/j;}+}/t j6|+| | zt j<|+t j>|+t j.|+||+}0t'|#|0D12cgc]I\}1}2|1 t)d|2j+d j#|2j-d z|zz z K}#}1}2t j.|0|#t j||0ycc}"}!wcc}&wcc}&wcc}&wcc}&wcc}*})wcc})wcc}/wcc}2}1w)Nrrkz2dtype is needed to compute eps1 when eps1 is unsetrcpu)rrsr rlrmrnr>Tror@rr) r4rrrrr r5rxrr _foreach_neg_utils is_compilingis_cpu _foreach_add_r8rHryrqziprzr{r! _foreach_mul_ _foreach_div_rr_foreach_clamp_min_ _foreach_mulr_foreach_sqrt__foreach_reciprocal_)3rrPrQrRrSrTrYrZrrrrr[r\rr]rrr. is_multidimdevice_params_ device_grads_device_row_vars_device_col_vars_device_variances_device_state_steps_ device_params device_gradsdevice_state_stepsone_minus_beta2_tsbeta2_tsrho_tssr:ralphasdevice_row_varsdevice_col_varsrD row_means col_meansr?rA var_estimates row_var_meansdevice_variances grads_squaredvupdatesars3 r)_multi_tensor_adafactorrs. 6{a y0?>?0E (I{CO    v4 E;      T&\>: DL-8 !$v,0CD <! DC D!;;u%))D %a(4 44  --l;L ||((*/A!/D/K/K   "ELLU$C3     2C 8# :A  % %affh+&= > OOAK 77 8 MM#b!qvvx3"78 9 :M62 1 affQinn&!'')s*:;    ;DP;? 4R6I    9 5    l+SdDIIbM+S T    :    += >    ;),O_(M$GW'!M CR7> T 2M  % %mT :    } =#DL2CD  #/ BA B/"..|\JM    0( ;    /A B    0- @1AA1QWWYAMA !!-= ]+ ""=1 M<8!1 6B#c6;;q>..0V\\^s5Ja4OPQ R   GV, M73mv4X ,T,T &B s8$A W)W/#W4 W9W> XX %X(AX)single_tensor_fnrc tjjstd|Ds t d|rt }nt }|||||||| | | | |||||| y)zxFunctional API that performs Adafactor algorithm computation. See :class:`~torch.optim.Adafactor` for details. c3PK|]}t|tj ywrc)r r5r ).0ts r) zadafactor..ss 3() 1ell#3s$&z?`state_steps` argument must contain a list of singleton tensors) rrrrr[r\rrYrZr]N)r5rrallrFrr)rrPrQrRrSrTrrYrZr]rrrrr[r\rfuncs r)rrXs6 << $ $ &s3-830 M  &'  !  !r*)NNNF)typingrrrrrrr r5r optimizerr r rrrr__all__r__doc__r7rgrrr.rrrr*r)rsKJJ   $X Xx.^ 8 9_KXEYQ lS1 LS1 <S18F#$S18F#$S1HV$%S1fS1 S1S1 S1 femS1 !S1"#S1$ 5/%S1& 'S1()S1*+S1l!!  (5<< (5;;"7 =>hv  "!DW4 LW4 <W48F#$W48F#$W4HV$%W4fW4 W4W4 W4 femW4 !W4"#W4$ 5/%W4& 'W4()W4*+W4t 1IJ##'"&7 L7 <78F#$78F#$ 7 HV$% 7 f 7d^7 777 7 eVm7 !7"#7$ %7& '7()7K7r*