DŽgDdZddlmZmZmZmZmZddlZddlmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZmZddgZGd deZd d ed e d ed e d ze_deedeedeedeedeededededededededefdZdeedeedeedeedeededededededededefdZe e d deedeedeedeedeedeededededededededefdZy)!z1Implementation for the Resilient backpropagation.)castListOptionalTupleUnionN)Tensor) _capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _maximize_doc_use_grad_for_differentiable _view_as_real OptimizerParamsTRproprpropceZdZ dddddddedeeefdeeefdeeefded e ed ed effd Z fd Z dZ e ddZxZS)rFN) capturableforeachmaximizedifferentiableparamslretas step_sizesrrrrc ,t|tr|jdk7r tdd|kstd|d|dcxkrdcxkr|dksntd|dd|dt ||||||| } t ||| y) Nr zTensor lr must be 1-elementgzInvalid learning rate: r?zInvalid eta values: z, )rrr rrrr) isinstancernumel ValueErrordictsuper__init__) selfrrrr rrrrdefaults __class__s Y/home/mcse/projects/flask_80/flask-venv/lib/python3.12/site-packages/torch/optim/rprop.pyr(zRprop.__init__s b& !bhhjAo:; ;by6rd;< <T!W,s,T!W,3DG9BtAwiHI I!)!  *c0t|||jD]}|jdd|jdd|jdd|jdd|dD]}|jj |g}t |dk7s.tj|drGt|d}|dr*tj|t|j ntj|t |d<y) NrrFrrrrstepdtypedevicer1) r' __setstate__ param_groups setdefaultstategetlentorch is_tensorfloattensorrr2)r)r7grouppp_statestep_valr+s r,r4zRprop.__setstate__;s U#&& E   Y -   Z /   -u 5   \5 18_ **..B/w<1$U__WV_-M$WV_5H !. $,=,?#\\(:K:MN FO   r-c Rd}|dD]}|j|tj|z}|j||j} | jr t d|j| |j |} t| dk(r|dr*tjdt|jntjdt| d <tj|tj | d <|jjr*tj| t|d |d | d <ntj| |d | d <|j| d |j| d |j| d |S)NFrz'Rprop does not support sparse gradientsrrr0r3r/ memory_formatprevr step_size)gradr: is_complexappend is_sparse RuntimeErrorr7r9zerosrr2 zeros_likepreserve_formatr1 full_likecomplex) r)r>rgradsprevsr state_steps has_complexr?rHr7s r, _init_groupzRprop._init_groupNsv x .Avv~ 5++A. .K MM! 66D~~"#LMM LL JJqME5zQ\*KK*;*=ahhOR/@/BCf !& 0 0%BWBW Xf 77%%*/geDk5;?*E+&*/uT{)KE+& LLv '   eK0 1   uV} -A .Dr-cb|jd}|$tj5|}ddd|jD][}g}g}g}g}g}|d\} } |d\} } |d} |d}|j ||||||}t |||||| | | | | ||d|d|]|S#1swYuxYw) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrr rrrr) step_size_min step_size_maxetaminusetaplusrrrrrU) _cuda_graph_capture_health_checkr: enable_gradr5rVr)r)closurelossr>rrRrSr rTrZr[rXrYrrrUs r,r/z Rprop.stepts  --/  ""$ !y !&& E#%F"$E"$E')J(*K %f Hg+0+> (M=I&GZ(H**vueZK ++!!$%56 .' ! B I ! !s B%%B.)g{Gz?)g?g333333?)gư>2N)__name__ __module__ __qualname__rrr<rrboolrr(r4rVrr/ __classcell__)r+s@r,rrs$($.*4 +!"&$++ %- +E5L! + %,' ++$+++<&$L"/"/r-a Implements the resilient backpropagation algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \theta_0 \in \mathbf{R}^d \text{ (params)},f(\theta) \text{ (objective)}, \\ &\hspace{13mm} \eta_{+/-} \text{ (etaplus, etaminus)}, \Gamma_{max/min} \text{ (step sizes)} \\ &\textbf{initialize} : g^0_{prev} \leftarrow 0, \: \eta_0 \leftarrow \text{lr (learning rate)} \\ &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \textbf{for} \text{ } i = 0, 1, \ldots, d-1 \: \mathbf{do} \\ &\hspace{10mm} \textbf{if} \: g^i_{prev} g^i_t > 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{min}(\eta^i_{t-1} \eta_{+}, \Gamma_{max}) \\ &\hspace{10mm} \textbf{else if} \: g^i_{prev} g^i_t < 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{max}(\eta^i_{t-1} \eta_{-}, \Gamma_{min}) \\ &\hspace{15mm} g^i_t \leftarrow 0 \\ &\hspace{10mm} \textbf{else} \: \\ &\hspace{15mm} \eta^i_t \leftarrow \eta^i_{t-1} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \eta_t \mathrm{sign}(g_t) \\ &\hspace{5mm}g_{prev} \leftarrow g_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to the paper `A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm `_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) etas (Tuple[float, float], optional): pair of (etaminus, etaplus), that are multiplicative increase and decrease factors (default: (0.5, 1.2)) step_sizes (Tuple[float, float], optional): a pair of minimal and maximal allowed step sizes (default: (1e-6, 50)) z z rrRrSr rTrXrYrZr[rrrrUct|D]\} }|| }| s|n| }|| }|| }|| }tjjs\| rZt }|j j |j j k(r|j j |vs Jd|d|dz }tj|rTtj|}tj|}tj|}tj|}| r.|j|jj}n|j|j}| r|jtj|jd|||jtj|jd|||jtj|j!dd|n<|||jd<|||jd<d||j!d<|j#|j%|||jtj&}| r6|jtj|j!|d|nd||j!|<|j)|j|d|j|y)NIIf capturable=True, params and state_steps must be on supported devices: .r rrDvalue) enumerater:_utils is_compilingrr2typerI view_as_realmulclonesigncopy_wheregtlteqmul_clamp_rOaddcmul_)rrRrSr rTrXrYrZr[rrrrUiparamrHrFrGr/capturable_supported_devicesrts r,_single_tensor_rproprsx f%15Qx#t$QxqM 1~||((*z+L+N ( !!T[[%5%55LL%%)EE {[[wZxxyz {F     E "%%d+D%%d+D&&u-E**95I 88DJJL)..0D88D>&&(D  JJu{{4771:w= > JJu{{4771:x> ? JJu{{4771:q$7 8&D 'D  D  t##M=Azz(=(=z>  JJu{{4778#4a> ?&'D" # tyy{IR8 4c1r-c t|dk(ry| rJdtjjs5| r3t t fdt ||Ds Jddtj|||||g} | jD]%\\}}}}}}ttt|}ttt|}ttt|}ttt|}ttt|}tjjs=|djr.tj|tjddd ntj|d | rt!||||tj"||}| rtj$|tj&||| rtj$||}tj(|| r|D]}|j+tj,|j/d|||j+tj,|j1d|||j+tj,|j3dd |nC|D]>}|||j/d<|||j1d<d ||j3d<@tj4|||D]}|j7||t9|}t;t|D]@}||j+tj,||j3|d||B~|Dcgc]}|j=}}tj>|||d (ycc}w) Nrz#_foreach ops don't support autogradc3K|]N\}}|jj|jjk(xr|jjvPywra)r2rp).0r?r/rs r, z&_multi_tensor_rprop..:sQ 4 HHMMT[[-- - > !== > sAArhrir"cpu)r2)alphar rjrk) r9r:rnrorallzipr"_group_tensors_by_device_and_dtypevaluesrrris_cpu _foreach_add_r=r _foreach_mul _foreach_neg__foreach_copy__foreach_sign_rurvrwrxry _foreach_mul_r{listrangert_foreach_addcmul_) rrRrSr rTrXrYrZr[rrrrUgrouped_tensorsgrouped_params_grouped_grads_grouped_prevs_grouped_step_sizes_grouped_state_steps__grouped_params grouped_grads grouped_prevsgrouped_step_sizesgrouped_state_stepssignsrtrGr}rH grad_signsrs @r,_multi_tensor_rpropr"sw  6{aDDD  << $ $ &:'H'J$ v{3   wWWsVttu v  w  BB z;7O  " " $ J   d6lO<T&\>: T&\>: !$v,0CD"4<1EF ||((*/B1/E/L/L   #U\\#e%DC     3Q 7   }>P ""=-@     & ]M:     .%  U#  = 5;;twwqz7DAB 5;;twwqz8TBC 5;;twwqz1d;< =  %#*TWWQZ #+TWWQZ #$TWWQZ  % .6+ ;I   ]M : ; ]+ s=)* A !  " " E!HKK11mA6FG   /<.s 3() 1ell#3s$&zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizers)rXrYrZr[rrrrU) r:rnrorrLr jit is_scriptingrr)rrRrSr rTrrrrrUrXrYrZr[rfuncs r,rrs4 << $ $ &s3-830 ^  1 Ne 7599))+STTuyy--/"#  ##%r-)NFFFF)__doc__typingrrrrrr:r optimizerr r r r rrrrrrrr__all__rr<rerrrrCr-r,rs955      G HIHX"F     G2 nA LA <A <AV A f AAAAAAAAAHk Lk <k  <k V k f k k k k k k k k k d 1EF# ; L; <; <;V ; f ;d^;;;;;;; !;"#;G;r-