DŽg UdZddlZddlZddlmZmZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZddlmZmZmZddlZddlmcm Z ddl!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'm(Z(dd lm)Z)ee d fZ*ee+d <ee,e fZ-ee+d <ee,e fZ.ee+d<eeej^ej`fZ1e de*e-geee*e-ffZ2ee+d<e de*e-gdfZ3ee+d<gdZ4eZ5ee6e2fe+d<eZ7ee6e3fe+d<ej`ejpjrjtgZ;GddZ<e<Z=dZ>dZ?dZ@d4dZA d5deej`deBdeBdeeBeBffdZC d5d ej`d!eBddfd"ZDd#ZEd4d$ZFd6d%eBdee,fd&ZGd'ZHd(ZId)ZJd*ZKd+ZLd,e2de)fd-ZMd,e3de)fd.ZNeeej`eee,e ffZOee+d/<ed0ZPed1ZQed2ZRGd3dZSy)7zBase optimizer.N) defaultdict OrderedDict)deepcopy)chain)AnyCallablecast DefaultDictDictHashableIterableListOptionaloverloadSetTupleTypeVarUnion) ParamSpecSelf TypeAlias) is_compiling)&_get_foreach_kernels_supported_devices$_get_fused_kernels_supported_devices"_group_tensors_by_device_and_dtypeIndicesTensorListList)RemovableHandle.ArgsKwargs StateDict OptimizerGlobalOptimizerPreHookGlobalOptimizerPostHook)r" register_optimizer_step_pre_hook!register_optimizer_step_post_hook_global_optimizer_pre_hooks_global_optimizer_post_hooksceZdZdZdefdZy)_RequiredParameterzCSingleton class representing a required parameter for an Optimizer.returncy)Nzselfs ]/home/mcse/projects/flask_80/flask-venv/lib/python3.12/site-packages/torch/optim/optimizer.py__repr__z_RequiredParameter.__repr__@s%N)__name__ __module__ __qualname____doc__strr1r-r2r0r*r*=sM&#&r2r*c>fd}tj||S)Ncxddl}|j} |j|jd|jj |g|i|}|jj |j||S#|jj |j|wxYw)Nrdifferentiable) torch._dynamois_grad_enabledset_grad_enableddefaults_dynamo graph_break)r/argskwargstorch prev_gradretfuncs r0 _use_gradz/_use_grad_for_differentiable.._use_gradHs)E))+  . #E " "4==1A#B C MM % % 't-d-f-C MM % % ' "E " "9 -  MM % % ' "E " "9 -s AB .B9) functoolsupdate_wrapper)rFrGs` r0_use_grad_for_differentiablerJGs 2Y- r2ctjjs tr|St |tj r|j S|SN)rCjit is_scriptingr isinstanceTensoritemxs r0 _get_valuerTes9 99 ! ! # %a6qvvx=A=r2ctjjstrtj|S|SrL)rCrMrNrstackrRs r0_stack_if_compilingrWms) 99 ! ! # {{1~r2c>|r|t|j<d}|S)Nc.ddl}tj|jj}d t |j jdtjfd}|S#t$rdY/wxYw)NrT state_stepsFctrN|jddsr|r|djsd|vr|dr|ddjr|i|S|i|S)N capturableFrrZ)rgetis_cuda)rArB disabled_funcrFhas_state_stepsstate_steps_inds r0maybe_fallbackzG_disable_dynamo_if_unsupported..wrapper..maybe_fallbacksx~JJ|U3#/*tO/DQ/G/O/O!V+}-}-a088%d5f55T,V,,r2) inspectrC_disable_dynamo signature parameterslistkeysindex ValueErrorrHwraps)rFrcpsrbr_r`ras` @@@r0wrapperz/_disable_dynamo_if_unsupported..wrapper|s--d3   t $ / / $"2779o33MBO   -  -1 $#O $s(B BB)globalsr3)single_tensor_fnrms r0_disable_dynamo_if_unsupportedrpts' /? "++, D Nr2paramsr: use_fusedr+ctjjs|rytt |xrt fd|D}| xrt fd|D}||fS)N)FFc3K|]L}|duxsBt|tvxr/|jjvxrtj|NywrL)type_foreach_supported_typesdevicerCis_floating_point).0pfused_supported_devicess r0 z/_default_to_fused_or_foreach..s_  T  G/ / + !88 +''*  sAAc3K|]5}|duxs+t|tvxr|jjv7ywrL)rurvrw)ryrzforeach_supported_devicess r0r|z/_default_to_fused_or_foreach..sL   T  G/ / ; !::  s;>)rCrMrNrrall)rqr:rrfusedforeachr~r{s @@r0_default_to_fused_or_foreachrs| yy>BD F H  #EiC    G '>r2rzcuda_unsupportedct}|r|jd|jj|vrt j |s2t d|d|jd|jjy)NcudazX`fused=True` requires all the params to be floating point Tensors of supported devices: z but z and )rremoverwrurCrx RuntimeErrordtype)rzrr{s r0_device_dtype_check_for_fusedrszCD&&v. HHMM4 49P9PQR9S ""9!:%yahhmm_ ^  :Tr2ct|D]X\}}tj|stj||||<|D]}tj||||<ZyrL) enumeraterC is_complex view_as_real)rqstate_and_gradsirzss r0 _view_as_realrsg&!01   A **6!95F1I$ 0))!A$/! 00r2c|rtjStjtjk(rtjStjSrL)rCfloat32get_default_dtypefloat64)is_fuseds r0_get_scalar_dtypers;}}002emmC INr2 supports_xlacgd}tjjs-|jtjj |r|jd|S)z?Return the device type list that supports capturable optimizer.)rxpuhpuxla)rCrMrNappend_C_get_privateuse1_backend_name)rcapturable_supported_devicess r0!_get_capturable_supported_devicesrsJ#9 99 ! ! #$++EHH,R,R,TU$++E2 ''r2acforeach (bool, optional): whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to the intermediates being a tensorlist vs just one tensor. If memory is prohibitive, batch fewer parameters through the optimizer at a time or switch this flag to False (default: None)afused (bool, optional): whether the fused implementation is used. Currently, `torch.float64`, `torch.float32`, `torch.float16`, and `torch.bfloat16` are supported. (default: None) .. note:: The foreach and fused implementations are typically faster than the for-loop, single-tensor implementation, with fused being theoretically fastest with both vertical and horizontal fusion. As such, if the user has not specified either flag (i.e., when foreach = fused = None), we will attempt defaulting to the foreach implementation when the tensors are all on CUDA. Why not fused? Since the fused implementation is relatively new, we want to give it sufficient bake-in time. To specify fused, pass True for fused. To force running the for-loop implementation, pass False for either foreach or fused. acapturable (bool, optional): whether this instance is safe to capture in a CUDA graph. Passing True can impair ungraphed performance, so if you don't intend to graph capture this instance, leave it False (default: False)a]differentiable (bool, optional): whether autograd should occur through the optimizer step in training. Otherwise, the step() function runs in a torch.no_grad() context. Setting to True can impair performance, so leave it False if you don't intend to run autograd through this instance (default: False)zmaximize (bool, optional): maximize the objective with respect to the params, instead of minimizing (default: False)hookc^tjt}|t|j<|S)aRegister a pre hook common to all optimizers. The hook should have the following signature:: hook(optimizer, args, kwargs) -> None or modified args and kwargs Args: hook (Callable): A user defined hook which is registered on all optimizers. Returns: :class:`torch.utils.hooks.RemovableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )hooksrr'idrhandles r0r%r%s) " "#> ?F-1 * Mr2c^tjt}|t|j<|S)aRegister a post hook common to all optimizers. The hook should have the following signature:: hook(optimizer, args, kwargs) -> None Args: hook (Callable): A user defined hook which is registered on all optimizers. Returns: :class:`torch.utils.hooks.RemovableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )rrr(rrs r0r&r&#s) " "#? @F.2 + Mr2ParamsT_PRTc4eZdZUdZeeeegee eeffZ e e d<eeeegdfZ e e d<eee fe d<eee fe d<de d<d e d <d e d <de d <d edeeefddfdZdeeeffdZdeeefddfdZdefdZd7dZd7dZedeeefdeeeffdZe d8dededeee de ee ffee e!jDe!jFfe ee ffffdZ$d7dZ%de de&fdZ'de de&fd Z( d8dedgdfd!ede&fd"Z) d8dede*gee*fd!ede&fd#Z+e!jXde*fd$Z-e d9d%e!j\d&e!j\d'ed(e/eeefd)e0de!j\f d*Z1 d8dede*gee*fd!ede&fd+Z2 d8dedgdfd!ede&fd,Z3e!jXd-e*ddfd.Z4e!jXd:d/eddfd0Z5e6d;dr+ctjjd||_t |_t |_t |_t |_t |_ t |_ |jt|tjr!tdtj|zt!t"|_g|_t)|}t+|dk(r t-dt|dt"sd|ig}|D]!}|j/t1t"|#d|_y)Nzpython.optimizerzZparams argument given to the optimizer should be an iterable of Tensors or dicts, but got rz%optimizer got an empty parameter listrqT)rCr_log_api_usage_oncer>rrrrrrr_patch_step_functionrOrP TypeErrortypenamerdictstate param_groupsrglenrjadd_param_groupr $_warned_capturable_if_run_uncaptured)r/rqr>r param_groups r0__init__zOptimizer.__init__Ws $$%78  )4&*5-'/:},0; -4?M15@]2 !!# fell +<>CnnV>TU  6A5F 24F| |  !DE E,q/40%|45L' :K  dK!8 9 : 591r2cJ|j|j|jdS)Nr>rrrr.s r0 __getstate__zOptimizer.__getstate__zs# ZZ --  r2rc|jj|d|jvrt|_d|jvrt|_d|jvrt|_d|jvrt|_d|jvrt|_d|jvrt|_|j|jjddy) Nrrrrrrr:F) __dict__updaterrrrrrrrr> setdefault)r/rs r0 __setstate__zOptimizer.__setstate__s U# &dmm ;-8]D * 't}} <.9mD + ,DMM A3>=D 0 -T]] B4?MD 1 1 F8C D 5 2$-- G9DD 6 !!#   !159r2c|jjdz}t|jD]F\}}|dz }|d|dz }t |j D]}|dk7s |d|d||dz }H|dz }|S)Nz ( zParameter Group rqz z: )) __class__r3rrsortedrh)r/ format_stringrgroupkeys r0r1zOptimizer.__repr__s//$6 !$"3"34 BHAu T !M /s"5 5Mejjl+ B(?!tC55:,b%AAM B B  r2ctstjjj rtjj rtjj }|rAtd|jDs%td|jjzdzt|dds=td|jDr |stjdd|_yyyyyyy) Nc3&K|] }|d ywr\Nr-ryrs r0r|z=Optimizer._cuda_graph_capture_health_check..s%(-l#%z;Attempting CUDA graph capture of step() for an instance of z' but param_groups' capturable is False.rFc3&K|] }|d ywrr-rs r0r|z=Optimizer._cuda_graph_capture_health_check..sKl+Kra This instance was constructed with capturable=True or some of all the param_groups came with capturable=True, but step() is running without CUDA graph capture. If you never intend to graph-capture this instance, capturable=True can impair performance, and you should set capturable=False.T)rrCbackendsris_built is_availableis_current_stream_capturingrrrrr3getattrwarningswarnr)r/ capturings r0 _cuda_graph_capture_health_checkz*Optimizer._cuda_graph_capture_health_checks##,,. '') >>@I%151B1B%"#Qnn--.?@T#I5QK9J9JKK" m =A9#LR*/r2cy)aEntry point for `torch.profile.profiler`. When python tracing is enabled the profiler will hook into this function at the CPython level to inspect the optimizer's parameters and param groups. It is called it after `step()` since many optimizers lazily initialize state. This is a workaround due to lack of a proper step hook on the optimizer, and will be removed if it exists. Nr-r.s r0_optimizer_step_codezOptimizer._optimizer_step_codesr2rFctjdtjdtjdt ffd }|S)NrArBr+c|^}}tt|}d|jjd}tj j j|5ttj|jjD]C}||||}|t|trt|dk(r|\}}4t d|d |i|}|j!t|j"jt$jD] }|||||cdddS#1swYyxYw)NzOptimizer.step#z.stepz@ must return None or a tuple of (new_args, new_kwargs), but got .)r r"rr3rCautogradprofilerrecord_functionrr'valuesrrOtuplerrrrr() rArBr/_ profile_namepre_hookresultout post_hookrFs r0rmz,Optimizer.profile_hook_step..wrappersIHD1 4(D,T^^-D-D,EUKL((88F  %/6682299;! H&dD&9F)%fe4V9I+1LD&".#'&(hiohppq r# D+F+))+"'33::<0779"2IdD&1 2 3   sAD> BD>>E)rHrkrrArBr)rFrms` r0profile_hook_stepzOptimizer.profile_hook_stepsB   277 bii A   >r2tensorlistlist with_indicesNNc vtr$d|ttt|dfiSt ||S)zGroup a list of lists of tensors by device and dtype. Skips this step if we are compiling since this will occur during inductor lowering. rr)rrgrangerr)rrs r0rz,Optimizer._group_tensors_by_device_and_dtypes; > >4c.QRBS>T8U3V"WX X5nlS Sr2c(d|jjd|_t|jjdd}|sP|j |jj|j_d|jj_yy)NzOptimizer.zero_grad#z .zero_gradhookedT)rr3_zero_grad_profile_namersteprr)r/rs r0rzOptimizer._patch_step_functionsv"4>>#:#:";: F $,,h="&"8"89L9L"MDNN )-DNN   &r2rcvtj|j}||j|j<|S)aRegister an optimizer step pre hook which will be called before optimizer step. It should have the following signature:: hook(optimizer, args, kwargs) -> None or modified args and kwargs The ``optimizer`` argument is the optimizer instance being used. If args and kwargs are modified by the pre-hook, then the transformed values are returned as a tuple containing the new_args and new_kwargs. Args: hook (Callable): The user defined hook to be registered. Returns: :class:`torch.utils.hooks.RemovableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )rrrrr/rrs r0register_step_pre_hookz Optimizer.register_step_pre_hooks3&&&t'E'EF48&&vyy1 r2cvtj|j}||j|j<|S)aRegister an optimizer step post hook which will be called after optimizer step. It should have the following signature:: hook(optimizer, args, kwargs) -> None The ``optimizer`` argument is the optimizer instance being used. Args: hook (Callable): The user defined hook to be registered. Returns: :class:`torch.utils.hooks.RemovableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )rrrrrs r0register_step_post_hookz!Optimizer.register_step_post_hook&s3"&&t'F'FG59'' 2 r2prependctj|j}||j|j<|r'|jj |jd|S)a&Register a state dict pre-hook which will be called before :meth:`~torch.optim.Optimizer.state_dict` is called. It should have the following signature:: hook(optimizer) -> None The ``optimizer`` argument is the optimizer instance being used. The hook will be called with argument ``self`` before calling ``state_dict`` on ``self``. The registered hook can be used to perform pre-processing before the ``state_dict`` call is made. Args: hook (Callable): The user defined hook to be registered. prepend (bool): If True, the provided pre ``hook`` will be fired before all the already registered pre-hooks on ``state_dict``. Otherwise, the provided ``hook`` will be fired after all the already registered pre-hooks. (default: False) Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` Flast)rrrr move_to_endr/rrrs r0register_state_dict_pre_hookz&Optimizer.register_state_dict_pre_hook;sS4&&t'K'KL:>,,VYY7   0 0 < state_dict or None The hook will be called with arguments ``self`` and ``state_dict`` after generating a ``state_dict`` on ``self``. The hook may modify the state_dict inplace or optionally return a new one. The registered hook can be used to perform post-processing on the ``state_dict`` before it is returned. Args: hook (Callable): The user defined hook to be registered. prepend (bool): If True, the provided post ``hook`` will be fired before all the already registered post-hooks on ``state_dict``. Otherwise, the provided ``hook`` will be fired after all the already registered post-hooks. (default: False) Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` Fr)rrrrrrs r0register_state_dict_post_hookz'Optimizer.register_state_dict_post_hook[sS8&&t'L'LM;?--fii8   1 1 = =fiie = T r2c |jjD] }|| i d dtttfdtttff fd }|j Dcgc] }|| }}|j jDcic]0\}}t|tjr t|n||2}}}||d}|jjD]} | ||} | | }|Scc}wcc}}w)aReturn the state of the optimizer as a :class:`dict`. It contains two entries: * ``state``: a Dict holding current optimization state. Its content differs between optimizer classes, but some common characteristics hold. For example, state is saved per parameter, and the parameter itself is NOT saved. ``state`` is a Dictionary mapping parameter ids to a Dict with state corresponding to each parameter. * ``param_groups``: a List containing all parameter groups where each parameter group is a Dict. Each parameter group contains metadata specific to the optimizer, such as learning rate and weight decay, as well as a List of parameter IDs of the parameters in the group. NOTE: The parameter IDs may look like indices but they are just IDs associating state with param_group. When loading from a state_dict, the optimizer will zip the param_group ``params`` (int IDs) and the optimizer ``param_groups`` (actual ``nn.Parameter`` s) in order to match state WITHOUT additional verification. A returned state dict might look something like: .. code-block:: text { 'state': { 0: {'momentum_buffer': tensor(...), ...}, 1: {'momentum_buffer': tensor(...), ...}, 2: {'momentum_buffer': tensor(...), ...}, 3: {'momentum_buffer': tensor(...), ...} }, 'param_groups': [ { 'lr': 0.01, 'weight_decay': 0, ... 'params': [0] }, { 'lr': 0.001, 'weight_decay': 0.5, ... 'params': [1, 2, 3] } ] } rrr+c r|jDcic]\}}|dk7s ||}}}jt|dDcic]\}}t|vr t|| c}}|dDcgc]}t|c}|d<t |dz |Scc}}wcc}}wcc}wNrq)itemsrrrr)rkvpackedrrzparam_mappings start_indexs r0 pack_groupz(Optimizer.state_dict..pack_groups',{{}Ftq!X adFFF  ! !!*%/; G1!uN2qE1H @EXO!r!u 5OF8  3vh/0 0KMG Ps B(B(#B. :B4rr) rrr r7rrrrrOrCrPrr) r/rrgrrr packed_state state_dictr hook_resultrrs @@r0rzOptimizer.state_dict}sd<<CCE H TN *,  d38n c3h 04/@/@A! 1 A A ((* 1'1ELL&A^BqE "q1 L  "( >>EEG )I#D*5K&(  )!B s )D5Dparamvalueparam_idrrcpd}d}|J|D]!}||dvs d|vr|dnd}d|vr|dnd}n|dk(r1|s|r+|jtj|jS|S|j r'|j|j |jS|j|jS)NFrqrr\r)rrw)rw)torCrrwrxr)rrrrrrr\pgs r0(_process_value_according_to_param_policyz2Optimizer._process_value_according_to_param_policys ''' B2h<''."}7 %1=1CR -    &=UxxemmELLxII &&(xxekk%,,xGGxxu||x44r2ctj|j}||j|j<|r'|jj |jd|S)aRegister a load_state_dict pre-hook which will be called before :meth:`~torch.optim.Optimizer.load_state_dict` is called. It should have the following signature:: hook(optimizer, state_dict) -> state_dict or None The ``optimizer`` argument is the optimizer instance being used and the ``state_dict`` argument is a shallow copy of the ``state_dict`` the user passed in to ``load_state_dict``. The hook may modify the state_dict inplace or optionally return a new one. If a state_dict is returned, it will be used to be loaded into the optimizer. The hook will be called with argument ``self`` and ``state_dict`` before calling ``load_state_dict`` on ``self``. The registered hook can be used to perform pre-processing before the ``load_state_dict`` call is made. Args: hook (Callable): The user defined hook to be registered. prepend (bool): If True, the provided pre ``hook`` will be fired before all the already registered pre-hooks on ``load_state_dict``. Otherwise, the provided ``hook`` will be fired after all the already registered pre-hooks. (default: False) Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` Fr)rrrrrrs r0!register_load_state_dict_pre_hookz+Optimizer.register_load_state_dict_pre_hooksUB&&t'P'PQ?C11&))<   5 5 A A&))RW A X r2ctj|j}||j|j<|r'|jj |jd|S)a^Register a load_state_dict post-hook which will be called after :meth:`~torch.optim.Optimizer.load_state_dict` is called. It should have the following signature:: hook(optimizer) -> None The ``optimizer`` argument is the optimizer instance being used. The hook will be called with argument ``self`` after calling ``load_state_dict`` on ``self``. The registered hook can be used to perform post-processing after ``load_state_dict`` has loaded the ``state_dict``. Args: hook (Callable): The user defined hook to be registered. prepend (bool): If True, the provided post ``hook`` will be fired before all the already registered post-hooks on ``load_state_dict``. Otherwise, the provided ``hook`` will be fired after all the already registered post-hooks. (default: False) Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` Fr)rrrrrrs r0"register_load_state_dict_post_hookz,Optimizer.register_load_state_dict_post_hooksT8&&t'Q'QR@D22699=   6 6 B B699SX B Y r2rc |j}|jjD]}|||}||}|j}t |d}t |t |k7r t dd|D}d|D}tdt||Dr t dtttjd|Dtjd |D}dfd tt} |d jD]&\} } | |vr|| } | | | |d | | <"| | | <(d ttt fdttt fdttt ffd} t||Dcgc]\}}| ||}}}|j#| |d|j$jD] }|| ycc}}w)zLoad the optimizer state. Args: state_dict (dict): optimizer state. Should be an object returned from a call to :meth:`state_dict`. Nrz.Xs71c!H+&7c38K|]}t|dywr'r(r)s r0r|z,Optimizer.load_state_dict..Ys=1c!H+&=r*c3,K|] \}}||k7ywrLr-)ryp_lens_lens r0r|z,Optimizer.load_state_dict..ZsN,%u~Nsz]loaded state dict contains a parameter group that doesn't match the size of optimizer's groupc3&K|] }|d ywr'r-r)s r0r|z,Optimizer.load_state_dict..cs#FAAhK#Frc3&K|] }|d ywr'r-r)s r0r|z,Optimizer.load_state_dict..ds#@AAhK#@rc Xt|tjrtj ||St|t r/|j Dcic]\}}|||c}}St|trt|fd|DS|Scc}}w)zBMake a deep copy of value, casting all tensors to device of param.)rrrc36K|]}|yw)rrNr-)ryr_castrrrs r0r|z;Optimizer.load_state_dict.._cast..vs!"pef5HS_#`#`"ps) rOrCrPr"r rrr ru)rrrrrrrr4s` `` r0r4z(Optimizer.load_state_dict.._casths%. II5(L#E4( !& 1uq8,TU E8,"tE{"pjo"ppp sB&rr3r new_groupr+c|d|d<|Sr r-)rr5s r0 update_groupz/Optimizer.load_state_dict..update_groups#(/Ih  r2r)NNN)copyrrrrrrjanyziprr from_iterablerrr r7rrr)r/rrrgroups saved_groups param_lens saved_lensid_maprrrrr7rngrrr4s @r0load_state_dictzOptimizer.load_state_dict>s __& AAHHJ )H"44K&(  ) "" > :; v;#l+ +Q 87 = = N#j*2MN NC   ###F#FF###@#@@   * .238n  #s(^  :=V\9RS2 Q+S S E<HICCJJL I dO TsG, set_to_nonec|jjddxs|jjdd}t|ds|j|r t d}nd}t j jj|j5|jD]}|dD]}|j|rd|_ |jj|jjn|jjd|r|jjr|jj!|J||jj"|jj$j'|j|rC|J|j)D],}|j)D]}t j*|.dddy#1swYyxYw)aReset the gradients of all optimized :class:`torch.Tensor` s. Args: set_to_none (bool): instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\ s are guaranteed to be None for params that did not receive a gradient. 3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether). rFrrc ttSrL)rrgr-r2r0z%Optimizer.zero_grad..s [=Nr2Nrq)r>r]hasattrrrrCrrrrrgradgrad_fndetach_requires_grad_ is_sparsezero_rwrrr_foreach_zero_)r/rCrper_device_and_dtype_gradsrrzper_dtype_gradsgradss r0 zero_gradzOptimizer.zero_grads --##Iu5 9J9J U: t67  % % ' )45N)O &)- & ^^ $ $ 4 4T5Q5Q R 4** 1x1Avv)&%)AF vv~~9 ! 0 ! 5 5e <#*aff.>.> ! 'A'M M'M :166== I$%FFLL!""(&.1 1"1==='A'H'H'J4O!0!7!7!94,,U344) 4 4 4s$G(DG((G1closurecyrLr-r/rSs r0rzOptimizer.step r2cyrLr-rUs r0rzOptimizer.steprVr2ct)afPerform a single optimization step to update parameter. Args: closure (Callable): A closure that reevaluates the model and returns the loss. Optional for most optimizers. .. note:: Unless otherwise specified, this function should not modify the ``.grad`` field of the parameters. )NotImplementedErrorrUs r0rzOptimizer.steps "!r2rct|tstdt||d}t|tj r|g|d<n)t|t r tdt||d<|dD]~}t|tj s!tdt j|z|jjddr[|jrh|jrutd|jjD]1\}}|tur||vrtd||j!||3|d}t#|t#t |k7rt%j&d d t }|j(D]}|j+t |d!|j-t |ds td |j(j/|y) aAdd a param group to the :class:`Optimizer` s `param_groups`. This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the :class:`Optimizer` as training progresses. Args: param_group (dict): Specifies what Tensors should be optimized along with group specific optimization options. z$param_group must be a dict, but got rqzoptimizer parameters need to be organized in ordered collections, but the ordering of tensors in sets will change between runs. Please use a list instead.z>optimizer can only optimize Tensors, but one of the params is r:Nz can't optimize a non-leaf TensorzJparameter group didn't specify a value of required optimization parameter zoptimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information) stacklevelz7some parameters appear in more than one parameter group)rOrrrurCrPsetrgrr>r]is_leaf retains_gradrjrrequiredrrrrrr isdisjointr)r/rrqrnamedefault param_setrs r0rzOptimizer.add_param_groups+t,B4 CTBUVW WX& fell +%+HK !  $g  %)LK ! * EEeU\\2027..2GH==$$%5t< !3!3 !CDD E"]]002 6MD'("t;'> `ae`fg&&tW5  6X& v;#c&k* * MMS  (+u && 3E   Sx1 2 3##C H(=$>?VW W   -r2)r+NFrLT).)rSNr+N):r3r4r5r6rrrr rrrr__annotations__rr intrr7rrrrr1rr staticmethodrrrrboolrrrCrwrrrrrrrr!r rdrrPrr r r"r$rBrRrrfloatrr-r2r0r"r">s0 #+D$+?%PTV\P\J]A^+^"_i_#+T4,@$,F#GyG#C)9$9:: $S*;%; <<%VV&qq*uu+\\!9w!9$sCx.!9T!9F d38n :$sCx.:T:" # %AN  !Q!HRUO!!F#T&TT  U: ng&= > >? U5<<, -u^W5L/M MN P TT .+;.,=/,DIk]D01<@ F  Y/)1DDE    D VIVVp  5||5||554S>* 5  5  55B% Y/)1DDE%%  %PDI k]D01 <@   D S)SSSj 44T44T4444l   HRY/ E   "HXb%i%89 "Xe_ " ;.4S>;.d;.;.r2rLrerf)Tr6rHr collectionsrrr8r itertoolsrtypingrrr r r r r rrrrrrrtyping_extensionsrrrrCtorch.utils.hooksutilsr torch._utilsrtorch.utils._foreach_utilsrrrrrrrrgr7r r!rwrP DeviceDictr#r$__all__r'rhr(nn parameter Parameterrvr*r`rJrTrWrprjrrrrr _foreach_doc _fused_doc_capturable_doc_differentiable_doc _maximize_docr%r&rrrrr"r-r2r0r~s0 98 !!%.S/i!cN "CH~ 9% (5<<(%,,6 7 %-$%f *=!>>% &.{D&.I4.O%PP  BMT#'="=>NCN=d3(?#?@P!LL%((*<*<*F*FG&&  <>*hIN  04AE 4::/4   ||  '+     0(D(DI(g  J   6 > +Ao(,C(8ELL18DcN3KKLLt_ CL CL^ .^ .r2