DŽgh^vdZddlZddlZddlmZddlZddlmZd'dZd'dZ d'dZ dZ d Z d'd Z d(d ed ed edeejdef dZ d(d edededeejdef dZ d)d ededed ed edeejdefdZd ededefdZd edefdZd edefdZdZd*dZdZ d+d ededeejdefdZ d+d ededeejdefdZdZ d,d ed ed ed!edeejf d"Z d,d ed ed ed!edeejf d#Z d-deejfd$Z d.deejfd%Z d&Z!e!eZ"e!eZ#e!eZ$e!eZ%e!eZ&e!eZ'e!eZ(e!eZ)e!eZ*e!eZ+e!e Z,y)/zHThis file contains utilities for initializing neural network parameters.N)Optional)Tensorc~tj5|j|||cdddS#1swYyxYwN generator)torchno_graduniform_tensorabrs U/home/mcse/projects/flask_80/flask-venv/lib/python3.12/site-packages/torch/nn/init.py_no_grad_uniform_rs2 :q!y9:::3<c~tj5|j|||cdddS#1swYyxYwr)r r normal_r meanstdrs r_no_grad_normal_rs2 >~~dC9~=>>>rcd}||d|zz ks ||d|zzkDrtjddtj5|||z |z }|||z |z }|j d|zdz d|zdz ||j |j |tjdz|j||j|||cdddS#1swYyxYw) Ncddtj|tjdz zdz S)N?@)matherfsqrt)xs rnorm_cdfz(_no_grad_trunc_normal_..norm_cdfs(dhhq499S>122c99zjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect. stacklevelrr)minmax) warningswarnr r r erfinv_mul_rradd_clamp_) r rrrrrr!lus r_no_grad_trunc_normal_r1s: q1s7{q1s7{ 2  ;  a$h#% & a$h#% & A 1q519 B   C$))C.() D  ! #+s BC((C1cxtj5|j|cdddS#1swYyxYwN)r r fill_r vals r_no_grad_fill_r7>s, !||C !!!s09cvtj5|jcdddS#1swYyxYwr3)r r zero_r s r_no_grad_zero_r;Cs) ||~s/8c\gd}||vs|dk(ry|dk(ry|dk(rtjdS|dk(re|d }nBt|tst|tst|t r|}nt d |d tjdd|d zzz S|d k(r yt d|)aReturn the recommended gain value for the given nonlinearity function. The values are as follows: ================= ==================================================== nonlinearity gain ================= ==================================================== Linear / Identity :math:`1` Conv{1,2,3}D :math:`1` Sigmoid :math:`1` Tanh :math:`\frac{5}{3}` ReLU :math:`\sqrt{2}` Leaky Relu :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}` SELU :math:`\frac{3}{4}` ================= ==================================================== .. warning:: In order to implement `Self-Normalizing Neural Networks`_ , you should use ``nonlinearity='linear'`` instead of ``nonlinearity='selu'``. This gives the initial weights a variance of ``1 / N``, which is necessary to induce a stable fixed point in the forward pass. In contrast, the default gain for ``SELU`` sacrifices the normalization effect for more stable gradient flow in rectangular layers. Args: nonlinearity: the non-linear function (`nn.functional` name) param: optional parameter for the non-linear function Examples: >>> gain = nn.init.calculate_gain('leaky_relu', 0.2) # leaky_relu with negative_slope=0.2 .. _Self-Normalizing Neural Networks: https://papers.nips.cc/paper/2017/hash/5d44ee6f2c3f71b73125876103c8f6c4-Abstract.html )linearconv1dconv2dconv3dconv_transpose1dconv_transpose2dconv_transpose3dsigmoidr&tanhg?relur leaky_relu{Gz?znegative_slope z not a valid numberr#selug?zUnsupported nonlinearity )rr isinstanceboolintfloat ValueError) nonlinearityparam linear_fnsnegative_slopes rcalculate_gainrSHsDJz!\Y%>    yy~  % =!N5$'5#&%'#Nug5HIJ JyyNA$5 5677    4\NCDDr"r rrrreturnctjj|r*tjjt|f||||St ||||S)aFill the input Tensor with values drawn from the uniform distribution. :math:`\mathcal{U}(a, b)`. Args: tensor: an n-dimensional `torch.Tensor` a: the lower bound of the uniform distribution b: the upper bound of the uniform distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.uniform_(w) r )r overrideshas_torch_function_variadichandle_torch_functionr rr s rr r sT( 226:44 vi!qI5   VQ9 55r"rrctjj|r*tjjt|f||||St ||||S)aFill the input Tensor with values drawn from the normal distribution. :math:`\mathcal{N}(\text{mean}, \text{std}^2)`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.normal_(w) r)r rVrWrXrrrs rrrsT( 226:44 fYvDcY5   FD#y 99r"c$t||||||S)aFill the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) r)r1)r rrrrrs r trunc_normal_r[s8 "&$QY OOr"r6ctjj|r(tjjt|f||St ||S)zFill the input Tensor with the value :math:`\text{val}`. Args: tensor: an n-dimensional `torch.Tensor` val: the value to fill the tensor with Examples: >>> w = torch.empty(3, 5) >>> nn.init.constant_(w, 0.3) r5)r rVrWrX constant_r7r5s rr]r]sL 226:44 yS5   &# &&r"ct|dS)zFill the input Tensor with the scalar value `1`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.ones_(w) r)r7r:s rones_r_s &# &&r"ct|S)zFill the input Tensor with the scalar value `0`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.zeros_(w) )r;r:s rzeros_ras & !!r"c|jdk7r tdtj5tj|j ||j dddd|S#1swY|SxYw)a=Fill the 2-dimensional input `Tensor` with the identity matrix. Preserves the identity of the inputs in `Linear` layers, where as many inputs are preserved as possible. Args: tensor: a 2-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.eye_(w) r#,Only tensors with 2 dimensions are supported)out requires_gradN) ndimensionrNr r eyeshaperer:s reye_risaaGHH Q 6<1, each group of channels preserves identity Args: tensor: a {3, 4, 5}-dimensional `torch.Tensor` groups (int, optional): number of groups in the conv layer (default: 1) Examples: >>> w = torch.empty(3, 16, 5, 5) >>> nn.init.dirac_(w) >>> w = torch.empty(3, 24, 5, 5) >>> nn.init.dirac_(w, 3) )z5Only tensors with 3, 4, or 5 dimensions are supportedrz!dim 0 must be divisible by groupsr&rkr#rlN)rfrNsizer'r r r9range)r groups dimensionssizesout_chans_per_grpmin_dimgds rdirac_rw$s ""$J"PQQ KKME Qx&A<==aF*#U1X.G  v A7^ ?PQF10014aQ19LLM1_  --1 A!+ A!+- --1 A!+ A!+ A!+ -  , M-, Ms 1CEEc|j}|dkr td|jd}|jd}d}|jdkDr|jddD]}||z} ||z}||z}||fS)Nr#zNFan in and fan out can not be computed for tensor with fewer than 2 dimensionsr&r)dimrNrnrh)r rqnum_input_fmapsnum_output_fmapsreceptive_field_sizesfan_infan_outs r_calculate_fan_in_and_fan_outrYsJA~ \  kk!nO{{1~ zz|aab! &A A %  & 3 3F!55G 7?r"gainct|\}}|tjdt||zz z}tjd|z}t || ||S)aFill the input `Tensor` with values using a Xavier uniform distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{U}(-a, a)` where .. math:: a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu')) Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.xavier_uniform_(w.T, ...)``. r@)rrrrMr)r rrr~rrrs rxavier_uniform_rnsZD4F;OFG 3v'7!889 9C #A VaRI 66r"ct|\}}|tjdt||zz z}t |d||S)aFill the input `Tensor` with values using a Xavier normal distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_normal_(w) Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.xavier_normal_(w.T, ...)``. r)rrrrMr)r rrr~rrs rxavier_normal_rsFB4F;OFG 3v'7!889 9C FCi 88r"c|j}ddg}||vrtd|d|t|\}}|dk(r|S|S)Nr~rzMode z" not supported, please use one of )lowerrNr)r mode valid_modesr~rs r_calculate_correct_fanrsX ::>> w = torch.empty(3, 5) >>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu') Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_uniform_(w.T, ...)``. )r rrrOrr,Initializing zero-element tensors is a no-oprrN)r rVrWrXkaiming_uniform_rhr)r*rrSrrr r ) r rrrOrfanrrbounds rrrsV 226:44  I%5   FLL DE  .C , *D 3 C IIcNS E Cvu BCCCs C++C4c(d|jvrtjd|St||}t ||}|t j |z }tj5|jd||cdddS#1swYyxYw)aFill the input `Tensor` with values using a Kaiming normal distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu') Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_normal_(w.T, ...)``. rrrN) rhr)r*rrSrrr r r)r rrrOrrrrs rkaiming_normal_r s}V FLL DE  .C , *D 3 C ;~~a ~:;;;s *BBc|jdkr td|jdk(r|S|jd}|j|z}|j ||fj dd|}||kr|j tjj|\}}tj|d}|j} || z}||kr|j tj5|j|j||j|ddd|S#1swY|SxYw)aFill the input `Tensor` with a (semi) orthogonal matrix. Described in `Exact solutions to the nonlinear dynamics of learning in deep linear neural networks` - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened. Args: tensor: an n-dimensional `torch.Tensor`, where :math:`n \geq 2` gain: optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> w = torch.empty(3, 5) >>> nn.init.orthogonal_(w) r#z4Only tensors with 2 or more dimensions are supportedrr&rN)rfrNnumelrn new_emptyrt_r linalgqrdiagsignr view_ascopy_r,) r rrrowscols flattenedqrrvphs r orthogonal_r>s,QOPP ||~ ;;q>D <<>T !D  $.66q!y6QI d{  <>> w = torch.empty(3, 5) >>> nn.init.sparse_(w, sparsity=0.1) r#rcrrN) rfrNrhrLrceilr r rrorandperm) r sparsityrrrr num_zeroscol_idx row_indices zero_indicess rsparse_rqs.aGHHJD$DIIho./I .q#3T{ .G...K&z 2L,-F<( ) .. M . Ms #AB44B>cljddfd}dddd|_|_|S)NcZtjdddtd|i|S)Nz `nn.init.z)` is now deprecated in favor of `nn.init.z`.r#r$)r)r* FutureWarning)argskwargsmethnew_nameold_names rdeprecated_initz(_make_deprecate..deprecated_inits; z!J8*TV W  T$V$$r"z z_(...) .. warning:: This method is now deprecated in favor of :func:`torch.nn.init.z"`. See :func:`~torch.nn.init.z` for details.)__name____doc__)rrrrs` @@r_make_deprecaters\}}H}H%$ JHIQzR'j :O (O r"r3)rrN)rrgrN)r&)rN)rr~rGN)r&N)rHN)-rrr)typingr _Optionalr rrrr1r7r;rSrM Generatorr rr[r]r_rarirwrrrrstrrrrrruniformnormalconstantrgdiracxavier_uniform xavier_normalkaiming_uniformkaiming_normal orthogonalsparser"rrsON ( : > "J!  CEP,0 6 6 6 6) 6  6:,0 : : : :) :  ::,0 P P P P P  P ) P P>'f'5'V'$ '& 'V ' "6 "f "*2j.,0&7 &7 &7)&7 &7V,0$9 $9 $9)$9 $9N3$,0 >C >C >C >C >C ) >CF$,0 2; 2; 2; 2; 2; ) 2;n ,00)0l ,0 #) #N. ( #  ! 9 %d 1/ !"23 1 [ )  !r"