DŽgi0,ddlZddlmZddlmZmZmZmZmZm Z m Z m Z m Z ddl ZddlmZddlmZmZddlmZgdZe dd Zd efd Zed GddeeZedGddeZedGddeeZy)N) defaultdict) AnyCallable DefaultDictIteratorListOptionalSizedTypeTypeVar)functional_datapipe) DataChunk IterDataPipe)_check_unpickable_fn)BatcherIterDataPipeGrouperIterDataPipeUnBatcherIterDataPipe_T_coT) covariantnamec|dvretjd|d|dtdttj j jjj|Stdtd|) N)SHARDING_PRIORITIESShardingFilterIterDataPipe`zc` from `torch.utils.data.datapipes.iter.grouping` is going to be removed in PyTorch 2.1Please use `z5` from the `torch.utils.data.datapipes.iter.sharding`)category stacklevelzmodule z has no attribute ) warningswarn FutureWarninggetattrtorchutilsdata datapipesitershardingAttributeError__name__)rs p/home/mcse/projects/flask_80/flask-venv/lib/python3.12/site-packages/torch/utils/data/datapipes/iter/grouping.py __getattr__r+ s{ DD v& U W"  u{{''1166??FF 78*,>tfE FFbatchc eZdZUdZeed<eed<eed<defdededede eddf fd Z de efd Z defd Z xZS) ra2 Creates mini-batches of data (functional name: ``batch``). An outer dimension will be added as ``batch_size`` if ``drop_last`` is set to ``True``, or ``length % batch_size`` for the last batch if ``drop_last`` is set to ``False``. Args: datapipe: Iterable DataPipe being batched batch_size: The size of each batch drop_last: Option to drop the last batch if it's not full wrapper_class: wrapper to apply onto each batch (type ``List``) before yielding, defaults to ``DataChunk`` Example: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> dp = IterableWrapper(range(10)) >>> dp = dp.batch(batch_size=3, drop_last=True) >>> list(dp) [[0, 1, 2], [3, 4, 5], [6, 7, 8]] datapipe batch_size drop_lastF wrapper_classreturnNcr|dkDsJdt|||_||_||_||_y)Nrz+Batch size is required to be larger than 0!)super__init__r/r0r1r2)selfr/r0r1r2 __class__s r*r6zBatcherIterDataPipe.__init__Js?A~LLL~   $"*r,c#Kg}|jD]A}|j|t||jk(s-|j |g}Ct|dkDr!|j s|j |yyywNr)r/appendlenr0r2r1)r7r-xs r*__iter__zBatcherIterDataPipe.__iter__Xs~ A LLO5zT__,((//   u:>>>((//" s ;BABc8t|jtr`|jr"t |j|j zSt |j|j zdz |j zSt t|jd)Nz# instance doesn't have valid length) isinstancer/r r1r<r0 TypeErrortyper)r7s r*__len__zBatcherIterDataPipe.__len__css dmmU +~~4==)T__<<DMM*T__rE __classcell__)r8s@r*rr.sz,OO  )2 + + + + I +  + 0(9- 0YYr,runbatchc.eZdZdZddedefdZdZdZy) ra  Undos batching of data (functional name: ``unbatch``). In other words, it flattens the data up to the specified level within a batched DataPipe. Args: datapipe: Iterable DataPipe being un-batched unbatch_level: Defaults to ``1`` (only flattening the top level). If set to ``2``, it will flatten the top two levels, and ``-1`` will flatten the entire DataPipe. Example: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> source_dp = IterableWrapper([[[0, 1], [2]], [[3, 4], [5]], [[6]]]) >>> dp1 = source_dp.unbatch() >>> list(dp1) [[0, 1], [2], [3, 4], [5], [6]] >>> dp2 = source_dp.unbatch(unbatch_level=2) >>> list(dp2) [0, 1, 2, 3, 4, 5, 6] r/ unbatch_levelc ||_||_yN)r/rO)r7r/rOs r*r6zUnBatcherIterDataPipe.__init__s  *r,c#|K|jD]'}|j||jEd{)y7w)NrO)r/_diverO)r7elements r*r>zUnBatcherIterDataPipe.__iter__s:}} MGzz'9K9KzL L L M Ls 0<:<c#vK|dkr td|dk(r>t|ttfr#|D]}|j |dEd{y|y|dk(r|yt|ttfr&|D] }|j ||dz Ed{"yt d|j d7k7"w)Nz unbatch_level must be -1 or >= 0rSrr@zunbatch_level z" exceeds the depth of the DataPipe) ValueErrorrAlistrrT IndexErrorrO)r7rUrOitems r*rTzUnBatcherIterDataPipe._dives 2 ?@ @ B 'D)#45#BD#zz$bzAAAB a M'D)#45#QD#zz$ma>OzPPPQ!$T%7%7$88Z[BQs%AB9 B5 A B9B7!B97B9N)r@) r)rFrGrHrrJr6r>rTr,r*rrms%,++c+Mr,rgroupbyceZdZdZdddddddeedeegefded e d e e d e e d efd Z dZ dZ ddZdZdZdZy)ra! Groups data from IterDataPipe by keys from ``group_key_fn``, yielding a ``DataChunk`` with batch size up to ``group_size``. (functional name: ``groupby``). The samples are read sequentially from the source ``datapipe``, and a batch of samples belonging to the same group will be yielded as soon as the size of the batch reaches ``group_size``. When the buffer is full, the DataPipe will yield the largest batch with the same key, provided that its size is larger than ``guaranteed_group_size``. If its size is smaller, it will be dropped if ``drop_remaining=True``. After iterating through the entirety of source ``datapipe``, everything not dropped due to the buffer capacity will be yielded from the buffer, even if the group sizes are smaller than ``guaranteed_group_size``. Args: datapipe: Iterable datapipe to be grouped group_key_fn: Function used to generate group key from the data of the source datapipe keep_key: Option to yield the matching key along with the items in a tuple, resulting in `(key, [items])` otherwise returning [items] buffer_size: The size of buffer for ungrouped data group_size: The max size of each group, a batch is yielded as soon as it reaches this size guaranteed_group_size: The guaranteed minimum group size to be yielded in case the buffer is full drop_remaining: Specifies if the group smaller than ``guaranteed_group_size`` will be dropped from buffer when the buffer is full Example: >>> import os >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> def group_fn(file): ... return os.path.basename(file).split(".")[0] >>> source_dp = IterableWrapper(["a.png", "b.png", "a.json", "b.json", "a.jpg", "c.json"]) >>> dp0 = source_dp.groupby(group_key_fn=group_fn) >>> list(dp0) [['a.png', 'a.json', 'a.jpg'], ['b.png', 'b.json'], ['c.json']] >>> # A group is yielded as soon as its size equals to `group_size` >>> dp1 = source_dp.groupby(group_key_fn=group_fn, group_size=2) >>> list(dp1) [['a.png', 'a.json'], ['b.png', 'b.json'], ['a.jpg'], ['c.json']] >>> # Scenario where `buffer` is full, and group 'a' needs to be yielded since its size > `guaranteed_group_size` >>> dp2 = source_dp.groupby(group_key_fn=group_fn, buffer_size=3, group_size=3, guaranteed_group_size=2) >>> list(dp2) [['a.png', 'a.json'], ['b.png', 'b.json'], ['a.jpg'], ['c.json']] Fi'N)keep_key buffer_size group_sizeguaranteed_group_sizedrop_remainingr/ group_key_fnr_r`rarbrcc4t|||_||_||_||_t t |_d|_||_ d|_ ||d|cxkr|ksJJ||_ ||d|cxkr|ksJJ||_ ||_ t|_ yr:)rr/rdr_max_buffer_sizerrYbuffer_elementscurr_buffer_sizerarbrcrr2)r7r/rdr_r`rarbrcs r*r6zGrouperIterDataPipe.__init__s \*  (  *7B47H !$%)"  !k&=z0[0 00 00)3D & ,)a2G.U:.U UU.U UU)>D &,&r,cd}d}d}|jjD]8}t|j||kDst|j|}|}:|j=||jkr.|js"t dt |j||j||jk\r|j|}|xj|zc_|j|=|S)NrzFailed to group items)rgkeysr<rbrc RuntimeErrorstrrh)r7 biggest_key biggest_sizeresult_to_yieldfindkeys r*_remove_biggest_keyz'GrouperIterDataPipe._remove_biggest_keys  ++002 &G4''01L@"4#7#7#@A %  &  & & 2t999'''T-A-A+-N)O   & & .t999"22;?O -   -r,c#K|jD]5}|j|}|j|j||xjdz c_|j |j t |j|k(rj|j|j|}|jr||fn||xjt |j|zc_|j|=|j|jk(s|j}||j|}|jr||fn|8t|jjD]^}|j|jj|}|xjt |zc_|jr||fn|`yw)Nr@)r/rdrgr;rhrar<r2r_rfrqtuplerjpop)r7r=keyresultros r*r>zGrouperIterDataPipe.__iter__ s EA##A&C   % , ,Q /  ! !Q & !*t#$$S)C0*.););Drxrrrr\r,r*rrs*b $(/3$'u%'w|,'  '  'SM' (}''<:=41"1 %r,r)r collectionsrtypingrrrrrr r r r (torch.utils.data.datapipes.iter.shardingr"%torch.utils.data.datapipes._decoratorr #torch.utils.data.datapipes.datapiperr'torch.utils.data.datapipes.utils.commonr__all__rrlr+rrrr\r,r*rs#   0EGH  4( Gc GW;Y,y1;Y;Y|Y1L1 1hYh%,y1h% h%r,