fiftyone.core.utils.ContentSizeBatcher

class documentation

class ContentSizeBatcher(Batcher): (source)

Constructor: ContentSizeBatcher(iterable, target_size, max_batch_size, transform_fn, ...)

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.

The batch sizes emitted when iterating over this object are dynamically scaled such that the total content size of the batch is as close as possible to a specified target size.

This batcher does not require backpressure feedback because it calculates the total size of the iterable object before batching.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.ContentSizeBatcher(
    elements,
    target_size=2**20,
    progress=True
)

with batcher:
    for batch in batcher:
        print("batch size: %d" % len(batch))

Parameters
iterable	an iterable to batch over. If `None`, the result of `next()` will be a batch size instead of a batch, and is an infinite iterator
target_size	the target batch BSON content size, in bytes
transform_fn	a transform function to apply to each item
size_calc_fn	a function that calculates the size of each item. This is applied after `transform_fn` if both are provided
progress	whether to render a progress bar tracking the consumption of the batches (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead
total	the length of `iterable`. Only applicable when `progress=True`. If not provided, it is computed via `len(iterable)`, if possible

Method	`__init__`	Undocumented
Method	`__iter__`	Undocumented
Method	`__next__`	Undocumented
Method	`set_encoding_ratio`	Undocumented
Instance Variable	`max_batch_size`	Undocumented
Instance Variable	`size_calc_fn`	Undocumented
Instance Variable	`target_size`	Undocumented
Instance Variable	`_encoding_ratio`	Undocumented
Instance Variable	`_last_batch_content_size`	Undocumented
Instance Variable	`_last_batch_size`	Undocumented
Instance Variable	`_next_element`	Undocumented

Inherited from Batcher:

Method	`__enter__`	Undocumented
Method	`__exit__`	Undocumented
Instance Variable	`iterable`	Undocumented
Instance Variable	`return_views`	Undocumented
Instance Variable	`_in_context`	Undocumented
Instance Variable	`_iter`	Undocumented
Instance Variable	`_last_offset`	Undocumented
Instance Variable	`_num_samples`	Undocumented
Instance Variable	`_pb`	Undocumented
Instance Variable	`_render_progress`	Undocumented