fiftyone.core.utils.ContentSizeBatcher

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.

The batch sizes emitted when iterating over this object are dynamically scaled such that the total content size of the batch is as close as possible to a specified target size.

This batcher does not require backpressure feedback because it calculates the total size of the iterable object before batching.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.ContentSizeBatcher(
    elements,
    target_size=2**20,
    progress=True
)

with batcher:
    for batch in batcher:
        print("batch size: %d" % len(batch))