fiftyone.core.utils.ContentSizeDynamicBatcher

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.

The batch sizes emitted when iterating over this object are dynamically scaled such that the total content size of the batch is as close as possible to a specified target size.

This batcher requires that backpressure feedback be provided, either by providing a BSON-able batch from which the content size can be computed, or by manually providing the content size.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.ContentSizeDynamicBatcher(
    elements, target_size=2**20, max_batch_beta=2.0
)

# Raises ValueError after first batch, we forgot to apply backpressure
for batch in batcher:
    print("batch size: %d" % len(batch))

# Now it works
for batch in batcher:
    print("batch size: %d" % len(batch))
    batcher.apply_backpressure(batch)

batcher = fou.ContentSizeDynamicBatcher(
    elements,
    target_size=2**20,
    max_batch_beta=2.0,
    progress=True
)

with batcher:
    for batch in batcher:
        print("batch size: %d" % len(batch))
        batcher.apply_backpressure(batch)