fiftyone.core.models

module documentation

(source)

FiftyOne models.

Class	`EmbeddingsMixin`	Mixin for `Model` classes that can generate embeddings for their predictions.
Class	`LogitsMixin`	Mixin for `Model` classes that can generate logits for their predictions.
Class	`Model`	Abstract base class for models.
Class	`ModelConfig`	Base configuration class that encapsulates the name of a `Model` and an instance of its associated Config class.
Class	`ModelManager`	Class for downloading FiftyOne models from the web.
Class	`ModelManagerConfig`	Config settings for a `ModelManager`.
Class	`PromptMixin`	Mixin for `Model` classes that can generate prompt embeddings.
Class	`SamplesMixin`	Mixin for `Model` classes that need samples for prediction.
Class	`TorchModelMixin`	Mixin for `Model` classes that support feeding data for inference via a `torch:torch.utils.data.DataLoader`.
Function	`apply_model`	Applies the model to the samples in the collection.
Function	`compute_embeddings`	Computes embeddings for the samples in the collection using the given model.
Function	`compute_patch_embeddings`	Computes embeddings for the image patches defined by `patches_field` of the samples in the collection using the given model.
Function	`load_model`	Loads the model specified by the given `ModelConfig` dict.
Variable	`foue`	Undocumented
Variable	`fouf`	Undocumented
Variable	`foui`	Undocumented
Variable	`foup`	Undocumented
Variable	`fous`	Undocumented
Variable	`fout`	Undocumented
Variable	`foutr`	Undocumented
Variable	`fouu`	Undocumented
Variable	`logger`	Undocumented
Variable	`tud`	Undocumented
Function	`_apply_image_model_batch`	Undocumented
Function	`_apply_image_model_data_loader`	Undocumented
Function	`_apply_image_model_single`	Undocumented
Function	`_apply_image_model_to_frames_batch`	Undocumented
Function	`_apply_image_model_to_frames_single`	Undocumented
Function	`_apply_video_model`	Undocumented
Function	`_compute_frame_embeddings_batch`	Undocumented
Function	`_compute_frame_embeddings_single`	Undocumented
Function	`_compute_image_embeddings_batch`	Undocumented
Function	`_compute_image_embeddings_data_loader`	Undocumented
Function	`_compute_image_embeddings_single`	Undocumented
Function	`_compute_video_embeddings`	Undocumented
Function	`_convert_model_if_necessary`	Undocumented
Function	`_do_export_array`	Undocumented
Function	`_embed_frame_patches`	Undocumented
Function	`_embed_patches`	Undocumented
Function	`_embed_patches_batch`	Undocumented
Function	`_embed_patches_data_loader`	Undocumented
Function	`_embed_patches_single`	Undocumented
Function	`_export_arrays`	Undocumented
Function	`_get_frame_counts`	Undocumented
Function	`_is_flash_model`	Undocumented
Function	`_is_super_gradients_models`	Undocumented
Function	`_is_transformers_model`	Undocumented
Function	`_is_ultralytics_model`	Undocumented
Function	`_iter_batches`	Undocumented
Function	`_make_data_loader`	Undocumented
Function	`_make_label_parser`	Undocumented
Function	`_make_patch_data_loader`	Undocumented
Function	`_merge_config`	Undocumented
Function	`_parse_batch_size`	Undocumented
Constant	`_ALLOWED_PATCH_TYPES`	Undocumented

def apply_model(samples, model, label_field='predictions', confidence_thresh=None, store_logits=False, batch_size=None, num_workers=None, skip_failures=True, output_dir=None, rel_dir=None, progress=None, **kwargs): (source) ¶

Applies the model to the samples in the collection.

This method supports all of the following cases:

Applying an image model to an image collection
Applying an image model to the frames of a video collection
Applying a video model to a video collection

Parameters
samples	a `fiftyone.core.collections.SampleCollection`
model	a `Model`, Hugging Face Transformers model, Ultralytics model, SuperGradients model, or Lightning Flash model
label_field:"predictions"	the name of the field in which to store the model predictions. When performing inference on video frames, the "frames." prefix is optional
confidence_thresh:`None`	an optional confidence threshold to apply to any applicable labels generated by the model
store_logits:`False`	whether to store logits for the model predictions. This is only supported when the provided `model` has logits, `model.has_logits == True`
batch_size:`None`	an optional batch size to use, if the model supports batching
num_workers:`None`	the number of workers to use when loading images. Only applicable for Torch-based models
skip_failures:`True`	whether to gracefully continue without raising an error if predictions cannot be generated for a sample. Only applicable to `Model` instances
output_dir:`None`	an optional output directory in which to write segmentation images. Only applicable if the model generates segmentations. If none is provided, the segmentations are stored in the database
rel_dir:`None`	an optional relative directory to strip from each input filepath to generate a unique identifier that is joined with `output_dir` to generate an output path for each segmentation image. This argument allows for populating nested subdirectories in `output_dir` that match the shape of the input paths. The path is converted to an absolute path (if necessary) via `fiftyone.core.storage.normalize_path`
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead
**kwargs	optional model-specific keyword arguments passed through to the underlying inference implementation

def compute_embeddings(samples, model, embeddings_field=None, batch_size=None, num_workers=None, skip_failures=True, progress=None, **kwargs): (source) ¶

Computes embeddings for the samples in the collection using the given model.

This method supports all the following cases:

Using an image model to compute embeddings for an image collection
Using an image model to compute frame embeddings for a video collection
Using a video model to compute embeddings for a video collection

The model must expose embeddings, i.e., Model.has_embeddings must return True.

If an embeddings_field is provided, the embeddings are saved to the samples; otherwise, the embeddings are returned in-memory.

Parameters
samples	a `fiftyone.core.collections.SampleCollection`
model	a `Model`, Hugging Face Transformers model, Ultralytics model, SuperGradients model, or Lightning Flash model
embeddings_field:`None`	the name of a field in which to store the embeddings. When computing video frame embeddings, the "frames." prefix is optional
batch_size:`None`	an optional batch size to use, if the model supports batching
num_workers:`None`	the number of workers to use when loading images. Only applicable for Torch-based models
skip_failures:`True`	whether to gracefully continue without raising an error if embeddings cannot be generated for a sample. Only applicable to `Model` instances
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead
**kwargs	optional model-specific keyword arguments passed through to the underlying inference implementation
Returns
`one` of the following	`None`, if an `embeddings_field` is provided a `num_samples x num_dim` array of embeddings, when computing embeddings for image/video collections with image/video models, respectively, and no `embeddings_field` is provided. If `skip_failures` is `True` and any errors are detected, a list of length `num_samples` is returned instead containing all successfully computed embedding vectors along with `None` entries for samples for which embeddings could not be computed a dictionary mapping sample IDs to `num_frames x num_dim` arrays of embeddings, when computing frame embeddings for video collections using an image model. If `skip_failures` is `True` and any errors are detected, the values of this dictionary will contain arrays of embeddings for all frames 1, 2, ... until the error occurred, or `None` if no embeddings were computed at all

def compute_patch_embeddings(samples, model, patches_field, embeddings_field=None, force_square=False, alpha=None, handle_missing='skip', batch_size=None, num_workers=None, skip_failures=True, progress=None): (source) ¶

Computes embeddings for the image patches defined by patches_field of the samples in the collection using the given model.

This method supports all the following cases:

Using an image model to compute patch embeddings for an image collection
Using an image model to compute frame patch embeddings for a video collection

The model must expose embeddings, i.e., Model.has_embeddings must return True.

If an embeddings_field is provided, the embeddings are saved to the samples; otherwise, the embeddings are returned in-memory.

Parameters
samples	a `fiftyone.core.collections.SampleCollection`
model	a `Model`, Hugging Face Transformers model, Ultralytics model, SuperGradients model, or Lightning Flash model
patches_field	the name of the field defining the image patches in each sample to embed. Must be of type `fiftyone.core.labels.Detection`, `fiftyone.core.labels.Detections`, `fiftyone.core.labels.Polyline`, or `fiftyone.core.labels.Polylines`. When computing video frame embeddings, the "frames." prefix is optional
embeddings_field:`None`	the name of a label attribute in which to store the embeddings
force_square:`False`	whether to minimally manipulate the patch bounding boxes into squares prior to extraction
alpha:`None`	an optional expansion/contraction to apply to the patches before extracting them, in `[-1, inf)`. If provided, the length and width of the box are expanded (or contracted, when `alpha < 0`) by `(100 * alpha)%`. For example, set `alpha = 0.1` to expand the boxes by 10%, and set `alpha = -0.1` to contract the boxes by 10%
handle_missing:"skip"	how to handle images with no patches. Supported values are: "skip": skip the image and assign its embedding as `None` "image": use the whole image as a single patch "error": raise an error
batch_size:`None`	an optional batch size to use, if the model supports batching
num_workers:`None`	the number of workers to use when loading images. Only applicable for Torch models
skip_failures:`True`	whether to gracefully continue without raising an error if embeddings cannot be generated for a sample
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead
Returns
`one` of the following	`None`, if an `embeddings_field` is provided a dict mapping sample IDs to `num_patches x num_dim` arrays of patch embeddings, when computing patch embeddings for image collections and no `embeddings_field` is provided. If `skip_failures` is `True` and any errors are detected, this dictionary will contain `None` values for any samples for which embeddings could not be computed a dict of dicts mapping sample IDs to frame numbers to `num_patches x num_dim` arrays of patch embeddings, when computing patch embeddings for the frames of video collections and no `embeddings_field` is provided. If `skip_failures` is `True` and any errors are detected, this nested dict will contain missing or `None` values to indicate uncomputable embeddings

def load_model(model_config_dict, model_path=None, **kwargs): (source) ¶

Loads the model specified by the given ModelConfig dict.

Parameters
model_config_dict	a `ModelConfig` dict
model_path:`None`	an optional model path to inject into the `model_path` field of the model's `Config` instance, which must implement the `eta.core.learning.HasPublishedModel` interface. This is useful when working with a model whose weights are stored locally and do not need to be downloaded
**kwargs	optional keyword arguments to inject into the model's `Config` instance
Returns
a `Model` instance

foue = (source) ¶

Undocumented

fouf = (source) ¶

Undocumented

foui = (source) ¶

Undocumented

foup = (source) ¶

Undocumented

fous = (source) ¶

Undocumented

fout = (source) ¶

Undocumented

foutr = (source) ¶

Undocumented

fouu = (source) ¶

Undocumented

logger = (source) ¶

Undocumented

tud = (source) ¶

Undocumented

def _apply_image_model_batch(samples, model, label_field, confidence_thresh, batch_size, skip_failures, filename_maker, progress): (source) ¶

Undocumented

def _apply_image_model_data_loader(samples, model, label_field, confidence_thresh, batch_size, num_workers, skip_failures, filename_maker, progress): (source) ¶

Undocumented

def _apply_image_model_single(samples, model, label_field, confidence_thresh, skip_failures, filename_maker, progress): (source) ¶

Undocumented

def _apply_image_model_to_frames_batch(samples, model, label_field, confidence_thresh, batch_size, skip_failures, filename_maker, progress): (source) ¶

Undocumented

def _apply_image_model_to_frames_single(samples, model, label_field, confidence_thresh, skip_failures, filename_maker, progress): (source) ¶

Undocumented

def _apply_video_model(samples, model, label_field, confidence_thresh, skip_failures, filename_maker, progress): (source) ¶

Undocumented

def _compute_frame_embeddings_batch(samples, model, embeddings_field, batch_size, skip_failures, progress): (source) ¶

Undocumented

def _compute_frame_embeddings_single(samples, model, embeddings_field, skip_failures, progress): (source) ¶

Undocumented

def _compute_image_embeddings_batch(samples, model, embeddings_field, batch_size, skip_failures, progress): (source) ¶

Undocumented

def _compute_image_embeddings_data_loader(samples, model, embeddings_field, batch_size, num_workers, skip_failures, progress): (source) ¶

Undocumented

def _compute_image_embeddings_single(samples, model, embeddings_field, skip_failures, progress): (source) ¶

Undocumented

def _compute_video_embeddings(samples, model, embeddings_field, skip_failures, progress): (source) ¶

Undocumented

def _convert_model_if_necessary(model): (source) ¶

Undocumented

def _do_export_array(label, input_path, filename_maker): (source) ¶

Undocumented

def _embed_frame_patches(samples, model, patches_field, embeddings_field, force_square, alpha, handle_missing, batch_size, skip_failures, progress): (source) ¶

Undocumented

def _embed_patches(samples, model, patches_field, embeddings_field, force_square, alpha, handle_missing, batch_size, skip_failures, progress): (source) ¶

Undocumented

def _embed_patches_batch(model, img, detections, force_square, alpha, batch_size): (source) ¶

Undocumented

def _embed_patches_data_loader(samples, model, patches_field, embeddings_field, force_square, alpha, handle_missing, batch_size, num_workers, skip_failures, progress): (source) ¶

Undocumented

def _embed_patches_single(model, img, detections, force_square, alpha): (source) ¶

Undocumented

def _export_arrays(label, input_path, filename_maker): (source) ¶

Undocumented

def _get_frame_counts(samples): (source) ¶

Undocumented

def _is_flash_model(model): (source) ¶

Undocumented

def _is_super_gradients_models(model): (source) ¶

Undocumented

def _is_transformers_model(model): (source) ¶

Undocumented

def _is_ultralytics_model(model): (source) ¶

Undocumented

def _iter_batches(video_reader, batch_size): (source) ¶

Undocumented

def _make_data_loader(samples, model, batch_size, num_workers, skip_failures): (source) ¶

Undocumented

def _make_label_parser(samples, patches_field): (source) ¶

Undocumented

def _make_patch_data_loader(samples, model, patches_field, force_square, alpha, handle_missing, num_workers, skip_failures): (source) ¶

Undocumented

def _merge_config(d, kwargs): (source) ¶

Undocumented

def _parse_batch_size(batch_size, model, use_data_loader): (source) ¶

Undocumented

_ALLOWED_PATCH_TYPES = (source) ¶

Undocumented

Value

(fol.Detection, fol.Detections, fol.Polyline, fol.Polylines)