Database utilities.
Class |
|
Backing document for the database config. |
Function | aggregate |
Executes one or more aggregations on a collection. |
Function | bulk |
Performs a batch of write operations on a collection. |
Function | count |
Undocumented |
Function | delete |
Deletes the annotation run with the given key from the dataset with the given name. |
Function | delete |
Deletes all annotation runs from the dataset with the given name. |
Function | delete |
Deletes the brain method run with the given key from the dataset with the given name. |
Function | delete |
Deletes all brain method runs from the dataset with the given name. |
Function | delete |
Deletes the dataset with the given name. |
Function | delete |
Deletes the evaluation run with the given key from the dataset with the given name. |
Function | delete |
Deletes all evaluations from the dataset with the given name. |
Function | delete |
Deletes the run with the given key from the dataset with the given name. |
Function | delete |
Deletes all runs from the dataset with the given name. |
Function | delete |
Deletes the saved view with the given name from the dataset with the given name. |
Function | delete |
Deletes all saved views from the dataset with the given name. |
Function | drop |
Drops specified collection from the database. |
Function | drop |
Drops the database. |
Function | drop |
Drops all orphan collections from the database. |
Function | drop |
Drops all orphan runs from the database. |
Function | drop |
Drops all orphan saved views from the database. |
Function | drop |
Drops all orphan execution stores from the database. |
Function | ensure |
Ensures database connection exists |
Function | establish |
Establishes the database connection. |
Function | export |
Exports the collection to disk in JSON format. |
Function | export |
Exports the document to disk in JSON format. |
Function | get |
Returns an async database client. |
Function | get |
Returns an async connection to the database. |
Function | get |
Sets stats about the collection. |
Function | get |
Returns a database client. |
Function | get |
Retrieves the database config. |
Function | get |
Returns a connection to the database. |
Function | get |
Returns the values of the field(s) for all samples in the given collection that are covered by the index. Raises an error if the field is not indexed. |
Function | import |
Imports the collection from JSON on disk. |
Function | import |
Imports a document from JSON on disk. |
Function | insert |
Inserts documents into a collection. |
Function | list |
Returns a list of all collection names in the database. |
Function | list |
Returns the list of available FiftyOne datasets. |
Function | patch |
Ensures that the annotation runs in the runs collection for the given dataset exactly match the values in its dataset document. |
Function | patch |
Ensures that the brain method runs in the runs collection for the given dataset exactly match the values in its dataset document. |
Function | patch |
Ensures that the evaluation runs in the runs collection for the given dataset exactly match the values in its dataset document. |
Function | patch |
Ensures that the runs in the runs collection for the given dataset exactly match the values in its dataset document. |
Function | patch |
Ensures that the saved view documents in the views collection for the given dataset exactly match the IDs in its dataset document. |
Function | patch |
Ensures that the workspace documents in the workspaces collection for the given dataset exactly match the IDs in its dataset document. |
Function | stream |
Streams the contents of the collection to stdout. |
Function | sync |
Syncs all pending database writes to disk. |
Variable | foa |
Undocumented |
Variable | fob |
Undocumented |
Variable | fod |
Undocumented |
Variable | foe |
Undocumented |
Variable | fors |
Undocumented |
Variable | logger |
Undocumented |
Class | _ |
Undocumented |
Function | _apply |
Undocumented |
Function | _async |
Undocumented |
Function | _connect |
Undocumented |
Function | _delete |
Deletes all non-persistent datasets if and only if we are the only client currently connected to the database. |
Function | _delete |
Undocumented |
Function | _delete |
Undocumented |
Function | _delete |
Undocumented |
Function | _delete |
Undocumented |
Function | _delete |
Undocumented |
Function | _delete |
Undocumented |
Function | _disconnect |
Undocumented |
Async Function | _do |
Undocumented |
Async Function | _do |
Undocumented |
Function | _do |
Undocumented |
Function | _export |
Undocumented |
Function | _export |
Undocumented |
Function | _get |
Fetches the current FCV and server version, if we have permission to read the admin database. |
Function | _get |
Undocumented |
Function | _get |
Undocumented |
Function | _get |
Undocumented |
Function | _get |
Undocumented |
Function | _handle |
Undocumented |
Function | _import |
Undocumented |
Function | _import |
Undocumented |
Function | _is |
Tests to see if feature compatibility version (FCV) upgrade is possible. |
Function | _iter |
Undocumented |
Function | _patch |
Ensures that the referenced documents in the collection for the given dataset exactly match the IDs in its dataset document. |
Function | _patch |
Undocumented |
Function | _update |
Updates a database's feature compatibility version (FCV) if possible. |
Function | _validate |
Undocumented |
Constant | _RUNS |
Undocumented |
Variable | _async |
Undocumented |
Variable | _client |
Undocumented |
Variable | _connection |
Undocumented |
Variable | _db |
Undocumented |
Executes one or more aggregations on a collection.
Multiple aggregations are executed using multiple threads, and their results are returned as lists rather than cursors.
Parameters | |
collection | a pymongo.collection.Collection or motor.motor_asyncio.AsyncIOMotorCollection |
pipelines | a MongoDB aggregation pipeline or a list of pipelines |
hints:None | a corresponding index hint or list of index hints for each pipeline |
Returns | |
|
Performs a batch of write operations on a collection.
Parameters | |
ops | a list of pymongo operations |
coll | a pymongo collection |
ordered:False | whether the operations must be performed in order |
batcher:None | an optional fiftyone.core.utils.Batcher class
to use to batch the operations, or False to strictly perform
the operations in a single batch. By default,
fiftyone.config.default_batcher is used |
progress:False | whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead |
Returns | |
A list of pymongo.results.BulkWriteResult objects |
Deletes the annotation run with the given key from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_annotation_run
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
anno | the annotation key |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes all annotation runs from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_annotation_runs
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes the brain method run with the given key from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_brain_run
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
brain | the brain key |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes all brain method runs from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_brain_runs
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
, which is helpful if a dataset's
backing document or collections are corrupted and cannot be loaded via the
normal pathways.
Parameters | |
name | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes the evaluation run with the given key from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_evaluation
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
eval | the evaluation key |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes all evaluations from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_evaluations
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes the run with the given key from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_run
, which is
helpful if a dataset's backing document or collections are corrupted and
cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
run | the run key |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes all runs from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.delete_runs
, which is
helpful if a dataset's backing document or collections are corrupted and
cannot be loaded via the normal pathways.
Note that, as this method does not load fiftyone.core.runs.Run
instances, it does not call fiftyone.core.runs.Run.cleanup
.
Parameters | |
name | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes the saved view with the given name from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.load_saved_view
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Parameters | |
dataset | the name of the dataset |
view | the name of the saved view |
dryFalse | whether to log the actions that would be taken but not perform them |
Deletes all saved views from the dataset with the given name.
This is a low-level implementation of deletion that does not call
fiftyone.core.dataset.load_dataset
or
fiftyone.core.collections.SampleCollection.load_saved_view
,
which is helpful if a dataset's backing document or collections are
corrupted and cannot be loaded via the normal pathways.
Parameters | |
dataset | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Drops all orphan collections from the database.
Orphan collections are collections that are not associated with any known dataset or other collections used by FiftyOne.
Parameters | |
dryFalse | whether to log the actions that would be taken but not perform them |
Drops all orphan runs from the database.
Orphan runs are runs that are not associated with any known dataset or other collections used by FiftyOne.
Parameters | |
dryFalse | whether to log the actions that would be taken but not perform them |
Drops all orphan saved views from the database.
Orphan saved views are saved view documents that are not associated with any known dataset or other collections used by FiftyOne.
Parameters | |
dryFalse | whether to log the actions that would be taken but not perform them |
Drops all orphan execution stores from the database.
Orphan stores are those that are associated with a dataset that no longer exists in the database.
Parameters | |
dryFalse | whether to log the actions that would be taken but not perform them |
Establishes the database connection.
If fiftyone.config.database_uri is defined, then we connect to that
URI. Otherwise, a fiftyone.core.service.DatabaseService
is
created.
Parameters | |
config | a fiftyone.core.config.FiftyOneConfig |
Raises | |
ConnectionError | if a connection to mongod could not be established |
FiftyOneConfigError | if fiftyone.config.database_uri is not defined and mongod could not be found |
ServiceExecutableNotFound | if
fiftyone.core.service.DatabaseService startup was
attempted, but mongod was not found in fiftyone.db.bin |
RuntimeError | if the mongod found does not meet FiftyOne's requirements, or validation could not occur |
Exports the collection to disk in JSON format.
Parameters | |
docs | an iterable containing the documents to export |
json | the path to write a single JSON file containing the entire collection, or a directory in which to write per-document JSON files |
key:"documents" | the field name under which to store the documents when json_path is a single JSON file |
patt | Undocumented |
numNone | the total number of documents. If omitted, this must be computable via len(docs) |
progress:None | whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead |
patt ("{idx | 06d}-{id}.json"): a filename pattern to use when json_path is a directory. The pattern may contain idx to refer to the index of the document in docs or id to refer to the document's ID |
Exports the document to disk in JSON format.
Parameters | |
doc | a BSON document dict |
json | the path to write the JSON file |
Returns an async database client.
Parameters | |
use | whether to use the global client singleton |
Returns | |
a motor.motor_asyncio.AsyncIOMotorClient |
Sets stats about the collection.
Parameters | |
collection | the name of the collection |
Returns | |
a stats dict |
Returns the values of the field(s) for all samples in the given collection that are covered by the index. Raises an error if the field is not indexed.
Parameters | |
collection | a pymongo.collection.Collection or motor.motor_asyncio.AsyncIOMotorCollection |
field | the field name or list of field names to retrieve. |
indexNone | the name of the index to use. If None, the default index name will be constructed from the field name(s). |
query:None | a dict selection filter to apply when querying. For performance, this should only include fields that are in the specified index. |
valuesFalse | whether to remove field names from the resulting list. If True, the field names are removed and only the values will be returned as a list for each sample. If False, the field names are preserved and the values will be returned as a dict for each sample. |
Returns | |
a list of values for the specified field or index keys for each sample sorted in the same order as the index | |
Raises | |
ValueError | if the field is not indexed |
Imports the collection from JSON on disk.
Parameters | |
json | the path to a JSON file on disk, or a directory containing per-document JSON files |
key:"documents" | the field name under which the documents are stored when json_path is a single JSON file |
Returns | |
a tuple of
|
Imports a document from JSON on disk.
Parameters | |
json | the path to the document |
Returns | |
a BSON document dict |
Inserts documents into a collection.
The _id field of the input documents will be populated if it is not already set.
Parameters | |
docs | an iterable of BSON document dicts |
coll | a pymongo collection |
ordered:False | whether the documents must be inserted in order |
batcher:None | an optional fiftyone.core.utils.Batcher class
to use to batch the documents, or False to strictly insert the
documents in a single batch. By default,
fiftyone.config.default_batcher is used |
progress:None | whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead |
numNone | the total number of documents. Only used when progress=True. If omitted, this will be computed via len(docs), if possible |
Returns | |
a list of IDs of the inserted documents |
Returns the list of available FiftyOne datasets.
This is a low-level implementation of dataset listing that does not call
fiftyone.core.dataset.list_datasets
, which is helpful if a
database may be corrupted.
Returns | |
a list of Dataset names |
Ensures that the annotation runs in the runs collection for the given dataset exactly match the values in its dataset document.
Parameters | |
dataset | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Ensures that the brain method runs in the runs collection for the given dataset exactly match the values in its dataset document.
Parameters | |
dataset | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Ensures that the evaluation runs in the runs collection for the given dataset exactly match the values in its dataset document.
Parameters | |
dataset | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Ensures that the runs in the runs collection for the given dataset exactly match the values in its dataset document.
Parameters | |
dataset | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Ensures that the saved view documents in the views collection for the given dataset exactly match the IDs in its dataset document.
Parameters | |
dataset | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Ensures that the workspace documents in the workspaces collection for the given dataset exactly match the IDs in its dataset document.
Parameters | |
dataset | the name of the dataset |
dryFalse | whether to log the actions that would be taken but not perform them |
Streams the contents of the collection to stdout.
Parameters | |
collection | the name of the collection |
Deletes all non-persistent datasets if and only if we are the only client currently connected to the database.
pymongo.MongoClient
) -> tuple[ Version, Version]
:
(source)
¶
Fetches the current FCV and server version, if we have permission to read the admin database.
Parameters | |
client:pymongo.MongoClient | a pymongo.MongoClient to connect to the database |
Returns | |
tuple[ | a tuple of
|
Raises | |
ConnectionError | if a connection to mongod could not be established |
Tests to see if feature compatibility version (FCV) upgrade is possible.
The following conditions return False:
- If the FCV is greater than the server version, warn that this is an unexpected and return False.
- If the major versions between server and FCV is greater than we can handle, warn that this is unexpected and return False.
- If either the server's version are FCV are the oldest supported version, warn about any upcoming deprecations. If the server version is not greater than the FCV return False.
Note that MongoDB will fail to initialize if the server version and FCV differ by two or more major versions, so this check may be redundant.
Parameters | |
fcVersion | a packaging.Version representing the FCV |
serverVersion | a packaging.Version representing the Server's version |
Returns | |
bool | whether a version upgrade is possible |
Ensures that the referenced documents in the collection for the given dataset exactly match the IDs in its dataset document.
Updates a database's feature compatibility version (FCV) if possible.
Checks to see if a version upgrade for the FCV is required and possible. If it is, issue an upgrade and log as a warning.
Note that MongoDB will fail to initialize if the server version and FCV differ by two or more major versions, so this check may be redundant.
Parameters | |
client:pymongo.MongoClient | a pymongo.MongoClient to connect to the database |