fiftyone.core.storage

module documentation

(source)

File storage utilities.

Class	`FileSystem`	Enumeration of the available file systems.
Class	`TempDir`	Context manager that creates and destroys a temporary directory.
Function	`abspath`	Converts the given path to an absolute path, resolving relative path indicators such as `.` and `..`.
Function	`copy_dir`	Copies the input directory to the output directory.
Function	`copy_file`	Copies the input file to the output location.
Function	`copy_files`	Copies the files to the given locations.
Function	`delete_dir`	Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.
Function	`delete_file`	Deletes the file at the given path.
Function	`delete_files`	Deletes the files from the given locations.
Function	`ensure_basedir`	Makes the base directory of the given path, if necessary.
Function	`ensure_dir`	Makes the given directory, if necessary.
Function	`ensure_empty_dir`	Ensures that the given directory exists and is empty.
Function	`ensure_local`	Ensures that the given path is local.
Function	`exists`	Determines whether the given file or directory exists.
Function	`extract_archive`	Extracts the contents of an archive.
Function	`get_bucket_name`	Gets the bucket name from the given path.
Function	`get_file_system`	Returns the file system enum for the given path.
Function	`get_glob_matches`	Returns a list of file paths matching the given glob pattern.
Function	`get_glob_root`	Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.
Function	`is_local`	Determines whether the given path is local.
Function	`isabs`	Determines whether the given path is absolute.
Function	`isdir`	Determines whether the given directory exists.
Function	`isfile`	Determines whether the given file exists.
Function	`join`	Joins the given path components into a single path.
Function	`list_available_file_systems`	Lists the file systems that are currently available for use with methods like `list_files` and `list_buckets`.
Function	`list_buckets`	Lists the available buckets in the given file system.
Function	`list_files`	Lists the files in the given directory.
Function	`list_subdirs`	Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.
Function	`load_json`	Loads JSON from the input argument.
Function	`load_ndjson`	Loads NDJSON from the input argument.
Function	`make_archive`	Makes an archive containing the given directory.
Function	`make_temp_dir`	Makes a temporary directory.
Function	`move_dir`	Moves the contents of the given directory into the given output directory.
Function	`move_file`	Moves the given file to a new location.
Function	`move_files`	Moves the files to the given locations.
Function	`normalize_path`	Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.
Function	`normpath`	Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.
Function	`open_file`	Opens the given file for reading or writing.
Function	`open_files`	Opens the given files for reading or writing.
Function	`read_file`	Reads the file.
Function	`read_files`	Reads the specified files into memory.
Function	`read_json`	Reads a JSON file.
Function	`read_ndjson`	Reads an NDJSON file.
Function	`read_yaml`	Reads a YAML file.
Function	`realpath`	Converts the given path to absolute, resolving symlinks and relative path indicators such as `.` and `..`.
Function	`run`	Applies the given function to each element of the given tasks.
Function	`sep`	Returns the path separator for the given path.
Function	`split_prefix`	Splits the file system prefix from the given path.
Function	`write_file`	Writes the given string/bytes to a file.
Function	`write_json`	Writes JSON object to file.
Function	`write_ndjson`	Writes the list of JSON dicts in NDJSON format.
Function	`write_yaml`	Writes the object to a YAML file.
Variable	`logger`	Undocumented
Function	`_copy_file`	Undocumented
Function	`_copy_files`	Undocumented
Function	`_delete_file`	Undocumented
Function	`_do_copy_file`	Undocumented
Function	`_do_delete_file`	Undocumented
Function	`_do_move_file`	Undocumented
Function	`_do_open_file`	Undocumented
Function	`_do_read_file`	Undocumented
Function	`_get_local_metadata`	Undocumented
Function	`_open_file`	Undocumented
Function	`_read_file`	Undocumented
Function	`_run`	Undocumented
Function	`_to_bytes`	Undocumented

def abspath(path): (source) ¶

Converts the given path to an absolute path, resolving relative path indicators such as . and ...

Parameters
path	the filepath
Returns
the absolute path

def copy_dir(indir, outdir, overwrite=True, skip_failures=False, progress=None): (source) ¶

Copies the input directory to the output directory.

Parameters
indir	the input directory
outdir	the output directory
overwrite:`True`	whether to delete an existing output directory (True) or merge its contents (False)
skip_failures:`False`	whether to gracefully continue without raising an error if an operation fails
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead

def copy_file(inpath, outpath): (source) ¶

Copies the input file to the output location.

Parameters
inpath	the input path
outpath	the output path

def copy_files(inpaths, outpaths, skip_failures=False, progress=None): (source) ¶

Copies the files to the given locations.

Parameters
inpaths	a list of input paths
outpaths	a list of output paths
skip_failures:`False`	whether to gracefully continue without raising an error if an operation fails
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead

def delete_dir(dirpath): (source) ¶

Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.

Parameters
dirpath	the directory path

def delete_file(path): (source) ¶

Deletes the file at the given path.

Any empty directories are also recursively deleted from the resulting directory tree.

Parameters
path	the filepath

def delete_files(paths, skip_failures=False, progress=None): (source) ¶

Deletes the files from the given locations.

Any empty directories are also recursively deleted from the resulting directory tree.

Parameters
paths	a list of paths
skip_failures:`False`	whether to gracefully continue without raising an error if an operation fails
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead

def ensure_basedir(path): (source) ¶

Makes the base directory of the given path, if necessary.

Parameters
path	the filepath

def ensure_dir(dirpath): (source) ¶

Makes the given directory, if necessary.

Parameters
dirpath	the directory path

def ensure_empty_dir(dirpath, cleanup=False): (source) ¶

Ensures that the given directory exists and is empty.

Parameters
dirpath	the directory path
cleanup:`False`	whether to delete any existing directory contents
Raises
`ValueError`	if the directory is not empty and `cleanup` is False

def ensure_local(path): (source) ¶

Ensures that the given path is local.

Parameters
path	a path

def exists(path): (source) ¶

Determines whether the given file or directory exists.

Parameters
path	the file or directory path
Returns
True/False

def extract_archive(archive_path, outdir=None, cleanup=False): (source) ¶

Extracts the contents of an archive.

The following formats are guaranteed to work: .zip, .tar, .tar.gz, .tgz, .tar.bz, .tbz.

If an archive not in the above list is found, extraction will be attempted via the patool package, which supports many formats but may require that additional system packages be installed.

Parameters
archive_path	the archive path
outdir:`None`	the directory into which to extract the archive. By default, the directory containing the archive is used
cleanup:`False`	whether to delete the archive after extraction

def get_bucket_name(path): (source) ¶

Gets the bucket name from the given path.

The bucket name for local paths is "".

Example usages:

import fiftyone.core.storage as fos

fos.get_bucket_name("/path/to/file")       # ''
fos.get_bucket_name("a/file")              # ''

Parameters
path	a path
Returns
the bucket name string

def get_file_system(path): (source) ¶

Returns the file system enum for the given path.

Parameters
path	a path
Returns
a `FileSystem` value

def get_glob_matches(glob_patt): (source) ¶

Returns a list of file paths matching the given glob pattern.

The matches are returned in sorted order.

Parameters
glob_patt	a glob pattern like `/path/to/files-*.jpg`
Returns
a list of file paths

def get_glob_root(glob_patt): (source) ¶

Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.

Parameters
glob_patt	a glob pattern like `/path/to/files-*.jpg`
Returns
a tuple of	the root True/False whether the pattern contains any special characters

def is_local(path): (source) ¶

Determines whether the given path is local.

Parameters
path	a path
Returns
True/False

def isabs(path): (source) ¶

Determines whether the given path is absolute.

Parameters
path	the filepath
Returns
True/False

def isdir(dirpath): (source) ¶

Determines whether the given directory exists.

Cloud "folders" are deemed to exist only if they are non-empty.

Parameters
dirpath	the directory path
Returns
True/False

def isfile(path): (source) ¶

Determines whether the given file exists.

Parameters
path	the filepath
Returns
True/False

def join(a, *p): (source) ¶

Joins the given path components into a single path.

Parameters
a	the root
*p	additional path components
Returns
the joined path

def list_available_file_systems(): (source) ¶

Lists the file systems that are currently available for use with methods like list_files and list_buckets.

Returns
a list of `FileSystem` values

def list_buckets(fs, abs_paths=False): (source) ¶

Lists the available buckets in the given file system.

This method returns subdirectories of / (or the current drive on Windows).

Parameters
fs	a `FileSystem` value
abs_paths:`False`	whether to return absolute paths
Returns
a list of buckets

def list_files(dirpath, abs_paths=False, recursive=False, include_hidden_files=False, return_metadata=False, sort=True): (source) ¶

Lists the files in the given directory.

If the directory does not exist, an empty list is returned.

Parameters
dirpath	the path to the directory to list
abs_paths:`False`	whether to return the absolute paths to the files
recursive:`False`	whether to recursively traverse subdirectories
include_hidden_files:`False`	whether to include dot files
return_metadata:`False`	whether to return metadata dicts for each file instead of filepaths
sort:`True`	whether to sort the list of files
Returns
a list of filepaths or metadata dicts

def list_subdirs(dirpath, abs_paths=False, recursive=False): (source) ¶

Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.

Parameters
dirpath	the path to the directory to list
abs_paths:`False`	whether to return absolute paths
recursive:`False`	whether to recursively traverse subdirectories
Returns
a list of subdirectories

def load_json(path_or_str): (source) ¶

Loads JSON from the input argument.

Parameters
path_or_str	the filepath or JSON string
Returns
the loaded JSON

def load_ndjson(path_or_str): (source) ¶

Loads NDJSON from the input argument.

Parameters
path_or_str	the filepath or NDJSON string
Returns
a list of JSON dicts

def make_archive(dirpath, archive_path, cleanup=False): (source) ¶

Makes an archive containing the given directory.

Supported formats include .zip, .tar, .tar.gz, .tgz, .tar.bz and .tbz.

Parameters
dirpath	the directory to archive
archive_path	the archive path to write
cleanup:`False`	whether to delete the directory after archiving it

def make_temp_dir(basedir=None): (source) ¶

Makes a temporary directory.

Parameters
basedir:`None`	an optional directory in which to create the new directory. The default is `fiftyone.config.default_dataset_dir`
Returns
the temporary directory path

def move_dir(indir, outdir, overwrite=True, skip_failures=False, progress=None): (source) ¶

Moves the contents of the given directory into the given output directory.

Parameters
indir	the input directory
outdir	the output directory
overwrite:`True`	whether to delete an existing output directory (True) or merge its contents (False)
skip_failures:`False`	whether to gracefully continue without raising an error if an operation fails
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead

def move_file(inpath, outpath): (source) ¶

Moves the given file to a new location.

Parameters
inpath	the input path
outpath	the output path

def move_files(inpaths, outpaths, skip_failures=False, progress=None): (source) ¶

Moves the files to the given locations.

Parameters
inpaths	a list of input paths
outpaths	a list of output paths
skip_failures:`False`	whether to gracefully continue without raising an error if an operation fails
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead

def normalize_path(path): (source) ¶

Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.

Parameters
path	a path
Returns
the normalized path

def normpath(path): (source) ¶

Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.

Use this function when you need a version of os.path.normpath that converts \ to / on Unix.

Parameters
path	a path
Returns
the normalized path

def open_file(path, mode='r'): (source) ¶

Opens the given file for reading or writing.

Example usage:

import fiftyone.core.storage as fos

with fos.open_file("/tmp/file.txt", "w") as f:
    f.write("Hello, world!")

with fos.open_file("/tmp/file.txt", "r") as f:
    print(f.read())

Parameters
path	the path
mode:"r"	the mode. Supported values are `("r", "rb", "w", "wb")`
Returns
an open file-like object

def open_files(paths, mode='r', skip_failures=False, progress=None): (source) ¶

Opens the given files for reading or writing.

Parameters
paths	a list of paths
mode:"r"	the mode. Supported values are `("r", "rb", "w", "wb")`
skip_failures:`False`	whether to gracefully continue without raising an error if an operation fails
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead
Returns
a list of open file-like objects

def read_file(path, binary=False): (source) ¶

Reads the file.

Parameters
path	the filepath
binary:`False`	whether to read the file in binary mode
Returns
the file contents

def read_files(paths, binary=False, skip_failures=False, progress=None): (source) ¶

Reads the specified files into memory.

Parameters
paths	a list of filepaths
binary:`False`	whether to read the files in binary mode
skip_failures:`False`	whether to gracefully continue without raising an error if an operation fails
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead
Returns
a list of file contents

def read_json(path): (source) ¶

Reads a JSON file.

Parameters
path	the filepath
Returns
the JSON data

def read_ndjson(path): (source) ¶

Reads an NDJSON file.

Parameters
path	the filepath
Returns
a list of JSON dicts

def read_yaml(path): (source) ¶

Reads a YAML file.

Parameters
path	the filepath
Returns
a list of JSON dicts

def realpath(path): (source) ¶

Converts the given path to absolute, resolving symlinks and relative path indicators such as . and ...

Parameters
path	the filepath
Returns
the resolved path

def run(fcn, tasks, return_results=True, num_workers=None, progress=None): (source) ¶

Applies the given function to each element of the given tasks.

Parameters
fcn	a function that accepts a single argument
tasks	an iterable of function arguments
return_results:`True`	whether to return the function results
num_workers:`None`	a suggested number of threads to use
progress:`None`	whether to render a progress bar (True/False), use the default value `fiftyone.config.show_progress_bars` (None), or a progress callback function to invoke instead
Returns
the list of function outputs, or None if `return_results == False`

def sep(path): (source) ¶

Returns the path separator for the given path.

Parameters
path	the filepath
Returns
the path separator

def split_prefix(path): (source) ¶

Splits the file system prefix from the given path.

The prefix for local paths is "".

Example usages:

import fiftyone.core.storage as fos

fos.split_prefix("/path/to/file")       # ('', '/path/to/file')
fos.split_prefix("a/file")              # ('', 'a/file')