Convert Dataset Formats¶

This recipe demonstrates how to use FiftyOne to convert datasets on disk between common formats.

Setup¶

If you haven't already, install FiftyOne:

In [ ]:

pip install fiftyone

pip install fiftyone

This notebook contains bash commands. To run it as a notebook, you must install the Jupyter bash kernel via the command below.

Alternatively, you can just copy + paste the code blocks into your shell.

In [1]:

pip install bash_kernel
python -m bash_kernel.install

pip install bash_kernel python -m bash_kernel.install

In this recipe we'll use the FiftyOne Dataset Zoo to download some open source datasets to work with.

Specifically, we'll need TensorFlow and TensorFlow Datasets installed to access the datasets:

In [2]:

pip install tensorflow tensorflow-datasets

pip install tensorflow tensorflow-datasets

Download datasets¶

Download the test split of the CIFAR-10 dataset from the FiftyOne Dataset Zoo using the command below:

In [1]:

# Download the test split of CIFAR-10
fiftyone zoo datasets download cifar10 --split test

# Download the test split of CIFAR-10 fiftyone zoo datasets download cifar10 --split test

Downloading split 'test' to '~/fiftyone/cifar10/test'
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz
170500096it [00:04, 35887670.65it/s]                                            
Extracting ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download
 100% |███| 10000/10000 [5.2s elapsed, 0s remaining, 1.8K samples/s]      
Dataset info written to '~/fiftyone/cifar10/info.json'

Download the validation split of the KITTI dataset from the FiftyOne Dataset Zoo using the command below:

In [1]:

# Download the validation split of KITTI
fiftyone zoo datasets download kitti --split validation

# Download the validation split of KITTI fiftyone zoo datasets download kitti --split validation

Split 'validation' already downloaded

The fiftyone convert command¶

The FiftyOne CLI provides a number of utilities for importing and exporting datasets in a variety of common (or custom) formats.

Specifically, the fiftyone convert command provides a convenient way to convert datasets on disk between formats by specifying the fiftyone.types.Dataset type of the input and desired output.

FiftyOne provides a collection of builtin types that you can use to read/write datasets in common formats out-of-the-box:

Dataset format	Import Supported?	Export Supported?	Conversion Supported?
ImageDirectory	✓	✓	✓
VideoDirectory	✓	✓	✓
FiftyOneImageClassificationDataset	✓	✓	✓
ImageClassificationDirectoryTree	✓	✓	✓
TFImageClassificationDataset	✓	✓	✓
FiftyOneImageDetectionDataset	✓	✓	✓
COCODetectionDataset	✓	✓	✓
VOCDetectionDataset	✓	✓	✓
KITTIDetectionDataset	✓	✓	✓
YOLODataset	✓	✓	✓
TFObjectDetectionDataset	✓	✓	✓
CVATImageDataset	✓	✓	✓
CVATVideoDataset	✓	✓	✓
FiftyOneImageLabelsDataset	✓	✓	✓
FiftyOneVideoLabelsDataset	✓	✓	✓
BDDDataset	✓	✓	✓

In addition, you can define your own custom dataset types to read/write datasets in your own formats.

The usage of the fiftyone convert command is as follows:

In [1]:

fiftyone convert -h

fiftyone convert -h

usage: fiftyone convert [-h] --input-type INPUT_TYPE --output-type OUTPUT_TYPE
                        [--input-dir INPUT_DIR]
                        [--input-kwargs KEY=VAL [KEY=VAL ...]]
                        [--output-dir OUTPUT_DIR]
                        [--output-kwargs KEY=VAL [KEY=VAL ...]] [-o]

Convert datasets on disk between supported formats.

    Examples::

        # Convert an image classification directory tree to TFRecords format
        fiftyone convert \
            --input-dir /path/to/image-classification-directory-tree \
            --input-type fiftyone.types.ImageClassificationDirectoryTree \
            --output-dir /path/for/tf-image-classification-dataset \
            --output-type fiftyone.types.TFImageClassificationDataset

        # Convert a COCO detection dataset to CVAT image format
        fiftyone convert \
            --input-dir /path/to/coco-detection-dataset \
            --input-type fiftyone.types.COCODetectionDataset \
            --output-dir /path/for/cvat-image-dataset \
            --output-type fiftyone.types.CVATImageDataset

        # Perform a customized conversion via optional kwargs
        fiftyone convert \
            --input-dir /path/to/coco-detection-dataset \
            --input-type fiftyone.types.COCODetectionDataset \
            --input-kwargs max_samples=100 shuffle=True \
            --output-dir /path/for/cvat-image-dataset \
            --output-type fiftyone.types.TFObjectDetectionDataset \
            --output-kwargs force_rgb=True \
            --overwrite

optional arguments:
  -h, --help            show this help message and exit
  --input-dir INPUT_DIR
                        the directory containing the dataset
  --input-kwargs KEY=VAL [KEY=VAL ...]
                        additional keyword arguments for `fiftyone.utils.data.convert_dataset(..., input_kwargs=)`
  --output-dir OUTPUT_DIR
                        the directory to which to write the output dataset
  --output-kwargs KEY=VAL [KEY=VAL ...]
                        additional keyword arguments for `fiftyone.utils.data.convert_dataset(..., output_kwargs=)`
  -o, --overwrite       whether to overwrite an existing output directory

required arguments:
  --input-type INPUT_TYPE
                        the fiftyone.types.Dataset type of the input dataset
  --output-type OUTPUT_TYPE
                        the fiftyone.types.Dataset type to output

Convert CIFAR-10 dataset¶

When you downloaded the test split of the CIFAR-10 dataset above, it was written to disk as a dataset in fiftyone.types.FiftyOneImageClassificationDataset format.

You can verify this by printing information about the downloaded dataset:

In [6]:

fiftyone zoo datasets info cifar10

fiftyone zoo datasets info cifar10

***** Dataset description *****
The CIFAR-10 dataset consists of 60000 32 x 32 color images in 10
    classes, with 6000 images per class. There are 50000 training images and
    10000 test images.

    Dataset size:
        132.40 MiB

    Source:
        https://www.cs.toronto.edu/~kriz/cifar.html
    
***** Supported splits *****
test, train

***** Dataset location *****
~/fiftyone/cifar10

***** Dataset info *****
{
    "name": "cifar10",
    "zoo_dataset": "fiftyone.zoo.datasets.torch.CIFAR10Dataset",
    "dataset_type": "fiftyone.types.dataset_types.FiftyOneImageClassificationDataset",
    "num_samples": 10000,
    "downloaded_splits": {
        "test": {
            "split": "test",
            "num_samples": 10000
        }
    },
    "classes": [
        "airplane",
        "automobile",
        "bird",
        "cat",
        "deer",
        "dog",
        "frog",
        "horse",
        "ship",
        "truck"
    ]
}

The snippet below uses fiftyone convert to convert the test split of the CIFAR-10 dataset to fiftyone.types.ImageClassificationDirectoryTree format, which stores classification datasets on disk in a directory tree structure with images organized per-class:

<dataset_dir>
├── <classA>/
│   ├── <image1>.<ext>
│   ├── <image2>.<ext>
│   └── ...
├── <classB>/
│   ├── <image1>.<ext>
│   ├── <image2>.<ext>
│   └── ...
└── ...

In [7]:

INPUT_DIR=$(fiftyone zoo datasets find cifar10 --split test)
OUTPUT_DIR=/tmp/fiftyone/cifar10-dir-tree

fiftyone convert \
    --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageClassificationDataset \
    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.ImageClassificationDirectoryTree

INPUT_DIR=$(fiftyone zoo datasets find cifar10 --split test) OUTPUT_DIR=/tmp/fiftyone/cifar10-dir-tree fiftyone convert \ --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageClassificationDataset \ --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.ImageClassificationDirectoryTree

Loading dataset from '~/fiftyone/cifar10/test'
Input format 'fiftyone.types.dataset_types.FiftyOneImageClassificationDataset'
 100% |███| 10000/10000 [4.2s elapsed, 0s remaining, 2.4K samples/s]      
Import complete
Exporting dataset to '/tmp/fiftyone/cifar10-dir-tree'
Export format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'
 100% |███| 10000/10000 [6.2s elapsed, 0s remaining, 1.7K samples/s]        
Export complete

Let's verify that the conversion happened as expected:

In [8]:

ls -lah /tmp/fiftyone/cifar10-dir-tree/

ls -lah /tmp/fiftyone/cifar10-dir-tree/

total 0
drwxr-xr-x    12 voxel51  wheel   384B Jul 14 11:08 .
drwxr-xr-x     3 voxel51  wheel    96B Jul 14 11:08 ..
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 airplane
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 automobile
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 bird
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 cat
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 deer
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 dog
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 frog
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 horse
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 ship
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 truck

In [9]:

ls -lah /tmp/fiftyone/cifar10-dir-tree/airplane/ | head

ls -lah /tmp/fiftyone/cifar10-dir-tree/airplane/ | head

total 8000
drwxr-xr-x  1002 voxel51  wheel    31K Jul 14 11:08 .
drwxr-xr-x    12 voxel51  wheel   384B Jul 14 11:08 ..
-rw-r--r--     1 voxel51  wheel   1.2K Jul 14 11:23 000004.jpg
-rw-r--r--     1 voxel51  wheel   1.1K Jul 14 11:23 000011.jpg
-rw-r--r--     1 voxel51  wheel   1.1K Jul 14 11:23 000022.jpg
-rw-r--r--     1 voxel51  wheel   1.3K Jul 14 11:23 000028.jpg
-rw-r--r--     1 voxel51  wheel   1.2K Jul 14 11:23 000045.jpg
-rw-r--r--     1 voxel51  wheel   1.2K Jul 14 11:23 000053.jpg
-rw-r--r--     1 voxel51  wheel   1.3K Jul 14 11:23 000075.jpg

Now let's convert the classification directory tree to TFRecords format!

In [10]:

INPUT_DIR=/tmp/fiftyone/cifar10-dir-tree
OUTPUT_DIR=/tmp/fiftyone/cifar10-tfrecords

fiftyone convert \
    --input-dir ${INPUT_DIR} --input-type fiftyone.types.ImageClassificationDirectoryTree \
    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.TFImageClassificationDataset

INPUT_DIR=/tmp/fiftyone/cifar10-dir-tree OUTPUT_DIR=/tmp/fiftyone/cifar10-tfrecords fiftyone convert \ --input-dir ${INPUT_DIR} --input-type fiftyone.types.ImageClassificationDirectoryTree \ --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.TFImageClassificationDataset

Loading dataset from '/tmp/fiftyone/cifar10-dir-tree'
Input format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'
 100% |███| 10000/10000 [4.0s elapsed, 0s remaining, 2.5K samples/s]      
Import complete
Exporting dataset to '/tmp/fiftyone/cifar10-tfrecords'
Export format 'fiftyone.types.dataset_types.TFImageClassificationDataset'
   0% ||--|     1/10000 [23.2ms elapsed, 3.9m remaining, 43.2 samples/s] 2020-07-14 11:24:15.187387: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-07-14 11:24:15.201384: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f83df428f60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-14 11:24:15.201405: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
 100% |███| 10000/10000 [8.2s elapsed, 0s remaining, 1.3K samples/s]        
Export complete

Let's verify that the conversion happened as expected:

In [11]:

ls -lah /tmp/fiftyone/cifar10-tfrecords

ls -lah /tmp/fiftyone/cifar10-tfrecords

total 29696
drwxr-xr-x  3 voxel51  wheel    96B Jul 14 11:24 .
drwxr-xr-x  4 voxel51  wheel   128B Jul 14 11:24 ..
-rw-r--r--  1 voxel51  wheel    14M Jul 14 11:24 tf.records

Convert KITTI dataset¶

When you downloaded the validation split of the KITTI dataset above, it was written to disk as a dataset in fiftyone.types.FiftyOneImageDetectionDataset format.

You can verify this by printing information about the downloaded dataset:

In [12]:

fiftyone zoo datasets info kitti

fiftyone zoo datasets info kitti

***** Dataset description *****
KITTI contains a suite of vision tasks built using an autonomous
    driving platform.

    The full benchmark contains many tasks such as stereo, optical flow, visual
    odometry, etc. This dataset contains the object detection dataset,
    including the monocular images and bounding boxes. The dataset contains
    7481 training images annotated with 3D bounding boxes. A full description
    of the annotations can be found in the README of the object development kit
    on the KITTI homepage.

    Dataset size:
        5.27 GiB

    Source:
        http://www.cvlibs.net/datasets/kitti
    
***** Supported splits *****
test, train, validation

***** Dataset location *****
~/fiftyone/kitti

***** Dataset info *****
{
    "name": "kitti",
    "zoo_dataset": "fiftyone.zoo.datasets.tf.KITTIDataset",
    "dataset_type": "fiftyone.types.dataset_types.FiftyOneImageDetectionDataset",
    "num_samples": 423,
    "downloaded_splits": {
        "validation": {
            "split": "validation",
            "num_samples": 423
        }
    },
    "classes": [
        "Car",
        "Van",
        "Truck",
        "Pedestrian",
        "Person_sitting",
        "Cyclist",
        "Tram",
        "Misc"
    ]
}

The snippet below uses fiftyone convert to convert the test split of the CIFAR-10 dataset to fiftyone.types.COCODetectionDataset format, which writes the dataset to disk with annotations in COCO format.

In [13]:

INPUT_DIR=$(fiftyone zoo datasets find kitti --split validation)
OUTPUT_DIR=/tmp/fiftyone/kitti-coco

fiftyone convert \
    --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageDetectionDataset \
    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.COCODetectionDataset

INPUT_DIR=$(fiftyone zoo datasets find kitti --split validation) OUTPUT_DIR=/tmp/fiftyone/kitti-coco fiftyone convert \ --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageDetectionDataset \ --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.COCODetectionDataset

Loading dataset from '~/fiftyone/kitti/validation'
Input format 'fiftyone.types.dataset_types.FiftyOneImageDetectionDataset'
 100% |███████| 423/423 [1.2s elapsed, 0s remaining, 351.0 samples/s]         
Import complete
Exporting dataset to '/tmp/fiftyone/kitti-coco'
Export format 'fiftyone.types.dataset_types.COCODetectionDataset'
 100% |███████| 423/423 [4.4s elapsed, 0s remaining, 96.1 samples/s]       
Export complete

Let's verify that the conversion happened as expected:

In [14]:

ls -lah /tmp/fiftyone/kitti-coco/

ls -lah /tmp/fiftyone/kitti-coco/

total 880
drwxr-xr-x    4 voxel51  wheel   128B Jul 14 11:24 .
drwxr-xr-x    5 voxel51  wheel   160B Jul 14 11:24 ..
drwxr-xr-x  425 voxel51  wheel    13K Jul 14 11:24 data
-rw-r--r--    1 voxel51  wheel   437K Jul 14 11:24 labels.json

In [15]:

ls -lah /tmp/fiftyone/kitti-coco/data | head

ls -lah /tmp/fiftyone/kitti-coco/data | head

total 171008
drwxr-xr-x  425 voxel51  wheel    13K Jul 14 11:24 .
drwxr-xr-x    4 voxel51  wheel   128B Jul 14 11:24 ..
-rw-r--r--    1 voxel51  wheel   195K Jul 14 11:24 000001.jpg
-rw-r--r--    1 voxel51  wheel   191K Jul 14 11:24 000002.jpg
-rw-r--r--    1 voxel51  wheel   167K Jul 14 11:24 000003.jpg
-rw-r--r--    1 voxel51  wheel   196K Jul 14 11:24 000004.jpg
-rw-r--r--    1 voxel51  wheel   224K Jul 14 11:24 000005.jpg
-rw-r--r--    1 voxel51  wheel   195K Jul 14 11:24 000006.jpg
-rw-r--r--    1 voxel51  wheel   177K Jul 14 11:24 000007.jpg

In [19]:

cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | head -20
echo "..."
cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | tail -20

cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | head -20 echo "..." cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | tail -20

{
    "info": {
        "year": "",
        "version": "",
        "description": "Exported from FiftyOne",
        "contributor": "",
        "url": "https://voxel51.com/fiftyone",
        "date_created": "2020-07-14T11:24:40"
    },
    "licenses": [],
    "categories": [
        {
            "id": 0,
            "name": "Car",
            "supercategory": "none"
        },
        {
            "id": 1,
            "name": "Cyclist",
            "supercategory": "none"
...
            "area": 4545.8,
            "segmentation": null,
            "iscrowd": 0
        },
        {
            "id": 3196,
            "image_id": 422,
            "category_id": 3,
            "bbox": [
                367.2,
                107.3,
                36.2,
                105.2
            ],
            "area": 3808.2,
            "segmentation": null,
            "iscrowd": 0
        }
    ]
}

Now let's convert from COCO format to CVAT Image format format!

In [20]:

INPUT_DIR=/tmp/fiftyone/kitti-coco
OUTPUT_DIR=/tmp/fiftyone/kitti-cvat

fiftyone convert \
    --input-dir ${INPUT_DIR} --input-type fiftyone.types.COCODetectionDataset \
    --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.CVATImageDataset

INPUT_DIR=/tmp/fiftyone/kitti-coco OUTPUT_DIR=/tmp/fiftyone/kitti-cvat fiftyone convert \ --input-dir ${INPUT_DIR} --input-type fiftyone.types.COCODetectionDataset \ --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.CVATImageDataset

Loading dataset from '/tmp/fiftyone/kitti-coco'
Input format 'fiftyone.types.dataset_types.COCODetectionDataset'
 100% |███████| 423/423 [2.0s elapsed, 0s remaining, 206.4 samples/s]      
Import complete
Exporting dataset to '/tmp/fiftyone/kitti-cvat'
Export format 'fiftyone.types.dataset_types.CVATImageDataset'
 100% |███████| 423/423 [1.3s elapsed, 0s remaining, 323.7 samples/s]         
Export complete

Let's verify that the conversion happened as expected:

In [21]:

ls -lah /tmp/fiftyone/kitti-cvat

ls -lah /tmp/fiftyone/kitti-cvat

total 584
drwxr-xr-x    4 voxel51  wheel   128B Jul 14 11:25 .
drwxr-xr-x    6 voxel51  wheel   192B Jul 14 11:25 ..
drwxr-xr-x  425 voxel51  wheel    13K Jul 14 11:25 data
-rw-r--r--    1 voxel51  wheel   289K Jul 14 11:25 labels.xml

In [22]:

cat /tmp/fiftyone/kitti-cvat/labels.xml | head -20
echo "..."
cat /tmp/fiftyone/kitti-cvat/labels.xml | tail -20

cat /tmp/fiftyone/kitti-cvat/labels.xml | head -20 echo "..." cat /tmp/fiftyone/kitti-cvat/labels.xml | tail -20

<?xml version="1.0" encoding="utf-8"?>
<annotations>
    <version>1.1</version>
    <meta>
        <task>
            <size>423</size>
            <mode>annotation</mode>
            <labels>
                <label>
                    <name>Car</name>
                    <attributes>
                    </attributes>
                </label>
                <label>
                    <name>Cyclist</name>
                    <attributes>
                    </attributes>
                </label>
                <label>
                    <name>Misc</name>
...
        <box label="Pedestrian" xtl="360" ytl="116" xbr="402" ybr="212">
        </box>
        <box label="Pedestrian" xtl="396" ytl="120" xbr="430" ybr="212">
        </box>
        <box label="Pedestrian" xtl="413" ytl="112" xbr="483" ybr="212">
        </box>
        <box label="Pedestrian" xtl="585" ytl="80" xbr="646" ybr="215">
        </box>
        <box label="Pedestrian" xtl="635" ytl="94" xbr="688" ybr="212">
        </box>
        <box label="Pedestrian" xtl="422" ytl="85" xbr="469" ybr="210">
        </box>
        <box label="Pedestrian" xtl="457" ytl="93" xbr="520" ybr="213">
        </box>
        <box label="Pedestrian" xtl="505" ytl="101" xbr="548" ybr="206">
        </box>
        <box label="Pedestrian" xtl="367" ytl="107" xbr="403" ybr="212">
        </box>
    </image>
</annotations>

Cleanup¶

You can cleanup the files generated by this recipe by running the command below:

In [23]:

rm -rf /tmp/fiftyone

rm -rf /tmp/fiftyone