Step 2: Analyzing with Model Evaluation Panel¶

In our last step we showed some basic ways on how to evaluate models. In this step, we will show how to take it even further with the Model Eval Panel in the app. With the Model Eval Panel you can:

• See all evaluation runs on a dataset
• View summary statistics of each run
• Filter dataset based on FP, TP, and more
• Analyze class-wise evaluation metrics and filter based on them
• View confusion matrices and histograms of evaluation results

Let's hop into an example to see how to get started!

In [1]:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# View summary info about the dataset
print(dataset)

import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") # View summary info about the dataset print(dataset)

Dataset already downloaded
Loading 'quickstart'
 100% |█████████████████| 200/200 [1.1s elapsed, 0s remaining, 174.4 samples/s]         
Dataset 'quickstart' created
Name:        quickstart
Media type:  image
Num samples: 200
Persistent:  False
Tags:        []
Sample fields:
    id:               fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    ground_truth:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    uniqueness:       fiftyone.core.fields.FloatField
    predictions:      fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)

Let's quickly rerun evaluation in case we do not have it from the previous step:

In [2]:

results = dataset.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
    compute_mAP=True,
)

results = dataset.evaluate_detections( "predictions", gt_field="ground_truth", eval_key="eval", compute_mAP=True, )

Evaluating detections...
 100% |█████████████████| 200/200 [6.9s elapsed, 0s remaining, 19.1 samples/s]      
Performing IoU sweep...
 100% |█████████████████| 200/200 [2.2s elapsed, 0s remaining, 84.6 samples/s]       

In [3]:

from fiftyone import ViewField as F

# Only contains detections with confidence >= 0.75
high_conf_view = dataset.filter_labels("predictions", F("confidence") > 0.75, only_matches=False)

results = high_conf_view.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval_high_conf",
    compute_mAP=True,
)

from fiftyone import ViewField as F # Only contains detections with confidence >= 0.75 high_conf_view = dataset.filter_labels("predictions", F("confidence") > 0.75, only_matches=False) results = high_conf_view.evaluate_detections( "predictions", gt_field="ground_truth", eval_key="eval_high_conf", compute_mAP=True, )

Evaluating detections...
 100% |█████████████████| 200/200 [1.3s elapsed, 0s remaining, 140.1 samples/s]         
Performing IoU sweep...
 100% |█████████████████| 200/200 [950.3ms elapsed, 0s remaining, 210.5 samples/s]      

Now we can open up our dataset and view are results in the Model Eval Panel! I recommend opening the app in browser for the best experience at http://localhost:5151/!

In [4]:

session = fo.launch_app(dataset)

session = fo.launch_app(dataset)

Welcome to

███████╗██╗███████╗████████╗██╗   ██╗ ██████╗ ███╗   ██╗███████╗
██╔════╝██║██╔════╝╚══██╔══╝╚██╗ ██╔╝██╔═══██╗████╗  ██║██╔════╝
█████╗  ██║█████╗     ██║    ╚████╔╝ ██║   ██║██╔██╗ ██║█████╗
██╔══╝  ██║██╔══╝     ██║     ╚██╔╝  ██║   ██║██║╚██╗██║██╔══╝
██║     ██║██║        ██║      ██║   ╚██████╔╝██║ ╚████║███████╗
╚═╝     ╚═╝╚═╝        ╚═╝      ╚═╝    ╚═════╝ ╚═╝  ╚═══╝╚══════╝ v1.4.0

If you're finding FiftyOne helpful, here's how you can get involved:

|
|  ⭐⭐⭐ Give the project a star on GitHub ⭐⭐⭐
|  https://github.com/voxel51/fiftyone
|
|  🚀🚀🚀 Join the FiftyOne Discord community 🚀🚀🚀
|  https://community.voxel51.com/
|

Now we can begin to explore the Model Eval Panel! Start by clicking the "+" button above the samples grid and selecting "Model Evaluaton". Once its open you will see all of your available evals:

Select the evaluation of your choice and begin to explore all the different views of the Model Eval Panel! See summary statistics, individual class performance, confusion matrices and more!

You can even compare multiple evals to each other inside the Model Eval Panel! At the top of an open eval, select "Compare against" to select your second eval. From there you can now see all the differences between your two evaluation runs!

The Model Eval Panel is also completely interactive! Click on any of the histograms or confusion matrices to pull up a view of the data you are looking at!