YOLOv7 model with Cityscapes dataset

This project utilizes the YOLOv7 model to perform object detection on Cityscapes dataset. The cityscapes dataset is a large-scale dataset that stands as one of the standard ADAS benchmarks used for multiple vision-related tasks.

Tensorleap helps to explore the latent space of a dataset and find new and important insights. It can also be used to find unlabeled clusters or miss-labeling in the dataset in the most efficient way.

The following plot illustrates a population exploration map, depicting the similarity among samples based on the latent space of a trained model. This map is constructed using the extracted features of the model.

Upon conducting additional analysis, it was found that when the number of small bounding boxes decreases, the object detection loss increases.

 

Upon conducting further analysis, it has been discovered that images containing cars exhibit a high loss. Subsequent research has revealed that the algorithm is actually recognizing cars accurately, but the ground truth (gt) labels are incorrect. The cars in the images are mislabeled. To address this issue, it is essential to correct the labels for the cars in order to improve the performance of the algorithm.

 

We can observe that the class “motorcycle” is underrepresented in the data. The majority of images have zero labeled ground truth bounding boxes for motorcycles. To address this, we should investigate if there is any mislabeling or consider adding more images containing motorcycles to improve the representation of this class in the dataset.

Cluster number 14 of k-means clusters characterized in dark ambiance images, featuring a road surrounded by vegetation, while cars are the primary objects present in the image.

Within this section, we delve into the examination of features that impact the model’s predictions. Tensorleap automatically generates a heatmap that quantifies the effect different features have on loss/ specific prediction. In the image depicted below, the heat map displays warmer areas, indicating regions in the image where features are more prominent, and thereby increasing the likelihood that the object in those areas is a building.

After implementing the Intersection over Union (IoU) metric for each category’s class_id, it becomes evident that the IoU scores for the ‘road’ category are consistently high across most of the images. This suggests that the model’s performance in detecting and accurately segmenting the ‘road’ class is notably successful in many instances.

We notice the model didn’t learn ‘bus’ class properly. In the dashboard, we plot count of bus instances per image vs the amount of bus instance predictions per image. The model never outputs bus class. Additionally, the “bus” prediction heatmaps do not highlight any relevant pixel areas. We fine-tuned the model and see the new model is able to detect the bus objects. We can support it quantitatively from the metrics dashboard and qualitatively from the heatmaps, compared to the original model:

PE: we color based on bus prediction and observe model outputs 0 for all samples: 

Figure bus prediction vs bus GT amount: original model vs fine-tuned model: 

“Bus” Prediction Heatmaps: original model vs fine-tuned model:   

This quick start guide will walk you through the steps to get started with this example repository project.

Before you begin, ensure that you have the following prerequisites installed:

• Python (version 3.7 or higher).
• Poetry.
• Tensorleap platform access. To request a free trial click here.
• Tensorleap CLI.

with curl:

curl -s https://raw.githubusercontent.com/tensorleap/leap-cli/master/install.sh | bash

To login to Tensorleap:

tensorleap auth login [api key] [api url].

• API Key is your Tensorleap token (see how to generate a CLI token in the section below).
• API URL is your Tensorleap environment URL: https://api.CLIENT_NAME.tensorleap.ai/api/v2

How To Generate CLI Token from the UI

1. Login to the platform in ‘CLIENT_NAME.tensorleap.ai’
2. Scroll down to the bottom of the Resources Management page, then click GENERATE CLI TOKEN in the bottom-left corner.
3. Once a CLI token is generated, just copy the whole text and paste it into your shell.

To deploy your local changes:

leap project push

Tensorleap files in the repository include leap_binder.py and leap.yaml. The files consist of the required configurations to make the code integrate with the Tensorleap engine:

leap.yaml

leap.yaml file is configured to a dataset in your Tensorleap environment and is synced to the dataset saved in the environment.

For any additional file being used, we add its path under include parameter:

include:
    - leap_binder.py
    - cityscapes_od/configs.py
    - [...]

leap_binder.py file

leap_binder.py configures all binding functions used to bind to Tensorleap engine. These are the functions used to evaluate and train the model, visualize the variables, and enrich the analysis with external metadata variables

To test the system we can run leap_test.py file using poetry:

poetry run test

This file will execute several tests on leap_binder.py script to assert that the implemented binding functions: preprocess, encoders, metadata, etc., run smoothly.

For further explanation please refer to the docs