How to do Object Detection in Kolena - Task Doc Template #583

marko-polo-cheno · 2024-04-26T21:24:22Z

Linked issue(s)

Fixes KOL-5750

What change does this PR introduce and why?

The beginning of a series of How-to guides in Kolena. This one is for object detection.

Will need thoughts on document organization. From this thread, I think all future How-to's would live under the quickstart page. Quickstart > How to test Object Detection models

codecov · 2024-04-26T22:12:41Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.61%. Comparing base (7c8e8c9) to head (4bc189c).

Additional details and impacted files

@@            Coverage Diff             @@
##            trunk     #583      +/-   ##
==========================================
- Coverage   94.53%   93.61%   -0.92%     
==========================================
  Files          86       86              
  Lines        5359     5359              
  Branches      782      782              
==========================================
- Hits         5066     5017      -49     
- Misses        217      256      +39     
- Partials       76       86      +10

Flag	Coverage Δ
integration	`76.17% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

marko-polo-cheno · 2024-05-24T12:33:02Z

With the task-metrics page, there could be redundancies cleared up within the OD guide. Would love to hear thoughts on what should be added/removed to a How-To guide (as this would likely become a template to other guides), particularly for where the How-To guides would live in the table of contents.

mkdocs.yml

y27choi · 2024-05-27T18:11:19Z

docs/dataset/object-detection.md

+inferences for an image are the labeled bounding boxes having confidence scores (the model's certainty of
+correctness), as if the model annotated the image.
+
+??? info "Evaluation Metrics for Object Detection Models: Precision, Recall, F1-Score, and AP"


Does this need to be in a collapsible section? It's a bit odd to me that I don't see any list of metrics when I come to the metrics section of the docs. I normally use these collapsible callouts when there are information that doesn't need to be but nice to have on the page.

It's fine to keep some details in the callout for further learning but we should at least list the metrics in the main paragraph of the section. Right now the paragraph just looks incomplete as it doesn't really talk about the metrics.

docs/dataset/object-detection.md

y27choi · 2024-05-27T18:36:41Z

docs/dataset/object-detection.md

+</figure>
+
+Rather than uploading raw inferences with metrics yourself, if you upload the
+[TP / FP / FN](../metrics/tp-fp-fn-tn.md) results from your own [bounding box matcher](../metrics/geometry-matching.md)


Aside from using their own bounding box matcher, we can also mention our util functions that can compute the necessary model results columns from raw inference.

compute_object_detection_results

upload_object_detection_results

I still think we should surface these two options (1. use your own bbox matcher and give us the TP / FP / FN results and 2. upload your raw inferences and use Kolena's bbox matcher) very clearly under the step #3 section.

marko-polo-cheno · 2024-06-19T17:29:26Z

One thing I am on the fence about is whether or not formatting of data needs to more in depth. If so, would that be within the format dataset > CV pages, or within the how-to-[task] pages?

@y27choi What if it was very code-snippet example-based to reduce the overhead of linking to example code?

y27choi

@munkyshi I went over it again and left some comments to address. I think we can improve the "Using Kolena to Evaluate Object Detection Models" section to be more clear but I'm curious to hear your thoughts on this. If you are new to Kolena, does this section make sense?

y27choi · 2024-07-09T19:51:46Z

docs/dataset/object-detection.md

+[`List[ScoredLabeledBoundingBox]`](../reference/annotation.md#kolena.annotation.ScoredLabeledBoundingBox).
+Note that a [`ScoredLabeledBoundingBox`](../reference/annotation.md#kolena.annotation.ScoredLabeledBoundingBox)
+has an extra `score` attribute for the confidence of the model:
+```


Using this format as an example could be overwhelming for users. Is this example necessary?

y27choi · 2024-07-09T19:55:50Z

docs/dataset/object-detection.md

+{""top_left"": ... ""label"": ""car"", ""score"": 0.94, ""data_type"":""ANNOTATION/BOUNDING_BOX""}
+```
+
+The expected CSV format for model results should look like the CSV below:


Doesn't this conflict with this API doc? It's confusing to me whether I should have a column called raw_inferences or TP / FP / FN or both. Just like the step #1, it might bet better to just point to the formatting doc.

y27choi · 2024-07-09T19:59:48Z

docs/dataset/object-detection.md

+For this example, we will upload object detection model results
+of [YOLO X](https://github.com/Megvii-BaseDetection/YOLOX) for the single class object detection task in the web app,
+and upload the results of [Faster R-CNN](https://github.com/facebookresearch/Detectron) for the
+multiclass object detection task through the SDK.


It seems like it now includes the uploading of both models in the SDK tab. I believe the intention here is to show examples of both single class and multiclass OD but I think it might be worth focusing on a single class in this example and potentially adding a separate section in future for multiclass for things like "how to view per-class metrics? / how to plot different graphs per class?"

Agreed -- let's keep it simple.

y27choi · 2024-07-09T20:02:58Z

docs/dataset/object-detection.md

+</figure>
+
+Rather than uploading raw inferences with metrics yourself, if you upload the
+[TP / FP / FN](../metrics/tp-fp-fn-tn.md) results from your own [bounding box matcher](../metrics/geometry-matching.md)


I still think we should surface these two options (1. use your own bbox matcher and give us the TP / FP / FN results and 2. upload your raw inferences and use Kolena's bbox matcher) very clearly under the step #3 section.

munkyshi · 2024-07-11T03:42:14Z

docs/dataset/object-detection.md

+    It's also important to note down the evaluation configuration of the model being uploaded, for example:
+    `iou_threshold: 0.5` or `iou_threshold: 0.3`.


It feels like this part isn't clear enough -- we should either drop it for brevity, or flesh it out (in this example, there should be an eval config with iou_threshold, min_confidence_score, and threshold_strategy -- we don't really touch on them outside of the short blurb you have for iou_threshold)

munkyshi · 2024-07-11T03:44:05Z

docs/dataset/object-detection.md

+
+### Step 2: Upload your Object Detection Dataset
+
+Model evaluation on Kolena starts with datasets. Upload your dataset of datapoints (e.g. locators to


datasets can probably link to https://docs.kolena.com/dataset/core-concepts/#dataset

docs/dataset/object-detection.md

munkyshi · 2024-10-11T19:40:26Z

docs/dataset/object-detection.md

+    [uploading object detection model results](../dataset/advanced-usage/dataset-formatting/computer-vision.md#uploading-model-results).
+
+    ```shell
+    poetry run python3 object_detection_2d/upload_results.py faster_rcnn


Suggested change

poetry run python3 object_detection_2d/upload_results.py faster_rcnn

uv run python3 object_detection_2d/upload_results.py faster_rcnn

munkyshi · 2024-10-11T19:41:19Z

docs/dataset/object-detection.md

+    [Open Images](https://paperswithcode.com/dataset/open-images-v7),
+    and [ImageNet](https://paperswithcode.com/dataset/imagenet).
+
+## Object Detection Model Evaluation


Probably best to not have OD in every header, given that the page is for OD already

Suggested change

## Object Detection Model Evaluation

## Model Evaluation

munkyshi · 2024-10-11T19:41:26Z

docs/dataset/object-detection.md

+It is important to understand some core performance metrics and plots that are typically seen when
+evaluating object detection models.
+
+### Object Detection Metrics


Suggested change

### Object Detection Metrics

### Metrics

munkyshi · 2024-10-11T19:41:36Z

docs/dataset/object-detection.md

+    Read the [averaging methods](../metrics/averaging-methods.md) guide if you are not familiar with "macro"
+    and "micro" terminology.
+
+### Object Detection Plots


Suggested change

### Object Detection Plots

### Plots

munkyshi

A few small comments, but approving to unblock merge after they are addressed.

create new OD task doc

bba561c

marko-polo-cheno requested a review from a team as a code owner April 26, 2024 21:24

marko-polo-cheno requested review from y27choi and gordonhart May 24, 2024 12:27

y27choi reviewed May 27, 2024

View reviewed changes

marko-polo-cheno added 6 commits June 17, 2024 10:55

Merge branch 'trunk' into create-how-to-OD-doc

4809f30

fix 404s

dd29c22

stash

ea412a8

Merge branch 'trunk' into create-how-to-OD-doc

ec9c3a1

update to new links and sync with new docs

6a343af

proofread

4bc189c

marko-polo-cheno requested review from y27choi and mkaramlou June 19, 2024 14:34

y27choi reviewed Jul 9, 2024

View reviewed changes

munkyshi reviewed Jul 11, 2024

View reviewed changes

munkyshi mentioned this pull request Jul 11, 2024

Extend OD matching functions to accept an optional required_match_fields parameter #643

Merged

3 tasks

marko-polo-cheno added 3 commits October 11, 2024 09:36

Merge branch 'trunk' into create-how-to-OD-doc

4ebfc5e

Simplify content and mention util functions

57180b4

Fix links and special char

31c74ea

marko-polo-cheno requested review from mkaramlou and removed request for gordonhart and mkaramlou October 11, 2024 14:10

small edits

aed06f9

munkyshi reviewed Oct 11, 2024

View reviewed changes

munkyshi approved these changes Oct 11, 2024

View reviewed changes

nits

3827f37

marko-polo-cheno merged commit 28562fd into trunk Oct 11, 2024
32 checks passed

marko-polo-cheno deleted the create-how-to-OD-doc branch October 11, 2024 21:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to do Object Detection in Kolena - Task Doc Template #583

How to do Object Detection in Kolena - Task Doc Template #583

marko-polo-cheno commented Apr 26, 2024

codecov bot commented Apr 26, 2024 •

edited

Loading

marko-polo-cheno commented May 24, 2024

y27choi May 27, 2024

y27choi May 27, 2024

y27choi Jul 9, 2024

marko-polo-cheno commented Jun 19, 2024 •

edited

Loading

y27choi left a comment

y27choi Jul 9, 2024

y27choi Jul 9, 2024

y27choi Jul 9, 2024

munkyshi Jul 11, 2024

y27choi Jul 9, 2024

munkyshi Jul 11, 2024

munkyshi Jul 11, 2024

munkyshi Oct 11, 2024

munkyshi Oct 11, 2024

munkyshi Oct 11, 2024

munkyshi Oct 11, 2024

munkyshi left a comment

		It's also important to note down the evaluation configuration of the model being uploaded, for example:
		`iou_threshold: 0.5` or `iou_threshold: 0.3`.


		### Step 2: Upload your Object Detection Dataset

		Model evaluation on Kolena starts with datasets. Upload your dataset of datapoints (e.g. locators to

	poetry run python3 object_detection_2d/upload_results.py faster_rcnn
	uv run python3 object_detection_2d/upload_results.py faster_rcnn

How to do Object Detection in Kolena - Task Doc Template #583

How to do Object Detection in Kolena - Task Doc Template #583

Conversation

marko-polo-cheno commented Apr 26, 2024

Linked issue(s)

What change does this PR introduce and why?

codecov bot commented Apr 26, 2024 • edited Loading

Codecov Report

marko-polo-cheno commented May 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marko-polo-cheno commented Jun 19, 2024 • edited Loading

y27choi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

munkyshi left a comment

Choose a reason for hiding this comment

codecov bot commented Apr 26, 2024 •

edited

Loading

marko-polo-cheno commented Jun 19, 2024 •

edited

Loading