Some Questions for an Object Detection Model

[RoyiAvital]

I want to use Skorch for Object Detection.

I am using a PyTorch model: FASTERRCNN_RESNET50_FPN.

The output of the model is a list of dicts:

I was wondering, how can I utilize Skorch in that scenario? As the output is not regression nor classification.
How can I create the loss for this?

[BenjaminBossan]

skorch does not necessarily have to be classification or regression, the NeuralNet class is neutral when it comes to the task.

I haven't worked with object detection in skorch, but gave it a try. Following the, admittedly fairly simplified, examples from the torchvision FasterRCNN docs, I could get something to work. I attached a Python script that demonstrates inference and training (GH doesn't allow to attach jupyter notebooks, so I exported the script, rename it to have a .py file ending). The script is fairly bare bones, but maybe you could give it a try and tell me if you could adopt it to your use case. If so, we could develop this into a proper example and add it to skorch.

In general, it was possible to make object detection work with skorch, despite the unusual output. There is a bit of fiddling required to align the output format, but it works. I commented that part of the code.

There was, however, a second issue that required a workaround. Namely, for this model, the forward method calculates the loss during training mode and the detections during eval mode. This is annoying, because we cannot, for instance, calculate the loss on a validation dataset, since this uses eval mode and therefore, we don't get the loss! The "solution" is to disable validation in skorch :(

The fundamental problem is that torchvision combines inference and loss calculation into a single forward call. I'm sure there are good reasons for that, but it makes working with it difficult. Ideally, forward would only predict the detections and the loss calculation would happen in a separate criterion. This would allow us to properly calculate a validation loss with skorch. However, I don't see how this can be done, short of rewriting large parts of torchvision RCNN module. (pinging @ottonemo in case you have an idea.)

skorch-torchvision-fasterrcnn.txt

[BenjaminBossan]

@RoyiAvital Did you have time to test it?

[RoyiAvital]

@BenjaminBossan, Not yet unfortunately.

[DavidZSchneider]

What about YOLO models?

[BenjaminBossan]

What do you mean by "recreate", training those models from scratch? What would be the use case for that?

[DavidZSchneider]

It can use the COCO128 data set. It will create a simple example how to achieve this with skorch.

[BenjaminBossan]

I can see how that could be useful for some users. I'm not very familiar with yolo or object detection in general, so I'm not qualified to provide that. But if someone (you?) is willing to create a notebook or script, we could add it to skorch.

[DavidZSchneider]

I can try.

[BenjaminBossan]

That would be great. Let us know if you need help. Also, feel free to create a draft PR to get early feedback.

Using one of the existing notebooks or examples as inspiration/starting point may be useful.