My Attempt at a Muppet Classifier Using fast.ai

Or at least a Sesame Street Muppet Classifier.

The thing is - my son is super into Sesame Street right now. He knows most, if not all, of the muppet characters. His absolute favorites are Elmo, Snuffy, and Cookie Monster.

But really, he likes them all.

Over the past month or so, I’ve been taking fast.ai’s course Practical Deep Learning for Coders. The first “assignment” (this isn’t for credit, so all assignments are optional) is to create my own image classifier.

Since my son likes Sesame Street so much, I decided to make a Sesame Street Muppet classifier.

Things didn’t turn out as straightforward as the example in the course. In the example (found in chapter two of the Deep Learning for Coders textbook), the system is trained to distinguish between three types of bears: grizzly bears, black bears, and teddy bears.

Examples of the bear types that the example classified.

There are many more muppets than three. In fact, the Wikipedia list of Sesame Street Muppets has 106 different muppets. To be fair, some of those are effectively duplicates - such as “Grover” and “Super Grover.”

Can you tell the difference between Grover and Super Grover? sources: Super Grover and Grover

After narrowing down the list by removing duplicates and generic muppets such as the Anything Muppets, I had a list of 100 muppet names.

Bing Image Search

For the fast.ai code, the images are downloaded from a Bing image search. To help make sure I got Sesame Street Muppets rather than someone (or something) else, I appended “sesame street muppet” to each of my queries. While Cookie Monster is Cookie Monster, there are other entities named “Zoe.” Check out the difference between the two searched:

Bing image search for “Zoe”

Bing image search for “Zoe sesame street muppet”

Google Colab

Unfortunately, I don’t have a graphics card - and I am even running Windows at home, whereas fast.ai is meant for Linux. Thankfully, that doesn’t matter because I can do the “homework” (and run the example jupyter notebooks) online, for free, via Google Colab.

Google Colab allows me (or anyone, really) to run my code on a GPU. I just need to specify which runtime type I want by selecting Runtime -> Change runtime type - which gives me the following handy-dandy dialog:

Google Colab runtime types dialog

None is the default setting, so I had to remember to change my runtime type before I ran any code - I noticed that if I switched runtime types after running any code, my session would be cleared (so all my variables would be reset and temporary files - including my image files! - would be deleted). The menu options Run all, Run before, and Run after underneath the Runtime menu were helpful at getting me back where I was at whenever this happened.

Also, Since Google Colab is free (or rather, I am using the free tier), there are limitations. I was simply processing too many images, so I eventually ran out of RAM, and got the following error:

Your session crashed after using all available RAM. If you are interested in access to high-RAM runtimes, you may want to check out Colab Pro.

Here’s a screenshot of the error:

Back to My Muppet Classifier

100 muppets were simply too many muppets to identify on my free tier of Google Colab.

Not only did it use too much RAM, but it also made the confusion matrix to messy (and confusing!):

A confusion matrix jam-packed with 100 entities.

A confusion matrix is simply a table that maps what the image was identified to be to what the image actually is.

I decided to go with just my son’s three favorite muppets (Elmo, Cookie Monster, and Snuffy) plus Zoe - to make four muppets and one more class than the three bear types.

Now my confusion matrix is a lot less… confusing:

A confusion matrix with just four different muppets.

The Actual Classifier

Or, rather, the attempt.

I can get it to work on Google Colab - and I even installed Jupyter Notebooks and all the required dependencies locally and got it to work there:

What I haven’t been able to do is get it to work somewhere that anyone can access.

I’ve tried putting it on a service called Binder and that didn’t work. I also tried it on Heroku, but that didn’t work either.

If anyone wants to take a look, the code is hosted on GitHub - the “app” part is muppet_webpage.ipynb specifically.

I am able to embed the Heroku app here, but it gets stuck “thinking”: