Hackathon – Image Recognition Slackbot

Our internal hackathons give us extra opportunities to play with software and carry out quick projects we can’t usually justify sinking time into.

A while ago we built our Slackbot, it’s a bit of fun, from providing inspirational quotes to tracking who’s made the most cups of tea. It’s well on it’s way to doing everything you never wanted.

The Problem

DogsBOT reacts to discussions regarding lunch. Initially this was with ? emoji. Since we don’t eat spaghetti every day I switched it to the ? emoji but this also became routine.  What we needed was a less predictable, more intelligent reaction.  My criteria for this was:

• Linked to the current weekly menu
• Some element of chance

A Solution

The canteen menu for the week is available as a PDF on the site intranet. It was an easy choice to use it as the source for DogsBOT’s lunch reactions.  As there are multiple options each day DogsBOT would be able to pick one for today at random.

With that decided we just needed to convert the PDF of the menu to text.  Given my bias to use python, a quick google indicated PyPDF2 would be the perfect tool. Unfortunately the menu turned out to be almost entirely images.  All I was able to pull out was the title and a bunch of meta data.  I needed to switch to an image recognition tool.

For the image recognition I wanted to run the code locally so ignored the various cloud solutions.  This still left a lot of choice of software.  I found this guide which did everything I was looking for using OpenCV (to clean up the image) and Tesseract (to pull out the text).

Now I’d got the text, I needed to work out which day the food would be served on. I was starting to run out of time so I set-up a quick loop.  It assumes each item is from the day of the week it most recently passed.  Happy that this was mostly working so I moved on to converting the food to emoji.  I did this by attempting to convert each word to an emoji and using the ones that were successful.  In hindsight I would have been better off selecting reactions based on shortnames in this data as Slack reactions use them rather than emoji.

Conclusion

The new improved lunch plugin mostly works.  I had fun playing with PyPDF2, ImageMagick, OpenCV and Tesseract as well as learning a little more about how Slack reactions work.

Tesseract is impressively good.   Given that I’d been planning to just pull the text out of the PDF I didn’t spend huge amount of time comparing it with the alternatives since it did everything I needed.  If spent more time on this I’d like to spend more time tweaking the OpenCV commands to work with different lower quality images.

More hackathons

Whether we’re fitting out a storm trooper or camping in a field we all love spending a day out doing something different.  For our next hackathon we’re planning to try and break in to our internal servers.  If you’ve got any hackathons lined up you think we’re missing out on let us know in the comments.