To help you navigate in an ocean of content, we carefully select a few articles once every month to keep you up to date.
Sign up now ›
How Do You Get Your Fridge to Buy Your Milk?
25 Nov 2021
Nikolaj Sonne from So Ein Ding recently rolled over the screen with an episode searching for the perfect kitchen. His wish was for an app that purchases basic groceries automatically, preventing him from running out of e.g., milk. Shape created the small demo app for the show, and here we explain how it works, what we did, and why it may be a while before we see something like this at a larger scale.
It’s Monday morning, 7.00 AM, and the alarm goes off. You walk towards the kitchen to make the morning coffee and make yourself a bowl of cereal. You pull the milk from the fridge, but it’s close to empty. No cereal, and almost no milk for the coffee.
This is not the best way to start the morning, so wouldn’t it be great if the fridge automatically ordered your basic groceries for you?
Nikolaj Sonne, from the Danish tv gadget show, So Ein Ding, asked himself that same question in a recent episode, where he explores the perfect kitchen. As he says, “there’s an app for that.” And we made that app.
How the app works
In the episode, Nikolaj demonstrates an app called SMRTKLR, which can identify a carton of milk, register the weight, and simulate placing an order for a new carton of milk. This may seem simple, however, the potential for larger scale usage of this type of solution requires more evolved technology, than what we currently have – something Amazon is currently looking into with their attempt at implementing cameras and machine learning technology.
The SMRTKLR app is essentially made up of three elements:
Recognising the milk carton
Recognising the number on the scale
Sending a request for purchase of a new carton of milk.
The app fires up the camera and starts taking pictures every 2 seconds. Then it cuts the image in two – two thirds and one third. The two thirds get fed through machine learning to see if there is a milk carton on it, and if there is, it sends the bottom (one third) picture to google for analysation. Finally, it gets an answer back and if the weight is below the threshold, it uses Amazon to send a text message – simulating sending a request for a new carton of milk.
To get started we had taken tons of pictures of the milk carton from different angles and in different environments, to make sure that it would be recognised in the fridge. Then we needed to annotate the picture and create a model and put this model in the programme. We used a tool provided by Apple called Create ML to create a model we used to recognise the milk carton. We tried two different programmes for this purpose and created several models for testing, however, Create ML was the most reliable. Lastly, we fed the model into the Apple Vision framework to actually recognise the milk carton.
The real head scratcher was reading the scale. We tried some different image recognition programmes, but many were too unreliable to use. Google’s Cloud Service for optical character recognition was the most reliable and therefore used for this purpose. The ideal scale would feed the exact number to the phone via Bluetooth, however, that was not possible for this experiment.
So why does this technology not already exist?
There are several apps that can do different elements of what the entire app can. There are apps for recognising plants and body movements, and we already have fridges with cameras, but there doesn’t seem to be a specific app for the purpose of purchasing items directly from the fridge.
The short answer to the question is that the technology isn’t reliable enough. While this app succeeded with a very specific task, it would be much more of a challenge at a bigger scale – potentially creating problems with placement in the fridge, recognition software and reading the scale. At the moment, it seems to be more effort than it’s worth, but perhaps when the technology matures it will become the gadget, we all didn’t know we needed.