When AI has a Person-al problem!

When AI has a Person-al problem!

After some research, I had decided to try out Arlo’s home security camera system and see how Arlo’s AI recognition features perform with the real-time image/video data.

Installation was quick and I was really impressed with the Wi-Fi connectivity, 2K resolution video feed, local storage, clod-based features and off course AI capabilities.

One of the features I was looking forward to test is how this built-in (cloud based?) AI algorithm recognizes People, Vehicles, Animals and detect Motion. I don’t have the complete insights into how and where this detection is performed, at the camera level, base station or in the cloud.

So, I went ahead and turned on the alerts for all the possible combinations and within few minutes I started receiving notifications on my mobile app. That was pretty quick and made me trust the system more.

Most of the notifications are accurate but for detecting a Person almost 80% of the notifications are false alarms.

There is a helpful feature built-in and this feature captures few seconds of video whenever a Person is detected. Majority of the time Person detection is a result of a motion in the front yard like a moving vehicle or motion caused by wind. But after reviewing footage for Person Detected alerts, 80% of the time I did not see an Person in the video clips. This made me think that I had to configure AI detection features in the system more accurately but found no way to do the same.

At this point I thought I need to trust the AI and assume that Arlo did a good job of building this Person Detection Machine Leaning Model. Arlo might have used a significantly big training dataset (Images), tested against big dataset (Images) and also validated against some real life scenarios. Some scenarios like multiple people in an image (ex: a family walking on side way in from of your home).

Meanwhile mystery continues and at one point I thought of reaching out to Arlo, which I will do in future any way, and explain the problem.

This is one the main challenge we would face while developing or implementing AI capabilities or solutions.

As part of building a ML Model, for example, detecting a Person in an image, we often rely on Supervised Leaning approach, using massive datasets of images for training, testing (fit, score and predict) and pick the model with a higher R Squared value aka Coefficient of Determination Factor.

Experience with my security system made me think that it’s not just about building a model and leave the model in a static state. When I say static what I mean is that model is not designed to take new information, fine tune and come up with more accurate predictions.

This may lead us into adopting Unsupervised Learning approach but still we still need a Data Science team to classify/refine and validate predictions.

In my opinion every Data Science project or while building AI capabilities we need to think like we are building a Product rather than simply creating a Solution. From a Product perspective, Product architecture should allow configuration, automation and a Strategy to keep AI capabilities or ML Models to up-to-date. This is very important otherwise your ML Models will become stale and so are the predictions.

Back to my original story, after studying the Arlo’s mobile app (where I receive notifications) I found out that when ever Arlo’s AI detects a Person, Arlo’s system starts capturing a quick video for future reference and in the actual notification it displays a snapshot of the image (still image) where actual Person, which AI thinks is highlighted.

This provides an instant view of the entire front yard, highlighting the area where AI thinks that there is a Person.

Finally mystery is solved!

The way our mail box, my neighbor’s mail box, mail box of the home (opposite to our home) , angle of the Camera and most importantly the way Camera sees all these 3 mail boxes. From the camera angle they look like a Person 😊

So, whenever there is motion in front of yard, Camera takes a snapshot and performs “predict” to identify, Person, Vehicle and Animal and ultimately predicts that all three mail boxes resemble a Person.

Until Arlo’s ML Model is updated with the latest dataset, I will end up receiving these alerts for the rest of my life or rest of the Security System’s life. I will try to reach out to Arlo’s team and hopefully they will provide a solution.

Meanwhile other solutions I can think of are

  • Change the Camera Spot
  • Paint my mail box with white color
  • Disable Person Alerts
  • Look for a new spot for my mail box

It’s very important to have a strategy and plan to keep every component of the AI or ML solution/product up-to-date. If not, your AI solution will have a “Person“al problem like my security system.