Reviews You Can Rely On

How We Tested Robot Vacuums

Sunday September 29, 2024
robot vacuum - a few of our test models lined up for their glamour shots.
A few of our test models lined up for their glamour shots.
Credit: Abriah Wofford

To see which robot vacuum is really the best for getting your floors to shine, we conducted extensive research, then bought all of the best 'bots available today to test head-to-head. We've spent over two years reviewing and testing these products, so have come to love — and hate — all of their various quirks and other odd behaviors that we have seen over the years. We've conducted over 30 different evaluations to score and rank these products, grouping them into six weighted rating metrics. The sections below detail our exact testing and scoring procedures in each test. If you are curious about how specific models performed and which robot vacuum truly reigned supreme, head on over to our comprehensive robot vacuum best-in-class review here.

Eufy Robo Vac X8 Hybrid deftly maneuvering your home's furniture.
Credit: Jessica Riconscente

Navigation


Navigation is our most heavily weighted test metric, comprising 30% of each vac's overall score. We wanted to know how these vacuums could navigate around furniture and other challenges. Our test course included a dining room table with chairs, a couch, a coffee table, a floor lamp, and an accent chair.

robot vacuum - the jet bot ai+ navigating through a simulated table and chairs.
The Jet Bot AI+ navigating through a simulated table and chairs.
Credit: Jessica Riconscente

We assessed each robot's ability to navigate the room without our assistance. (Our next metric covers overall cleaning effectiveness.) Robots lost points for overly erratic behaviors, such as repeatedly cleaning under a table for the bulk of the time.

Most robot vacuums are not built with object recognition in their navigation algorithms, and won't avoid poo or other smaller objects.
Credit: Jessica Riconscente

We also added a poo-avoidance challenge, that evaluated each model's ability to detect and avoid two sizes of simulated pet waste. Robots were awarded points for avoidance, as well as how close to the poop and how thoroughly they clean near the mess.

The Roborock S7 has incredible stool avoidance technology for those with pets.
Credit: Jessica Riconscente

Lastly, we evaluated each robot's ability to navigate back to a docking station for a recharge, and award points for efficient travel, and furniture avoidance.

Home Coverage


Next, we looked at how efficiently each vacuum can clean your home. This differs from the previous metric, as it judges where each robot actually cleans, compared to its ability to navigate around obstacles and essentially not become stuck. This metric comprises 20% of each vacuum's total score.

robot vacuum - the roborock covered an impressive amount of our test room.
The Roborock covered an impressive amount of our test room.
Credit: Austin Palmer

First, and most important for this metric, we tested how well each vacuum does at cleaning multiple rooms. The majority of the vacuums have mapping capabilities, allowing them to clean multiple rooms during cleaning, returning to their docking station, and recharging as necessary.

The app lets users configure rooms, zones, and spot cleaning.
The app lets users configure rooms, zones, and spot cleaning.
Apps allow you to manage multiple rooms as well as virtual barriers.
Apps allow you to manage multiple rooms as well as virtual barriers.

However, there are still a few that bounce around randomly, meaning they can't really reliably clean multiple rooms, causing them to be penalized in this assessment. We also checked that each vacuum could clear a standard threshold, which they all could do easily.

Next we tested edge and corner coverage. For the test, we spread coffee grounds along the edges of a robot pen we made, then ran each vacuum. We used the coffee grounds for maximum contrast and scored each product based on the measured distance of the effective cleaning area from edges and corners.

robot vacuum - our edge testing environment measures the edging and corner...
Our edge testing environment measures the edging and corner performance of each model.
Credit: Jessica Riconscente

Lastly, we tested how well each vacuum did at cleaning a single room, as well as how effective its spot cleaning abilities are.

robot vacuum - precisely measuring cleaning thoroughness and coverage.
Precisely measuring cleaning thoroughness and coverage.
Credit: Abriah Wofford

Carpet Cleaning


After scoring how well each robot moved throughout your home, we moved on to scoring how each robot did at actually cleaning. We started off by evaluating and scoring how well each robot did at keeping soft floors clean, using a variety of different debris as sample messes, as well as comparing how close each vacuum cleaned to the walls or other edges of a room. This metric is responsible for 15% of the total score.

The brush head of this model is designed for performance.
The brush head of this model is designed for performance.
The Roomba has a nifty dual brush cleaning system.
The Roomba has a nifty dual brush cleaning system.
A unique mop design is incorporated into this model.
A unique mop design is incorporated into this model.

For all of our debris tests, we spread a consistent amount of the material on the floor, then scored each robot vacuum by comparing the before and after photos. We thoroughly cleaned the carpet in between trials with an upright vacuum, to ensure that there was no residual mess carried over between products.

For the first test, we used flour as our fine debris. We only conducted this test on flat, low-pile carpet, as extracting flour from fluffy carpets is really beyond the current cleaning capabilities of these products.

robot vacuum - our research analyst, jessica riconscente, spread out debris over a...
Our research analyst, Jessica Riconscente, spread out debris over a short pile carpet for our robot vacuum testing.
Credit: Abriah Wofford

After flour, we repeated the test with both rice and oats, for our medium debris, though this time we did the test on both flatter, low-pile carpet and on fluffier, medium-pile carpet.

robot vacuum - measuring carpet cleaning performance with flour as our test material.
Measuring carpet cleaning performance with flour as our test material.
Credit: Abriah Wofford

Finally, we used the small size of Mini-Wheat for our large debris test, to see if the vacuums had sufficient power to crush them up or had the ability to just suck them up whole. We also did this test on both types of carpet.

robot vacuum - this cereal went for a free ride but wasn't actually cleaned up.
This cereal went for a free ride but wasn't actually cleaned up.
Credit: Jessica Riconscente

Hard Surface Cleaning


For our next metric, we repeated the exact same set of tests that we did on carpet, though this time they were all conducted on a section of hardwood laminate flooring. This group of tests is also worth 15% of the total score.

robot vacuum - the i3+ evo sometimes had challenges with clearance.
The i3+ EVO sometimes had challenges with clearance.
Credit: Jessica Riconscente

We paid particularly close attention to see if the vacuums flung debris around, rather than cleaning it, as they were much more prone to do this on the smoother surface.

Flour on hard surfaces proved to be very difficult for a large...
Flour on hard surfaces proved to be very difficult for a large number of robots.
Cereal laid out on a hard surface for our next challenger.
Cereal laid out on a hard surface for our next challenger.

We also looked closely at the suction power and abilty of models to lift flour out of cracks and seems.

Pet Hair


Next, we scored how well each robot dealt with the extra hair that the furry members of your home leave behind. We visited a local groomer who kindly donated a substantial amount of leftover fur to use as our supply, then spread out a predetermined amount on a section of both fluffy and flat carpet. These tests account for 10% of the total score.

Setting up a test to evaluate a vacuums ability to pull pet hair out of carpet.
Credit: Abriah Wofford

We scored each vacuum on how much it picked up, as well as how much actually ended up in the collection bin of each product, rather than tangled up in the main extractor brush or the undercarriage of the robot.

Convenience


For the final 10% of the total score, we evaluated how convenient each model is. This metric evaluated the docking station features that increase convenience, like a dirt disposal system, as well as the size of the tower, and dirt collection capacity.

Some robot vacuums feature more than just a dust removal system.
Some robot vacuums feature more than just a dust removal system.

While docking and dust disposal are important factors, user-friendliness of operation is arguably more important. Our team primarily awarded points for robot apps and functions that enhance convenience. We awarded points for remote control functions, whether a physical remote or control via the app, the ability to draw a virtual barrier, battery life indicator, spot cleaning requests, suction power adjustment, maintenance tracking, robot finder, history reports, a repeat function, and cleaning maps.

robot vacuum - a remote is helpful, but an app is more convenient.
A remote is helpful, but an app is more convenient.
Credit: Abriah Wofford

Conclusion


Hopefully, this article has provided you with some more insight into how we tested these products and why they scored what they did in our complete review of robot vacuums. For even more background information on these products, such as why you would even want one and what to look for when shopping for a new one, head on over to our robot vac buying advice guide for a complete breakdown of the products.