Reviews You Can Rely On

How We Tested Earbuds

Saturday September 28, 2024
earbuds - our bruel and kjaer 5128-b head simulator in our insulated sound...
Our Bruel and Kjaer 5128-B Head Simulator in our insulated sound studio.
Credit: Abriah Wofford

How We Test Earbuds and Headphones


Here at GearLab, we conduct some of the most advanced testing of consumer audio products in the world. We've assembled a first-rate audio testing suite with the help of industry leader Listen, Inc. Listen's products are often found in the quality control and research and development labs of top audio production companies.

We've leveraged the same technology to bring accurate and comparable quantitative data directly to you. Listen, Inc.'s CEO Steve Temme, an audio engineer with decades of experience at the forefront of his field with multiple academic papers to his name, has a word for the system we use, “world-class”.

earbuds - soundcheck software analyses the recordings from our bruel and kjaer...
SoundCheck software analyses the recordings from our Bruel and Kjaer 5128-B Head Simulator.
Credit: Natalie Kafader

The GearLab Audio Testing System


Our audio test suite includes a Brüel & Kjaer Type 5128 Head Simulator that transfers high-quality recordings to a computer running SoundCheck software via a professional-quality audio interface.

Darwin joined our team of audio experts to lend a truly objective ear when testing earbuds head-to-head.
Credit: Natalie Kafader

Let's break that down. The Brüel & Kjaer Type 5128 Head Simulator is a precisely engineered and anatomically correct head and torso mannequin that simulates the acoustic properties of the human form. Highly sensitive microphones in its realistically crafted ears allow us to make accurate, real-world recordings of the headphones and earbuds we test. A top-notch speaker in its mouth replicates human speech, allowing us to test voice commands and call quality. We've spent a lot of time with our Type 5128, and have grown fond of its endless work ethic and trusted, objective ear. We've dubbed it Darwin.

While it records the performance of each pair of earbuds and headphones, it doesn't have the hardware or software to process them. So, we use an audio interface to pass its recordings to a computer running Listen, Inc.'s SoundCheck audio test and measurement software.

earbuds - a jds labs atom amp headphone amplifier and listen, inc...
A JDS Labs Atom Amp Headphone Amplifier and Listen, Inc. SoundConnect 2 audio interface transfers Darwin's experience to the SoundCheck software.
Credit: Natalie Kafader

It's the SoundCheck software that does all the clever analysis, measuring a wide range of audio characteristics such as frequency response, noise cancellation, distortion, and call quality. The software also compares the performance of each set of earbuds and headphones to our idealized Gear Lab Target Response Curve. That covers the basics; now let's dive into the details.

Darwin records audio that is then analyzed by SoundCheck software.
Credit: Natalie Kafader

Frequency Response


Frequency response is charted as a curve, with frequencies listed on the X-axis in Hertz (Hz) and sound intensity, measured in decibels (dB), on the y-axis. It measures the relative volume of all audible frequencies to find, for instance, if low bass tones are balanced by brightening, high treble notes or if one or the other is overly emphasized. A set of headphones or earbuds with an elevated response, or peak, in the lower frequencies will have a louder, more powerful bass sound. A peak in the middle frequencies will emphasize sounds like vocals and guitar.

Take a look at the frequency response chart below. The horizontal axis spans from low, bass frequencies on the left to high, treble frequencies on the right. The vertical axis represents relative sound intensity. When you see a curve above the horizontal line at zero, the earbuds emphasize sounds in that frequency, making them relatively louder. If the curve dips below that line, sounds at that frequency sound relatively softer.

The Bose QuietComfort Ultras frequency response is roughly the same as our GearLab House Curve.

The chart above is an example of a nicely equalized frequency curve. Notice how the blue curve closely matches the red target response curve? Accurate frequency response curves like that translate to an earbud with a higher sound quality. To provide a counterexample, the chart below represents a pair of earbuds with less-than-stellar audio quality. The bass and mid-levels are overemphasized, with much more deviation in the high range.


The Harman Curve


You may be asking yourself, is there an ideal frequency response curve for headphones and earbuds? Researchers have been laboring for years to answer that question. Harman International, a parent company of many leading audio brands, created one of the first idealized frequency response curves. It helps companies build headphones that work for most people and helps other organizations test their effectiveness. It's called the Harman curve, and about 60% of people find it to provide the best overall listening experience.

The House Curve


The Harman Curve is not perfect. It was created using older equipment, and it isn't specific to earbuds, which have unique acoustic properties since they nestle into your ear instead of resting over it. To honor our state-of-the-art audio testing suite and our expert tester's well-tuned ears, we created our own GearLab Target Response Curve, aka the House Curve. It emphasizes the bass and treble frequencies a touch more than the Harman Curve. We think it sounds better and hope you agree.


How We Measure Sound Quality


To test the sound quality of each set of earbuds and headphones, our team of audio experts listen closely and critically to a curated Spotify playlist using each earbud's baseline settings. We then rate each earbud independently on a range of characteristics, including their soundscape, balance, gain, and distortion in each frequency range — bass, mid-range, and treble. We also take copious notes about which earbuds sound better and why, citing specific, time-stamped examples from the playlist.

After these subjective tests, we bring the Bruel and Kjaer 5128-B head simulator, Listen, Inc., and SoundCheck in to provide objective rigor while testing each pair's frequency response curves, noise cancellation skills, call quality, and distortion.

We start with a panel of audio experts closely listening to a carefully curated playlist. Who says work can't be fun?
Credit: The GearLab Team

How We Measure Frequency Response


To measure the unique frequency response curves of each set of earbuds or headphones, we play a known sound file through each pair to the microphones in the ears of the Bruel and Kjaer 5128-B. It records the sounds it “hears,” and a Listen, Inc. audio interface transfers them to a computer running SoundCheck. The sophisticated software measures and displays the differences between the original source sound file and how each set of speakers reproduces it. Those differences represent the baseline equalization, or EQ settings for the model, and how well they optimize your listening experience.

earbuds - soundcheck's powerful software allows us to analyze darwin's...
SoundCheck's powerful software allows us to analyze Darwin's recordings in real time.
Credit: Natalie Kafader

We then compare each frequency response curve to both the Harman Curve and our very own House Curve. We find that the frequency response curves that most closely match our House Curve correlate well with our expert listener's preferences. These earbuds receive a higher score. Your tastes may differ from ours, but our ratings serve as a reference point to help you find the right headphones or earbuds for your needs.

Speakers and the Harman Curve


It's important to note that the Harman Curve was created using over-the-ear headphones and does not translate well to free-standing speakers. Our brains interpret sounds played directly into our ears much differently than when sounds emanate from a speaker in an open room. In fact, the frequency response curve is often meant to help headphones sound more like a room-based speaker system.

earbuds - by playing pure tones across the entire audible spectrum through...
By playing pure tones across the entire audible spectrum through each set of earbuds, the SoundCheck audio software can chart the decibel levels of each frequency.
Credit: Natalie Kafader


User Experience


We also tested what it's like to use these earbuds — including how comfortable they are, how securely they settle into your ear, and how easy it is to adjust their settings. These tests help us decipher which earbuds are best suited for exercise and which are meant for relaxing.

To test what it's like to use each set of earbuds in the real world, we wore them, a lot.
Credit: Abriah Wofford

How We Measure Comfort


To test the comfort of each set, we had a number of male and female testers with various ear sizes wear them. First we noted whether or not the earbuds come with a selection of ear tip sizes, which allow you to customize their fit and offer you a lot of control over their comfort. Then we asked each tester to take note of whether they could easily wear the earbuds in for an hour or more at a time. We also took note of additional features, like the shape of the ear tips, whether or not they are able to rotate to shift their position, and the feel of the ear tip material. All of these factors affect their comfort.

To test comfort, we wore each set of earbuds for days on end and tested them side-by-side to compare their fit directly.
Credit: Natalie Kafader

How We Measure Stability


Nothing is more irritating than expensive earbuds that fly out of your ears at every turn, or feel like they're going to. To test the stability of these earbuds, our testers wore them to work out, run, and even subjected them to the biggest stress test of all, headstands. If they can stay in your ears when you're upside down, you're probably good to go. If they fell out, they earned a lower score.

We don't take stability testing lightly, wearing each set of earbuds on active days and while conducting stress-tests like handstands and headbanging sessions.
Credit: Natalie Kafader

How We Measure Customization


We also tested how well each set of earbuds worked with different types of technology and how easy it is to access their various features. Popular features include ear fit tests to make sure you're getting a good seal, location features to help you find lost buds, and associated apps that adjust your settings.

We took the time to closely evaluate the offerings of each smart app.
Credit: Natalie Kafader

It's Better to Match Platforms


Not to spoil the surprise, but we found that Android earbuds do not play nice with Apple products, and vice versa.

We tried out each feature and noted how well it worked in comparison to other options in our testing. To help you understand what it's like to navigate each app, we created videos that walk you through the settings. These give you an accurate feel of how difficult it is to, say, adjust the relative volume of different frequencies, like the bass or treble notes, a process known as equalization (EQ).

The Jabra app is one of the most user-friendly we've encountered, making it a breeze to adjust to your tastes.
Credit: Natalie Kafader

How We Measure Connectivity


Ideally, each set of earbuds would offer easy-breezy device pairing, followed by an uninterrupted listening experience. However, that is not always the case. To measure the reliability of connection in each model, we take note of any issues we had while pairing to various devices, as well as if there were any random breaks in audio or inconsistencies in sound quality. Environmental factors like the age and operating system of your device, whether you have the latest firmware update, proximity to other devices, etc., can all affect the connection between your earbuds and phone or laptop. To augment our hands-on testing of connectivity, we also analyzed the frequency of 1-star Amazon user reviews that cite a connectivity issue as one of the key critiques. We used the statistical ratio of negative connectivity feedback relative to the total number of Amazon user reviews as a measure of connectivity issue dissatisfaction. Keep in mind that every product will have some negative feedback, as Bluetooth technology is not perfect. Our statistical connectivity complaint analysis helps us gauge the relative severity and frequency of connectivity issues in an apples-to-apples statistical manner. Our overall scoring of connectivity is based on the combination of our statistical analysis of Amazon user reviews and our direct observations of connectivity issues from our extensive hands-on testing with multiple devices.

earbuds - we connected each pair of earbuds to multiple devices throughout our...
We connected each pair of earbuds to multiple devices throughout our testing period and supplemented our in-house experiences with a survey of Amazon user reviews.
Credit: Abriah Wofford

Noise Cancellation


To judge how well each pair of earbuds and headphones block outside sounds to let you concentrate on your audio entertainment, we also test their noise cancellation abilities. Some of the earbuds in the test offer active noise cancellation (ANC), which they achieve by analyzing incoming sound waves and cancelling them by emitting an inverse wave. All of them offer some level of passive noise cancellation by physically blocking sound waves from entering your ear.

Fit Matters


If you're using earbuds, the best passive cancellation depends on an effective seal in your ear canal. To optimize it, make sure you try out the various ear tips that often come with a set of earbuds. Keep going until you get a good fit.

First, we use these earbuds in the real world to experience how well they block and cancel noise in day-to-day life. We test them back to back, taking copious notes and rating them comparatively. Then we turn to the audio lab for some objective data.

The Bose Quiet Comfort Ultra test compares the unfiltered sound of the cafe, against the earbud's passive isolation abilities, and their impressive noise cancelling chops.
Credit: Abriah Wofford

To test both active and passive noise cancellation, we pipe pink noise into a room and have the Bruel and Kjaer 5128-B record it. We start by playing the background sound without the earbuds in place. Then we add the earbuds, hit play on the pink noise, and the Bruel and Kjaer 5128-B records how much sound each set blocks out physically. If they offer ANC, we turn it on and run the test again to see how much more sound they can actively scrub out.

The Bose Ultra's are a prime example of the height of noise cancelling technology.

We use SoundCheck to calculate and graph the passive and active noise attenuation of each set and then add them together to calculate the total attenuation for each speaker set. If a pair of earbuds or headphones doesn't offer ANC, we only test their passive noise attenuation. This process shows us exactly how many decibels the earbuds or headphones can block or cancel in each frequency range.

Here is an example of less impressive active noise cancelling. Darwin's data collection confirms that this model underperforms.
Credit: Abriah Wofford

We also took note of which earbuds came with a transparency mode that can be turned on and off. Often activated by pressing or tapping your earbuds, transparency mode allows you to quickly bypass noise cancelling mode when you need to hear a colleague's question or an announcement at the airport.

The Apple EarPods only offer passive noise cancellation.
Credit: GearLab Staff

What Is Transparency Mode?


Transparency mode uses the earbud's microphones to bypass the passive noise isolation at your behest. Some earbuds offer additional awareness modes, some of which use artificial intelligence to decide when to let ambient sounds through based on your environment
.

Battery Life


To test the battery life of each set of earbuds, we took them from a full charge down to 0% battery life, playing music in a continuous loop the whole time. We kept their volume levels consistent, turning each to about 99 decibels. Then, we compared how long they lasted to their manufacturers' claims.

Knowing your battery level can be the difference between a fun trip to the gym and a fitness slog.
Credit: Natalie Kafader

How We Measure Call Quality


The magic of earbuds is that they allow you to exist in your own little world of sound. But they also connect you to friends in an instant with a phone call. To compare how well each set transmits your voice over the airwaves, we tested how well these earbuds isolate your voice from background noise. This is technically known as voice isolation.

To test it, we use a high tech speaker inside of our Bruel and Kjaer 5128-B simulator head, which is located where a human's mouth would be (as seen in the photo below).

earbuds - darwin's "mouth" is actually a speaker.
Darwin's “mouth” is actually a speaker.
Credit: Abriah Wofford

We use it to broadcast a script while blasting noise in the background. Then, we rated each earbud set by how well their mics isolated the voices. Our human testers judged the quality of these clips. Some earbuds did pretty well in these conditions. Others did not.

You can hear the Amazon Echo earbuds struggling to isolate our voices from the loud background noise.
Credit: Graham Faulknor

We varied the volume of the background track — playing voices talking in a quiet room, mid-level office noises, and downright blaring subway sounds. The highest levels were a true stress test, with concert-level decibels that would keep you from hearing a friend standing right beside you. That was the point.

In contrast, it's relatively easy to understand the audio from the Beats Studio Buds+, even in the subway.
Credit: Graham Faulknor