Can BuzzFeed Be Trusted To Write About Science?

A response to Can Fitbits be Trusted In Science

by Aaron Coleman Founder & CEO of Fitabase, a web platform for data collection, analysis, and export using Fitbit wearable devices. Fitabase has assisted with data collection and analysing Fitbit data for over 100 research studies.

It only took a few retweets today to catch my attention. BuzzFeed is talking about Fitbits in research? This should be good. I wanted to give them the benefit of the doubt, that they looked through the literature, contacted the right people, crunched some data, maybe even did testing of devices themselves.

Nope.

Author Stephanie M. Lee starts out touting some of the reasons why wearables are great in research: price, acceptability, data streaming, but drops it on us in the next sentence: “The only problem? The results may not be very accurate.” (hey BuzzFeed, here’s a good spot to add in a little thing we call a “citation”; science, you know?).

Later on we do get a little more of a clue as to where BuzzFeed is getting their data. They quote Hawley Montgomery-Downs who says “I have a heebie-jeebie factor about somebody using them in their science”. It’s true, in 2011 she co-authored this publication showing innacuracies in sleep:  http://hawley_montgomery-downs.psychology.wvu.edu/r/download/113307

The problem? The case they’re making broadly for Fitbits being unreliable in science relies on research published in 2011, so it has to have been done on a Fitbit Classic or Fitbit Ultra -- neither of which have been sold for years. This is the equivalent of comparing the original iPhone to the latest Android and declaring the iPhone inferior.

In fact, the rest of the article continues to rely exclusively on research done using just the Fitbit Classic or Ultra:

What’s worse? They brush over in one sentence that even in the outdated model, which predates many hardware and algorithm improvements, the Ultra actually did pretty well outside of sleep measurements:

“In the case of Fitbits, the consumer activity tracker most used in the studies reviewed for this article, step-counting seems to be highly accurate, independent reviews have found, supporting Fitbit’s claims that they are 95% to 97% accurate when worn correctly.”

AND, there’s another giant elephant in the room. The author is claiming this based on studies that tested it for sleep measurement, holding these (old gen) Fitbits to the standard polysomnography. You know what that looks like? It’s more wires, electrodes, and tape than anyone could reasonably be expected to wear and get a good night’s sleep ( https://upload.wikimedia.org/wikipedia/commons/e/e1/Sleep_studies.jpg ) This is like comparing a fighter jet to a Jetta.

Which brings me to the biggest miss for BuzzFeed -- but they almost got there: 

“Another benefit is cost. Actigraphs, medical-grade sleep-tracking devices, can strain budgets at prices that range between $300 and $1,000, and don’t always include associated software. In contrast, a sleep-tracking Fitbit starts at $100 — software included.”

Researchers have, for well over a decade, come to rely on actigraphy to measure physical activity, energy expenditure, and sleep in free living conditions. These are the devices that have been the basis for an enormous body of free living measurement, so perhaps these are the devices that we should be comparing Fitbits to?  Guess what!  When comparing the Fitbit to these devices, they do very, very well -- sometimes even better than the “gold standard” Actigraph:

Bottom Line - Fitbits are Excellent and Reliable for Science

Founding Fitabase, I’ve had the privilege of seeing what many very smart research institutions are doing with wearables. Much of this work is ongoing or awaiting publication, but keep a look out for impressive new metrics and benchmarks utilizing heart rate models that are influenced by both tri-axis accelerometer and continuous heart rate without a chest strap.

Fitbit and other types of wearables mark a huge shift in what a researcher can do:

  • Compliance: A device people will reliably wear and feel comfortable with. We’ve seen astounding compliance numbers from some of our customers.
  • Data availability: No more costly requirements for participants to come in every couple weeks, or worse, mail in devices. Our customers view all their participants data in just one dashboard.
  • The opportunity for feedback: Scientists are testing what kinds of behavioral interventions can be delivered with this new data. This has population-level health implications.
  • Battery Life: On the order of one week or 3 months for the Fitbit Zip.
  • Cost (this point was already mentioned but it bears mentioning again): Research budgets continue to be cut and the power of the science is strengthened by increasing the number of participants.
  • Precision Medicine: A few months ago the President outlined a new health delivery directive -- make care precise and tailored to the individual. Personal data capture and wearables have a big part to play in that future, and in the coming months the NIH will announce a slew of new funding opportunities for this.

I wish BuzzFeed well but when it comes to their science reporting maybe they should leave that to someone else and save the sensational headlines for celebrity gossip.

For anyone interested in running a research study using Fitbits, please shoot us an email at hello@fitabase.com, we’d love to hear what you’re planning.