They don't need to prove it works, they only need to prove that it doesn't. That's related to how defenders need to close all the holes but hackers only need to find one to get in, proving that it doesn't work for a subset of the tests is enough to put a substantial hole in the boat.

Proving that it works is a much harder problem that probably only Theranos is willing to finance.

In both cases it's going to be statistical though. If you find one instance of a flaw in the Theranos it'll be easy to challenge it by talking about variations in blood chemistry, blood draws, time of day, etc. Good controls can help, but unless you can bring a phlebotomist on site to wherever the Theranos is you're going to face external consistency issues.

So the right answer is to repeat the experiment. Perhaps many times. You'll end up with a statistical judgement of consistent inaccuracy and while it'll be easier to prove (in terms of statistical strength) than showing accuracy, it'll still be (a) reasonably expensive, (b) generalizable research, and (c) less punchy for headlines.

(b) is especially concerning because if you publish generalizable research involving humans, as would be the only way to do this, then you have to treat it like a legitimate clinical trial and are responsible for many more legal processes, e.g. IRB approval.

Good point, I agree, but there is still margin for some errors. I believe a scientific journal would want a large number of tests, not a "Blood exam shown to fail once".

Proving reliability does rely on examining internal methods, and showing they are consistent and reliable.

I concur it somewhat lends to Theranos' credibility that no competitor is disproving their accuracy.

No the problem is no one can get their hands on a sufficient quantity of the tests in order to run such an analysis. This is not a cheap or easy exercise to do rigorously, and would require a fair amount of staffing to carry out properly.

The company being non-cooperative is it's right - it's just not good science.

It doesn't seem that hard or labor-intensive to me. Have volunteers who were going to have a battery of tests done at just one provider visit a second the same day. (At most, 2x the retail testing costs – but far less if any of the providers is a partner in the study.)

When the results come back, tabulate and compare.

If there are major discrepancies, they'd probably turn up in the first few dozen tests. But you could run this for hundreds of people with a single analyst, and just a few doctors soliciting their patients' participation, and without needing cooperation from Theranos.

But that's not the problem: the problem is you need to study how the test behaves on given conditions. There's a serious issue that a bunch of the type of people you could recruit for such a study, are likely to have ordinary looking bloodwork. Designing a test which mostly gives results that show "ordinary" is actually fairly easy - designing one which correctly picks up unusual or important conditions is harder.

To be valid, you'd want to go in and supply a bunch of baked-samples which should produce a certain definitive result, without the tester knowing this in advance.

But suppose you were running this 2-channel validation constantly, for people with both routine and exceptional needs. You'd be getting fresh information, in exactly real-need/real-disorder proportions, from both the traditional and new tests, in the cheapest and fastest way possible.

Storing 'baked-sample' blood, or doing extra tests against people who weren't otherwise needing tests, couldn't possibly be as representative. Diverting a few drops from tests that are already going through the traditional process, for research/calibration purposes, is likely already authorized by existing patient relationiships. Your "n" for analysis, across any disorder or rare test, would quickly exceed what's "recruitable" for some formal study.

Look at the list of Theranos partners/customers who have brought them to "cash-flow positive": hospitals, drug companies, insurance companies, and the US military.

It's naive to the ways of industry R&D to assume that such constant cross-validation hasn't been (and isn't still) happening.

Erm I don't understand the disagreement, I tried to argue pretty much the same as you. Maybe I didn't put it down too well.