Testimonials

The Problems with Testimonials  

"For centuries doctors used leeches and lancets to relieve patients of their blood. They KNEW bloodletting worked. EVERYBODY said it did. When you had a fever and the doctor bled you, you got better. EVERYONE knew of a friend or relative who had been at death’s door until bloodletting cured him. Doctors could recount thousands of successful cases."

You can't tell in an individual case what would have happened if nothing was done, or something else

- particularly for a condition that has a variable clinical course. For example, most people with COVID survive; and many people with indolent cancers will live without ever needing treatment.  So like the blood-letting example they cannot assert that what they did mattered - or if they would have done well anyway. 

- Testimonials can't inform about the number of persons who have used the intervention and did not benefit or were harmed.

- Testimonials provide no denominator - the number of patients studied.
The denominator tells us if the results are likely to occur in 1 of 5, or rarely - in 1 of 300,000.  People sometimes win the lottery, but this does not make playing the lottery a good bet - particularly when betting your life.

Testimonials cannot estimate the rate of effects in others, or if the effect that is measured was even caused by the intervention. Without a pre-specified number of participants estimating the rate of good and bad effects in others is not possible.

Note that a control group is needed to establish causality - that the treatment  caused the good or bad effects. This is the reason that expensive and large controlled studies are often needed before a drug can be approved for the treatment of a disease.  In a randomized controlled trial the persons taking part are put into study groups by chance.  This protects against selection bias -- both groups will have similar risk factors if the study is large enough. 

People who die cannot tell their story

Patients who have tried and failed an alternative strategy cannot be accounted for. Only the "successful" outcomes are reported, which may be 1 in many thousands.

Compare with peer-review clinical trial where the number of patients receiving the treatment are known up front (prospectively), and the positive and negative outcomes are measured uniformly - and reviewed independently.


Is it truthful - is the story relating the entire story?

We cannot know if the person reporting the benefit really has the medical condition, or if he or she is reporting the outcome accurately?

With testimonials there is no follow up, or independent review of the outcome over time.

Did the story teller relapse shortly after the testimonial?


What are the biases or motives of the person telling the story?

Does she have a financial conflict of interest or strong belief?  Dose he sell the product or charge a fee for dispensing the information? Is the testimonial a way of validating her personal decision process and theories?

What are details of the case, such as the natural history of the disease?

Even for cancers with a very poor prognosis there are case reports in the literature of spontaneous remissions, independent of any intervention. Most cases of COVID are not severe!  Some types of cancer behave indolently-- and do not progress to a point requiring treatment. Cancers can behave like turtles, rabbits, or birds - even cancers with the same name.

How the outcomes were measured, when, and by whom?

- Was the reported success objectively measured and validated by independent physicians?

- Is it a patient-reported outcome? Was it that the patient felt better? What tests were used to measure it? What happened later? With testimonials you can't know if the intervention lead to a lasting or meaningful benefit.

- Is the condition self-limiting - does it sometime self-correct with time? 

What other medical treatments were given shortly before or after?

CT scans may show lesions after standard treatment that are necrotic scar tissue. Credit might be given to an alternative practice used AFTER this treatment, when it was merely the resolution of a scar tissue over time, a normal bodily process.

Was the diagnosis accurate?

- Was it a false diagnosis of a cancer, or a cancer of a type with an indolent course?

For all of these reasons it's prudent to regard testimonials with suspicion - particularly if the report is implausible, scientifically. Case reports by doctors have many of the above limitations - cannot establish causality, and can't be the basis for predicting outcomes in others.

Scientists, doctors, investigators, and regulators get cancer too, as do their children and loved-ones. IRBs, independent investigators, the FDA, Data Safety Monitoring committees are all gatekeepers to the conduct of clinical trials and the reporting. The reports are also subject to peer review.

 The FDA (not the sponsoring company) decides if a study drug can be marketed.  Only about 1 in 10 gain marketing approval.  There is no conspiracy.




Comments

Popular posts from this blog

Karl Schwartz, Bio / Advocacy CV

Lymphoma: Guidelines at Diagnosis