Features
Part of How creator economy research numbers lose their source
Creator economy research mistakes: five checks before citing
Creator economy research mistakes at the reading stage: a median taken as typical, two studies charted together, a weight with no population, and five checks.
These are errors of reading, not errors of arithmetic. Careful people make them, usually in public, usually while trying to be responsible about evidence. The ones below all happen after a report is already in your hands, at the point where you turn its numbers into a sentence or a chart. Where a method needs a worked reference, the examples come from the Bureau of Labor Statistics and the Census Bureau, whose methods are public.
What to take away
- A median describes a point in a distribution, not a person, so it is a poor stand-in for a typical creator.
- A chart drawn between two studies compares their methods before it compares creators.
- A figure repeated in ten places is one figure with nine retellings.
- "Weighted" means little until the report says what it was weighted to.
- "No reliable data exists" is a finding, and it is publishable.
Reading a median as a typical experience
A median is the midpoint of a distribution, with half of the people above it and half below. The Bureau of Labor Statistics defines its median earnings level as the midpoint of a given distribution. It describes a position, not a person.
In a field where a small number of participants earn a large share, that position can sit far from what most readers are living. The median says half earn less. It does not say what the person at the median does, spends, or keeps after costs.
The honest use of a median is comparative and internal: how a distribution moved between two waves of the same study, measured the same way. The dishonest use is as a target. A creator who is told the median is a goal has been handed a description of other people.
Comparing across studies that measured different things
Two reports, two years, two publishers, and a chart drawn between them. The chart looks like a finding. It is an artifact of the comparison, because different frames and different definitions produce different quantities.
Putting those quantities on the same axis manufactures a trend from methodology. Do this only within one study's own repeated waves, and only where the method note says the method held between them.
A figure that does not say whether it is before or after platform fees, refunds, costs and tax cannot sit on the same axis as one that does. The guide to income benchmarks explains why a missing definition is the normal condition rather than a rare lapse, so ask about it before any chart is drawn.
One Source, Four Sightings
- Original study collects the data
- Article cites the figure
- Three more cite the first
- Aggregator table repeats it
- Feels established, but one source
Before you chart two figures together, check that they are not one figure. A number appears in an article, then in three more that cite the first, then in an aggregator table, and it now feels established. Follow the trail back before treating agreement as confirmation.
Where the trail ends at a summary rather than at the document that collected the data, there is nothing to verify. That is the same trap that makes a published brand deal case study impossible to check.
Assuming weighting fixed the sample
Weighting corrects a sample that is skewed on measurable characteristics, against a population estimate you trust. Neither condition is easy to meet for creators, because nobody has counted them.
The Census Bureau's method for the Current Population Survey shows what the condition looks like when it is met: the sample weights are adjusted so that estimates match independent population controls. That works because an independent count of the population exists.
For creators there is no population estimate, so adjustment is usually made against something adjacent. A report that says it weighted, without saying to what, has told you nothing. Ask for the target, and ask who produced it.
A weight also cannot restore the people who are missing. Everyone who left the field is absent from every study of it, and a sample read as a population keeps pushing results in a predictable direction however it is weighted.
Citing the press release instead of the study
Press releases select the finding that reads best and drop the base size, the field dates and the definition. Everything you needed to assess the claim is in the part that was cut.
Keep a copy of the original document. Links move, reports get replaced quietly, and a citation to a page that has changed is worse than no citation.
Notice, too, who paid. Most research in this area is published by organizations selling to the people being measured. That is not a scandal and some of the work is careful, but the questions were chosen by someone with an interest in the answer, and the sample is often a customer list.
Publishing a number you cannot substantiate
Repeating a striking figure in your own post, or on a page selling something, makes it your claim. Where the copy came from a sponsor, it is still your claim, which is one reason approval rights deserve a close read in a brand agreement.
When a claim supports a sale or endorsement, substantiation is a legal expectation, not an editorial preference. The FTC's Guides Concerning Use of Endorsements and Testimonials in Advertising, 16 CFR Part 255, set out what a results claim must support.
Disclosure and substantiation are separate obligations. Meeting one does not satisfy the other, and a disclaimer does not rescue a claim that nobody checked.
Concluding nothing rather than concluding uncertainty
The last mistake is the opposite of the others. Faced with weak evidence, writers often drop the subject, and the reader is left with the strongest-sounding number in circulation.
"The population cannot be enumerated, so no reliable figure exists, and here is what to measure instead" is a genuine finding. It is more useful to a reader than a number with a caveat attached. The practical alternative is your own records rather than a better survey, and reading affiliate income metrics shows how to start from them.
A short check before you cite anything
Five Questions Before Citing
- Read the original document, not a summary
- Know the base size for the claim
- State the income definition in one sentence
- Know when the data was collected
- Accept publisher interest printed beside you
- Have you read the original document rather than a summary of it?
- Do you know the base size for the specific claim you are quoting?
- Can you state the income definition in one sentence?
- Do you know when the data was collected?
- Would you be comfortable if the publisher's commercial interest were printed beside your sentence?
Five questions, two minutes, and they catch nearly everything above.
Common questions
Is it ever acceptable to cite a figure with no method note?
Only as an example of a claim in circulation, and only if you say that is what you are doing. As evidence, no.
What if my audience wants a number and I have none?
Give them the reason and the alternative. Readers are more tolerant of "nobody can measure this, here is how to measure yourself" than most writers expect.
How do I handle a sponsor who wants a statistic in the copy?
Ask for the source, read it, and decline the sentence if it will not stand up. The claim runs under your name, whoever wrote it.
Does citing a study make my post more credible?
Only if the study supports the sentence. A citation attached to a claim it does not support is worse than none, because it borrows authority the source never gave.
