Creator income research metrics to check before citing a figure

A report about creator work contains two kinds of numbers: the findings, and the numbers that describe how the findings were produced. The second kind decides whether the first kind means anything, and it is usually printed in smaller type near the back.

This page is about the second kind. No finding from any report is quoted here, because the point is the check rather than the answer.

What to take away

The base size for the specific claim

Not the headline sample. The number of people who answered the particular question you are reading about.

A survey of several thousand can yield a chart built on a few dozen respondents after cuts by income line, country, and experience.

Where a report shows a cut without saying how many people are in it, that cut is illustration, not evidence. Where it shows the base and the base is small, the finding is honest and fragile. That differs from wrong.

The sample frame

Where the people came from before anyone tried to reach them. A platform's active accounts, a software company's mailing list, an agency's roster, a panel maintained by a research firm.

Everything downstream inherits this. There is no register of creators, so no study samples the population; each one samples a list, and the list decides who could not appear. People who quit are the group most often missing, and their absence pushes every result in the same direction. The research pillar goes through what each method can and cannot see.

The response rate, and what it is not

The share of people approached who answered. A low rate is not automatically fatal and a high rate is not a guarantee, which is why the metric has to be read alongside the frame rather than on its own.

What matters is whether people who answered differ from those who did not in a way tied to the subject. On income, they almost always do.

The American Association for Public Opinion Research publishes a statement of professional standards setting out what a survey is expected to disclose about itself. A report that meets it can be argued with, the highest compliment available.

The weighting scheme

What the researchers adjusted for, and against what source of truth.

Weighting is legitimate and normal, and it corrects a skewed sample on measurable traits using a population estimate you trust.

Both halves are the problem. No one has a population estimate for creators, so weighting here usually uses something adjacent, like general population age and country. Read the note, and be careful of any report that says it weighted but not what it weighted to.

The definition of the quantity

The most consequential number in the report is usually a sentence. What counts as income here: money a platform paid, money invoiced, money received after fees, or money left after costs.

Without it, a figure cannot be compared to your bank statement or to another report. With it, half the apparent disagreements between reports resolve into different questions rather than different answers. Ask it first and ask it of everything, including your own records, where the benchmarks pillar explains why writing the definitions down once is the whole exercise.

The question wording

Reports rarely print the questionnaire. Ask for it, or look for an appendix.

Wording moves answers more than most readers expect. Small changes in framing produce different distributions, and Pew Research Center's discussion of writing survey questions is a readable account written by people who spend their working lives on it.

A question about earnings that omits a period, a currency, or how costs are treated will draw different answers from different respondents. The resulting spread gets reported as variation in the field.

The field period

When the data was collected, which is not when the report was published. A gap of a year is common.

For a subject where platform terms change frequently, that gap can be the whole story. A finding about how creators were paid is a finding about how they were paid during the field period, and the payout mechanics may have been rewritten twice since.

What good disclosure looks like

A report supplying all five can be assessed on its merits. One supplying none is a document with charts in it.

Using this on a report you are about to cite

If you plan to quote a figure in your own work, you are taking responsibility for it. Read the five rows above, keep a copy of the original document rather than a link to a summary, and state the definition alongside the number when you use it.

When a claim supports something you sell, the standard is higher: a performance claim in advertising must be substantiated. The Federal Trade Commission's endorsement and testimonial guidance is the plain statement of that expectation in the United States. Read it before a figure goes on a sales page.

The same standard applies to a statistic a sponsor asks you to repeat, which is covered under brand deals.

Common questions

If the method note is missing, is the report definitely bad?

Not definitely, but you cannot tell, which for practical purposes is the same thing. Set it down and spend the attention on one that describes itself.

Which single check catches the most problems?

The definition of income. It is one sentence, it is often absent, and its absence makes every figure in the document uncomparable.

Is academic work better than industry work here?

Often more careful about disclosure, and usually older and narrower. The frame problem is the same for everybody, because it comes from the absence of a population register rather than from a lack of rigor.

How do I cite research responsibly in my own posts?

Name the publisher, the field period and the definition, and link the original document. If that makes the sentence unwieldy, the figure was probably doing less work than you thought.