How OneStar reads reviews, and how often it is right.

What a report reads

A report names its app or game, its stores, its window and how it picked reviews. These are the rules every report follows.

The same questions, asked of each review

Each review is read on its own by Jev, a language model built for typed judgments. It answers the same 12 questions for every report of a kind of product, and apps and games each have their own list of complaint and praise categories.

Every question is closed. A yes or no question comes back as a probability, a category question as one option from a fixed list with a confidence, and the two scales, how badly a problem hurts and how the reviewer feels overall, as one level of five. Jev never writes a free text answer that code then has to interpret.

Each report names the model and question set it was read with in its About panel. The newest public report was read by Jev (jev-1.14.0) with question set v5.

Jev answers, code does every count

Jev never counts. Shares, trends, star averages, dates and versions are computed by code from the stored answers, so every figure is exact for the reviews read and can be rebuilt without asking Jev again.

Shares carry 95% Wilson intervals on the effective sample size. A difference between apps, audiences or periods is called only when a two proportion test holds after a Benjamini-Hochberg correction for the number of tests made.

Where LLMs come in

On Max reports and on OneStar's public reports, a large language model (LLM) writes short passages next to the numbers: the Reading on the overview, a note beside each top theme, a line on each audience that stands out, notes on spikes and releases, the reasons on each side of an Ask, the verdict of a saved comparison and the opening of the weekly digest. It reads only what code has already counted and the stored excerpts of at most 200 characters, after personal details were removed.

Inside the top themes, the LLM also proposes a few candidate reasons. It does not decide which reviews give them: Jev judges each review of the theme against the candidates, and code counts each reason with a likely range, beside an other row and an unsure row.

It never types a figure or a quote. It writes a placeholder, and code fills in the number or the excerpt from the report. A check drops any passage whose figures do not resolve, that copies review text, or that claims a difference the statistics did not find. The LLM does not judge reviews and nothing it writes changes a count. OneStar is model agnostic, so the LLM behind these passages can change without changing the method.

Unsure answers are shown, not forced

A category answer below its question's confidence cutoff is not forced into a category. It goes to an unsure bucket that every report shows, next to other and unclear, which stay visible too.

The cutoffs were set from the gold sets, so that answers above them are right about as often as their confidence says. The table below lists each one.

On the gold sets, 24.8% of app complaints and 20.5% of game complaints went to unsure, and the complaint categories that remained were right 92.7% of the time for apps and 92.0% of the time for games.

Scored on labeled reviews

Accuracy is measured on gold sets: public store reviews labeled with the answer each question should get, scrubbed and cut to length exactly as a report does it.

The standard set holds 300 reviews for apps and 300 for games, each labeled for every question in the standard set.

The leaving set holds 640 reviews from public reports, labeled for what the reviewer says about their own choice to keep using the product: already left, quitting now, cancelled a paid plan, might leave, came back or nothing about leaving. The 542 app reviews shaped the wording of the leaving question, so the app score is measured on reviews it was tuned on. The 98 game reviews were held out and never used for tuning.

A smaller switching set holds reviews from people who left, labeled for where they went: the named rival, somewhere else, unclear or nowhere. It scores the switching question that comparisons ask, and is not in the table below.

The labeling guide asks for what the text says, as a careful analyst would, without looking at any model output. The star rating is context, not the answer: a five star review can carry a complaint and a one star review can carry praise. A row a second labeler could reasonably label differently is marked uncertain and kept.

The labels were drafted by a language model, and rows where the draft and Jev disagreed were read again by a model that could see Jev's answers (56 for apps and 56 for games). That makes every accuracy number below an upper bound. No person has checked a random sample of these labels yet.

How often Jev agrees with the labels

Measured October 7, 2026 on category taxonomy v7. A yes or no score is the share of reviews answered the same way as the label. A category score counts only confident answers; the share that went to unsure is beside it.

What the method cannot do

How to cite a report

Link the report page and name OneStar and the review window, for example: