Perspective
Write the Policy, Train People to Apply It, Then Let Models Take the Clear Cases
The order of work decides moderation quality. We start with the policy and an agreement test, then build models on your own examples and measure them before they reach your players.
4-minute read5 November 2026
In Brief
- A model bought first produces faster decisions, angrier players and an appeals queue no one can answer, because the rule that was applied cannot be named.
- We see four actions: test the policy for agreement between moderators, give models only the clear cases, measure them by category and language, and design appeals for the support desk.
- Thinqmesh builds an evaluation set from your own reports in week one, then builds the model feature with a person approving anything high-stakes.
When abuse reports begin to overwhelm a growing platform or game, the first proposal is often a classifier. It seems the obvious answer to a volume problem, and a vendor demonstration can be persuasive. A few months later a company that took this route can find itself with faster decisions, angrier players and an appeals queue no one can answer, because the rule that was applied cannot be named. Our view is that the order of work decides the outcome: policy first, then people who can apply it consistently, then models that take over the parts people have shown to be clear.
A Model Can Only Learn a Decision Someone Has Already Made
A moderation model is trained on labels, and a label is a past judgement under a policy. If the policy says no harassment and leaves the rest to instinct, the labels record the instincts of whoever was on shift, and the model repeats them at speed. Give the same batch of reports to two trained moderators and see how often they agree. Where they differ, the rule needs worked examples.
Models then route and rank, people handle ambiguous cases, severe ones and appeals, and support sees the rule and the evidence behind every decision. The EU's transparency reporting rules for platforms ask for a description of any automated moderation, so the working has to be explainable.
Four Actions That Put Policy Ahead of the Model
Each action is cheap, needs no machine learning to begin, and produces the labelled set that any later model will need.
Test the Policy for Agreement Between Moderators
Give the same batch of reports to two trained moderators working independently and see how often they reach the same decision.
Where they disagree, the rule needs rewriting with worked examples. A moderation model is trained on labels, and a label is a past judgement under a policy. A policy that leaves the rest to instinct produces labels that record the instincts of whoever was on shift.
Get started
- Pull a hundred recent reports and remove the decisions already made.
- Have two moderators decide each one separately and compare.
Give Models the Clear Cases
Models route and rank. They are well suited to the first pass: obvious spam, repeat material and sorting the queue.
The most serious reports then reach a person first. Ambiguous cases, severe ones and appeals stay with people, and their decisions become the training data, so moderation and model improvement run as one loop. Context a text classifier may never receive, such as banter between friends against harassment of a stranger, stays with a person.
Get started
- List the report types where moderators already agree almost every time.
- Mark the types that depend on context across several matches.
Measure Models by Category and Language, in Both Directions
One headline accuracy figure conceals both the harmful material missed and the innocent users penalised.
We measure each category and each language separately, including code-mixed Hindi and English written in Roman script, and report both kinds of error. The evaluation set is built from your own reports, so the figures describe your community.
Get started
- Choose the three languages that appear most in your reports.
- Count wrongly actioned and wrongly cleared cases for each.
Design Appeals for the People Who Answer Them
Appeals reach the support desk whether or not it was designed for them.
Agents who cannot see the rule applied and the evidence used can only apologise or escalate. An upheld appeal marks a place where the system was wrong, so appeal rates by rule and language should reach whoever owns the policy. We give agents the rule and the evidence on one screen.
Get started
- Ask support what they can see when a player appeals.
- Report last month's appeal rate by rule to the policy owner.
How Thinqmesh Solves This
We build evaluation in from the start, testing answers against real examples from your work. Your first working feature arrives within four weeks, and the full engagement runs to six.
- Week 1
An Evaluation Set From Your Reports
We build an evaluation set from real examples and give you a measured score, which shows whether AI or conventional software is the right tool before further investment.
- Week 2
The First Model, on the Clear Cases
A working feature is built on your own data, with the model chosen for answer quality, running cost and where your data may go.
- Week 3
Access Controls and an Activity Record
Access is limited to what each person may see, and every decision leaves an activity record that support can read.
- Week 4
Monitoring of Quality and Cost
Your feature is in use, with monitoring that tracks answer quality and running cost as use grows.
You finish with a working AI feature on your own data, an evaluation set that measures quality, access controls with an activity record, and monitoring of quality and cost.
