Check a script against rules you can read
An eval is a list of rules written before anyone looks at the answer. Pick a topic, read its rules, then see how two drafts do. Change a draft and run the checks again.
- Runs in your browser
- No model, no key
- Your draft is not sent anywhere
1. Pick a topic
- Written for
- developers new to AI
- How hard it is to write well
- Easy
- What you learn
- A loop that asks a model to improve its own work has no natural end. You learn to give it two exits: a quality threshold and a hard cap on rounds.
Approved by Hassan Mansour as a test case
The sources a script may cite
- [S1]Revision loops. A writer and critic loop repeats until the critic's score reaches a threshold. If the score never reaches it, the loop runs until something else stops it.
- [S2]Revision caps. A maximum number of revisions ends the loop even when the threshold is never met. The last draft is kept and marked as not meeting the bar.
- [S3]Recursion limits. Graph frameworks such as LangGraph stop a run that exceeds a configured number of steps, called the recursion limit. It is a backstop, not the main exit.
Demo sources written for this set. They are not quotes from publications.
2. Two drafts, same rules
Each draft starts as the work of a fake writer: a short program, not a model. One is careful and one is careless on purpose. Edit either one, or paste your own.
Draft A
Draft B
A script has a title, a hook, a body and a call to action, with a blank line between them. Cite a source like this: [S1]. The checks are written for scripts in English.
Try it: fix Draft A until every check passes.
3. The rules, and how each draft did
Every rule is checked by code. Open a rule to see why it exists and how to fix a draft that fails it.
Draft A
5 of 10 checks passed
105 of 140 words
2 checks not run: they need a model or a person
Computed when this page was built
The link holds your text, the settings and the result, so anyone who opens it can run the same check again. It is made in your browser: nothing is sent. Anyone you give it to can read your text.
Draft B
10 of 10 checks passed
76 of 140 words
2 checks not run: they need a model or a person
Every check that code can run passed. That is not the same as a good script: the questions below still need a model or a person.
Computed when this page was built
The link holds your text, the settings and the result, so anyone who opens it can run the same check again. It is made in your browser: nothing is sent. Anyone you give it to can read your text.
Every script, whatever the topic
Has a title, a hook, a body and a call to action
Draft A
Fail
citation in the title
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- A reader finds the point faster when a script has the same four parts every time. A citation in the title is a sign the parts got mixed up.
- How it is checked
- Code splits the text at blank lines and counts the blocks. Fewer than three fails. So does a [S1] mark in the first block.
- How to fix a draft that fails
- Put a blank line between the title, the hook, the body and the call to action. Keep citation marks out of the title.
Is at most 140 words long
Draft A
Pass
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- The script is read aloud in about a minute. A longer one gets cut by someone else, in the wrong place.
- How it is checked
- Code counts the words. Citation marks like [S1] are not counted.
- How to fix a draft that fails
- Cut. One idea per script.
Every citation points to a source that exists
Draft A
Fail
1 citation(s) to unknown sources
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- A citation to a source that is not there looks like evidence and is none. It is the easiest invention to catch, so it is caught first.
- How it is checked
- Code collects every [S…] mark and compares it with the list of sources for this topic.
- How to fix a draft that fails
- Cite only the ids listed under the sources for this topic.
Every sentence in the body carries a citation
Draft A
Fail
7 body sentence(s) without a citation
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- In the body, each sentence is a claim. A claim with no source cannot be checked by the reader.
- How it is checked
- Code splits the body into sentences and looks for a [S…] mark in each. It cannot tell a claim from a transition, so it is strict on purpose.
- How to fix a draft that fails
- Add the source to the sentence, or cut the sentence.
This topic must
Stay within 140 words
Draft A
Pass
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- The script is a one-minute clip: 140 words is about 56 seconds spoken.
- How it is checked
- Code counts the words. Citation marks are not counted.
- How to fix a draft that fails
- Cut. One idea per script.
Cite only sources that exist
Draft A
Fail
1 citation(s) to unknown sources
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- A citation to a source that doesn't exist is worse than none: it looks checked and isn't.
- How it is checked
- Code collects every [S…] mark and compares it with the list of sources for this topic.
- How to fix a draft that fails
- Cite only the ids listed under the sources for this topic.
Cite at least 2 different sources
Draft A
Pass
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- One source is an anecdote; the script should stand on more than one.
- How it is checked
- Code counts how many different source ids appear in [S…] marks.
- How to fix a draft that fails
- Use more of the sources, and mark each one where you use it.
Mention at least one of: “cap”, “limit”, “maximum”, “max ”
Draft A
Pass
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- The fix is a bound on rounds; a script that never names one hasn't answered the topic.
- How it is checked
- Code searches the text for each term, ignoring capitals. It is a blunt check: it cannot tell 'there is a limit' from 'there is no limit'.
- How to fix a draft that fails
- Say the thing the topic is about, in plain words.
This topic must not
Give a number without a citation
Draft A
Pass
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- A number without a source is the most common way a fluent script is wrong.
- How it is checked
- Code looks at every sentence after the title. A digit with no [S…] mark in the same sentence fails.
- How to fix a draft that fails
- Add the source to the sentence that has the number, or remove the number.
Contain any of: “revolutionary”, “game-changer”, “game changer”, “10x”, “magic”
Draft A
Fail
1 forbidden term(s) present
Draft B
Pass
Why, and how to fix it
- Why this rule exists
- Hype words claim more than the sources support and cost the reader's trust.
- How it is checked
- Code searches the text for each forbidden term, ignoring capitals.
- How to fix a draft that fails
- Remove the term. If it came from a source, the source was giving an order, not a fact.
Questions code cannot answer
These rules are yes-or-no questions about meaning. Code cannot answer them. A model or a person has to, and here nobody has.
Should be trueNames two different exits: reaching a quality threshold, and a hard cap on rounds.
Draft A
Not run
UNMEASURED
Draft B
Not run
UNMEASURED
Why, and how to fix it
- Why this rule exists
- Two exits is the idea; one alone is half of it.
- How it is checked
- Not by code. This is a question for a model or a person.
- How to fix a draft that fails
- Read the draft yourself and answer the question honestly.
Should be falseClaims that a model will always converge if given enough rounds.
Draft A
Not run
UNMEASURED
Draft B
Not run
UNMEASURED
Why, and how to fix it
- Why this rule exists
- No source says this, and it is the belief the topic exists to correct.
- How it is checked
- Not by code. This is a question for a model or a person.
- How to fix a draft that fails
- Read the draft yourself and answer the question honestly.
The whole set, both fake writers
The same checks, run on all 20 topics when this page was built.
| Writer | Topics where every code check passed |
|---|---|
| The careful fake writer | 20 of 20 |
| The careless fake writer | 0 of 20 |
These two are short programs, not models. The numbers show that the checks can pass and that they can fail. They say nothing about any model, and nothing about the quality of the pipeline these checks were written for.
Not run yet
These are the results a chatbot cannot give you, and this page cannot give them yet either. No number appears here until a real run exists.
Scores for the questions code cannot answer
Not run yet UNMEASURED
The set has 35 of these questions. A judge model can answer them, but that costs a model call each, and a judge has to be compared with a person before its scores mean anything. Neither has happened.
Two models, or two prompts, side by side
Not run yet UNMEASURED
You can already compare two drafts above: paste one model's output as Draft A and another's as Draft B. A comparison run by this page, on every topic, does not exist yet.
What a run costs
Not run yet UNMEASURED
The checks on this page cost nothing: no model is called. The cost of producing a draft with a model has not been measured here, so none is shown.
This version has no place to enter a model key. Nothing on this page calls a model.