TL;DR
I asked ChatGPT to choose a number between 1 and 30. It said 17.
Then I asked it to run a command that generates a random number between 1 and 30 using Python’s random package and show me the output. It said 4.
Both are a number in range. Only the second one came from a random number generator.
The lesson is not “ChatGPT is bad at random.” It is that the way you phrase the question decides which machine answers it. Most bad LLM output is not a model failure. It is a routing failure caused by a vague question.
The two prompts
Same intent. Same range. Two different subsystems.
What “training data” and “weights” mean
Two terms, because the rest of this depends on them.
Training data is the text the model was shown while training: web pages, books, code, forums, docs, plus human-written examples used for fine-tuning. It is a snapshot, and when training ends it is gone. The model does not keep a copy and cannot look anything up in it.
Weights are what is left. A model is a huge pile of numbers, and training tunes those numbers to get good at exactly one task: given some tokens, predict the next one. That is the entire objective. Writing SQL, explaining a race condition, all of it is a side effect of being very good at that one prediction.
After training the weights are frozen. Asking a question runs your tokens through those frozen numbers and produces a probability distribution over the next token. A sampler picks one, and it repeats.
So the weights are not a database of facts. They are a compressed model of how text tends to continue.
Which means asking for a random number is really asking: what token usually follows this question in human writing? That is not randomness. It is a popularity contest.
Why 17
Humans are bad at picking random numbers, and bad in very consistent ways.
We skip the endpoints, because 1 and 30 feel like we did not really choose. We skip round numbers like 10 and 20, because they feel deliberate. We skip the exact middle, because it feels like a cop-out. What is left is odd, non-round, slightly above center. For 1 to 10 people say 7. For 1 to 100 they say 37, 42 and 73. For 1 to 30, 17 lands right in that pocket.
The model learned that bias from the text those humans wrote. When you ask it to pick, it reproduces the shape of the human answer, because reproducing human text is the only thing it does.
Two objections that come up:
“Temperature makes it random.” Temperature adds noise, but it samples from the model’s distribution, not a flat one. If 17 holds a third of the probability, noise does not flatten that. It just lets a long shot through now and then.
“It gave me a different number last time.” Variance is not uniformity. A loaded die also gives different numbers. The question is not whether the output varies, it is whether every value is equally likely.
What a PRNG is
PRNG stands for pseudo-random number generator, and “pseudo” is doing real work in that name.
It is a deterministic algorithm with a chunk of internal state. You give it a starting state, called the seed. Each call stirs the state through a fixed transform, hands back a number, and moves to the next state. Same seed, same sequence, every time, on any machine.
That sounds like it defeats the purpose. It is the best part:
import random
random.seed(42)
print([random.randint(1, 30) for _ in range(5)]) # [21, 4, 1, 24, 9]
random.seed(42)
print([random.randint(1, 30) for _ in range(5)]) # [21, 4, 1, 24, 9]
If you never call random.seed(), Python seeds it at import from the operating system’s entropy pool, which is fed by genuinely unpredictable physical noise. So a normal run looks fresh, and a seeded run replays exactly. You pick which one you need.
That last part is what the weights can never give you. If a test picks a “random” record through the model and then fails, you cannot reproduce the failure. There is no seed to set.
| Uniform | Reproducible | |
|---|---|---|
random | Yes | Yes, if you seed it |
| Model weights | No | No |
The model is not a weaker random number generator sitting one step below random. It is not a random number generator at all. No state, no seed, no uniformity. It is a text predictor doing an impression of one.
What the second prompt changed
Run a command that generates a random number between 1 to 30. Use python random package to do it and show me the output.
Three things, and each one matters:
- It asks for an action, not an answer. “Run a command” cannot be satisfied by emitting a token, so the model reaches for a tool.
- It names the mechanism. “Use python random package” removes the freedom to improvise. Improvisation is where you get a right-looking answer produced the wrong way.
- It demands evidence. “Show me the output” gives you something to inspect instead of a claim.
The model’s job shrank from “be a random number generator” to “call one and report the result.” Much smaller job, and it is much better at it.
One security note, since I just told you to use random: it is not cryptographically secure. Feed an attacker enough consecutive outputs and they can recover the internal state and predict every future value. Fine for tests, sampling and shuffling. Not fine for tokens, passwords, session IDs, OTPs or password reset links. For those:
import secrets
secrets.randbelow(30) + 1 # 1..30, crypto-safe
Same lesson one level down. Both return a number in range, and the wrong choice is a bug no test catches until it is a CVE.
The rule
Ask for the mechanism, not the answer.
Every question goes to one of two places. The weights are fast and free, and they are the right destination for language work: explaining, drafting, summarizing, naming, refactoring, judgment calls. A tool is slower and leaves an artifact, and it is the right destination for anything with a correct answer that exists outside the model.
The model picks the route from how you phrase the question, and it leans toward the weights, because that path is cheap and always produces something that looks like an answer. Be vague, get the cheap path.
| Vague | Specific |
|---|---|
| “Pick a random user from this list” | “Run a script that picks one with random.choice, show the output” |
| “What’s the total of this column?” | “Sum it with pandas, show the code and output” |
| “How many days between these dates?” | “Compute it with datetime, show the output” |
| “What version of Flask does this repo use?” | “Grep the lockfile, show me the line” |
| “Is this JSON valid?” | “Run it through json.loads, report any exception” |
The left column is usually right, occasionally wrong, and never checkable. The right column is either right or it fails out loud. Loud failure beats quiet plausibility.
Some things are structurally unsafe to answer from weights, not just weak:
- Randomness. No entropy source.
- Arithmetic past small numbers. It learned the patterns of arithmetic, not arithmetic.
- Counting characters. The model sees subword tokens, not letters. That is the whole “how many r’s in strawberry” thing. It cannot see them.
- Anything current. Today’s date, your repo’s contents, whether a service is up. Training ended; the world did not.
- Anything you need to reproduce. No seed, no replay.
All of it is fine to hand to a model, as long as you hand it the tool too.
Check that it actually ran
Naming a mechanism only helps if the mechanism ran. If there is no code interpreter, or the sandbox failed, a helpful model will write the code and then produce output that looks exactly like the code ran. It did not. That output came from the weights wearing a costume.
- Look for the execution artifact. A tool call, a code cell, a stdout block. Prose saying “the output is 4” with no trace is not a run.
- Ask twice in one turn. Real generator output diverges. Faked output tends to repeat the number it already committed to.
- Make it fail on purpose. Ask it to run something that raises. No real traceback means nothing is executing.
That last one is the cheapest smoke test for any new AI surface. Feed it a division by zero. Either you get a ZeroDivisionError, or you just learned that every “output” it ever showed you was fiction.
Bottom line
17 was not a mistake. The model did exactly what it was asked: produce the most plausible continuation of “choose a number between 1 to 30.” The mistake was mine, for asking a text predictor to be a source of randomness.
The gap between the two prompts is not intelligence. It is specification. One asked for an answer. The other asked for a mechanism.
Same skill as writing a clear ticket or a clear API contract: say what you want, say how it should be produced, say how you will check it. Vague requirements have always produced wrong software. LLMs did not change that. They just made the wrong answer arrive faster, in full sentences, sounding sure of itself.
Ask for the mechanism. Then check that it ran.