Shotnest · Library
Keep punctuation differences visible
Comparing “note, ready” with “note ready” gives a token overlap of only 0.333, because the comma stays attached to the first word.
Why one comma costs two thirds of the overlap
After normalization the compare tool splits text on spaces and nothing else. Punctuation is not stripped, so “note,” with its comma and “note” without one are two different tokens. In the example the first text has the tokens note, and ready; the second has note and ready. The union holds three distinct tokens and only ready is shared, so the overlap is 1 ÷ 3.
The same mechanism applies to full stops, colons, closing brackets and quotation marks glued to a word. Shotnest’s app search behaves differently: it looks for each typed word inside the screenshot text, so “note” still finds “note,” there.
Longer text softens the effect without removing it. “Meet at the station. Bring tickets” and “Meet at the station bring tickets” share five of seven distinct tokens, an overlap of 0.7142857142857143, and the only differing tokens are “station.” with its full stop and “station” without it.
Try the original example
- First text
- note, ready
- Second text
- note ready
- Exactly equal
- No
- Equal after normalization
- No
- Token overlap
- 0.3333333333333333
Read the result line by line
- Exactly equal: No, because the strings differ by a comma.
- Equal after normalization: No; normalization changes case and accents, not punctuation.
- Token overlap: 0.3333333333333333.
- Only in first: “note,”. Only in second: “note”.
- Look at the screenshot to decide whether the comma is meaningful or an artefact of copying.
- Distinct tokens in the union
- 3 tokens
- Shared tokens
- 1 token
- Token overlap
- 0.3333333333333333
Accents and letter case are handled before the split, so “Café,” and “cafe,” become the same token; only the attached comma or full stop keeps two otherwise identical words apart.
Mistakes this example prevents
- Deciding that two captions are unrelated because a short phrase with one comma scores low.
- Assuming the lower-scoring text is the less accurate one; overlap has no notion of correctness.
- Removing punctuation by hand before comparing and then forgetting that the original had it.
- Using overlap on very short texts, where one token changes the value by a third or more.
What the number cannot settle
Overlap compares sets of tokens. It does not say which version was typed first, which one appears in the image, or whether the comma changes the meaning. The compare tool linked below shows the first-only and second-only lists so you can see exactly which attached marks made the difference.