Shotnest · Library
Understand why repeated words do not add token weight
“red red blue” and “red blue” score a token overlap of 1, because the compare tool counts each distinct word once, however often it repeats.
Overlap is measured on sets
The compare tool turns each normalized text into a set of distinct tokens. A set has no counts: red appears in it once whether the text says red once or ten times. For “red red blue” the set is red and blue; for “red blue” it is the same two tokens. Shared tokens 2, union 2, overlap 2 ÷ 2 = 1.
Exact equality and normalized equality still say No, because the strings are different. Those two lines are the place where repetition is still visible.
A longer case behaves the same way. “pay pay pay now” and “pay now” also give an overlap of 1, although the first text says pay three times. The first-only and second-only lists stay empty because no distinct token is missing from either side.
Try the original example
- First text
- red red blue
- Second text
- red blue
- Exactly equal
- No
- Equal after normalization
- No
- Token overlap
- 1
Check a repeated word
- Compare the two texts and note the overlap of 1.
- Read “Exactly equal” and “Equal after normalization”: both No.
- Notice that the first-only and second-only lists are empty, since no token is missing from either set.
- Count the repeated word yourself in the original when the number of repetitions matters.
- Distinct tokens, first text
- 2
- Distinct tokens, second text
- 2
- Token overlap
- 1
Normalization runs before the set is built, so “Red”, “RED” and “red” all collapse into one token as well; capital letters cannot make a repetition count either.
When repetition carries meaning
In a list, a repeated line can mean a quantity: two identical items, a doubled charge, an instruction given twice. Set overlap hides that difference completely. Shotnest’s duplicate finder uses a similar set idea for screenshots, but only after the images have the same width and height and nearly the same visual fingerprint, and it compares words without digits.
- Concluding that two receipts are the same because their overlap is 1.
- Ignoring empty first-only and second-only lists that hide a repeated line.
- Using overlap to count how many times a word occurs.
A set measure, read as one
The value answers one question: which distinct tokens do the texts share? Order and frequency are outside it. Use the compare tool linked below together with the original text whenever a count matters.