Picture: Vea's notes on "Tokens": "birthday laughing" splits into three puzzle pieces, "birthday", "laugh" and "-ing". Arrows point to "a whole word" and "a tiny part".
Think Like Trixie, page 33 of 38
Tokens
The little pieces AI breaks your words into, like word puzzle pieces. Some are whole words, like "birthday." Some are tiny parts, like "-ing."
In a sentence: An AI might split "sunflowers" into three tokens: "sun," "flower" and "-s."
Vea asks: What tokens might an AI find in "jumping"?