AI models don’t understand words; they understand numbers.
Words are converted into numbers through tokenisation. Each large language model uses its own token-to-number mapping, meaning the same word may be assigned a different numerical value depending on the model. Different tokenisation approaches reflect decisions made by developers, the nature of the training data, and broader architectural choices. This is an area where AI governance negotiators could explore convergence, for example, by pursuing international standards for tokenisation where every word or token would be assigned a consistent numerical representation, regardless of which LLM was used to develop or deploy it.
Practically, as you can see below, surveyed LLMs give different token numbers to the word ‘diplomacy‘. In some cases, it is borken down into two parts (‘diplom‘ and ‘acy‘) or even three, like ‘diplo‘, ‘m’, and ‘acy‘. This suggests that the term ‘diplomacy‘ may not be as common in the datasets these models were trained on.
For the sake of simplicity, we’ll consider ‘diplomacy’ as a single token, as it is treated in LLaMA 3.
In addition, subword tokenisation is a technique used to split words into smaller, meaningful units called subwords. This method is commonly used in natural language processing (NLP) models to handle out-of-vocabulary words, rare words, and morphemic structures. By breaking words into subwords, models can better understand complex and compound words, thereby improving their generalisation ability.