Vectors

Embedding

Next, tokens representing words, in our case the word ‘diplomacy’, are transformed into vectors through an embedding process. Each token is represented by a high-dimensional vector, often comprising thousands of dimensions. To visualise this complex multidimensional space, the following diagram offers a simplified representation in just three dimensions.

You can imagine vectors as coordinates on a massive, invisible map of ideas. Each point on this map corresponds to a specific concept or object. The closer two points are, the more semantically similar their meanings are. Meaning is represented by measurable distance between vectors. This is what allows AI systems to understand that ‘diplomacy’ is more closely related to ‘negotiation’ than to, let’s say, ‘apple’.

What makes vectors so powerful is their ability to represent the subtle relationships between ideas (Farnschläder, 2024). When we represent real-world objects and concepts, such as images, text, or sounds, as vector embeddings, we create a structure where semantic similarity (how similar two things are in meaning) becomes geometric proximity (how close their points are in vector space).

Vectors are determined according to a pre-defined ‘dictionary (the embedding table or matrix). For example, word diplomacy will be embedded according to pre-defined vectors developed during the training of each LLM (e.g. GPT, Gemini, DeepSeek).

A word’s vector is defined by its numeric coordinates, called floating-point numbers^[1]. In our example, the vector for the word ‘diplomacy’ is composed of the following values:

0.34, -0.58, 0.71, -0.15, 0.92, -0.37, 0.12, 0.48, -0.22, 0.65

The diagram below offers a simplified, two-dimensional visualisation of these numbers.

Each dimension in this vector captures a different aspect of the word’s meaning based on its relationship with other words. For ‘diplomacy,’ these relationships include:

• Peacefulness: Since diplomacy aims to resolve conflict, its vector is mathematically close to words like ‘peace,’ ‘negotiation,’ and ‘resolution.’
• Formality: The formal nature of diplomacy positions it near concepts such as ‘protocol,’ ‘etiquette,’ and ‘procedure.’
• International Context: The vector also reflects a global scope, linking it to terms like ‘treaty,’ ‘ambassador,’ ‘embassy,’ and ‘foreign policy.’

Notably, negative values in the vector help define the word by contrast, pushing it away from opposing concepts such as ‘war’ or ‘conflict.’

Once words are tokenised and embedded, they are combined into a single vector that captures the overall meaning of the entire input, whether that is a sentence, a paragraph, or a larger chunk. This is called a passage or sentence embedding. For example, the question ‘What is diplomacy?’ becomes a single vector representing its overall meaning. But how does this work, given that computers process only numbers, not words?

Sentence and chunk embeddings are created by analysing the numerical representations of all tokens in the input, taking into account word order, grammar, and relationships between words (Reimers & Gurevych, 2019). The result is a single dense vector that captures the semantic meaning of the whole passage, which is precisely what allows the system to compare the vector of your question against the vectors of all stored chunks and identify the most relevant ones.

You might also come across the term vector direction in AI discussions. This refers to how a vector is oriented in that multidimensional AI space. A data point is another term which is often used loosely in AI discussions. In technical communication, a data point refers to a single instance of data, represented as a vector, corresponding to something like a word or an image. In our example, the data point is the word ‘diplomacy’.

What are embeddings, and how are they created?

Let us dive deeper into embedding as the heart of LLMs. The process of creating embeddings maps words, sentences, or other inputs into a continuous numerical space where similar meanings are positioned closer together, capturing their underlying semantic relationships.

Embeddings encode the semantic essence of data, allowing systems to understand relationships beyond literal words (Bergmann & Stryker, n.d.-b). For instance, an embedding model takes text (e.g. ‘What is diplomacy?’) and produces a vector that captures its meaning, positioning it close to texts about negotiation, statecraft, and communication, and far from unrelated topics such as ‘how to brew green tea’.

So, how does an AI learn to position ‘negotiation’ near ‘dialogue’ and ‘treaty’? It learns from patterns. Again, we return to the fundamental principles of AI. The AI reads millions of texts, policy papers, treaties, and news articles, and learns which words and ideas tend to appear together. Words such as ‘embassy’, ‘envoy’, and ‘negotiation’ frequently go together, making a semantic cluster around diplomacy. The model then generates a dense numerical vector, an embedding, that captures the relationships it has learnt.

For context, in the early days of AI, engineers designed vectors manually. For example, in medical imaging, experts might create numerical features that describe colours, shapes, and textures. While effective in narrow domains, this approach required extensive domain knowledge and was impossible to scale. Today, modern AI learns embeddings automatically.

Because embeddings are contextual, documents mentioning ‘negotiation’ in a diplomatic context will be closer to one another than a document discussing negotiation in corporate or sports settings, assuming the embedding model is sensitive to context, as many today’s models are. This mathematical process mirrors how humans think. When we hear the word ‘cat’ or ‘dog’, we do not recall every fact about it; we recall associations such as ‘fur’, ‘pet’, or ‘animal’. The brain encodes meaning through patterns and proximity, just like how embeddings position related ideas near each other in vector space.

Vector databases: libraries of meaning

You will recall that in the earlier module, we discussed how our system (MyDepartment) broke documents you had uploaded into chunks, cleaned them, and converted them into vectors via embedding models. Vectors are stored in the vector databases, such as Weaviate.

Each chunk of text has two linked components within the vector database:

1. The embedding vector – a list of floating-point numbers – captures the meaning of each chunk of text in multidimensional space.
2. The reference to the original text allows the retrieval of its source document or chunk once the vector is identified.

It is important to note that when you ask a question, the system does not scan every word in every document. Instead, it transforms your question into its embedding vector, compares it against stored document vectors, and finds which ones lie closest in semantic space. The system then returns those corresponding text chunks.

Notes

[1]In embeddings, floating-point numbers allow the model to capture subtle differences between ideas. For example, 0.92 might mean ‘strongly related’, and 0.05 might mean ‘barely related’.