Generative AI Hands-On: LLMs, Prompts & RAG

How LLMs work (in one minute)

A large language model is a neural network (a transformer) trained on large amounts of text to predict the next token. A token is a chunk of text — often part of a word.

Generating a reply means predicting one token at a time, each conditioned on everything before it. The model has no live memory between requests: everything it should consider must be inside the prompt, within its context window.

• Context window — the maximum number of tokens the model can read at once
• Temperature — higher is more varied, lower is more deterministic
• Hallucination — confident output that isn't grounded in facts; reduce it by supplying sources

Prompt engineering patterns

• Be specific: state the task, audience, format and length
• Give context: paste the relevant data or document
• Show examples (few-shot) of the input → output you want
• Ask for structure: "Respond as JSON with keys title and summary"
• Break big tasks into steps, or ask the model to reason before answering
• Tell it what to do when unsure: "If the answer is not in the text, say so"

You are a support assistant for an online course platform.
Answer using ONLY the policy text below. If the answer is not there, reply "I don't know".
Keep the answer under 60 words.

<policy>
...paste policy here...
</policy>

Question: Can I switch batches after enrolling?

Calling a model from Python

Most providers offer a similar chat-style API: you send a list of messages and receive the model's reply.

# pip install anthropic   (other providers have similar SDKs)
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
reply = client.messages.create(
    model="<model-name>",
    max_tokens=300,
    messages=[{"role": "user", "content": "Explain overfitting in two sentences."}],
)
print(reply.content[0].text)

Embeddings and vector search

An embedding turns text into a list of numbers (a vector) so that similar meanings sit close together. Vector search finds the chunks of your documents closest in meaning to a question — even if they use different words.

Build a simple RAG pipeline

Retrieval-augmented generation grounds the model in your own documents: retrieve the most relevant passages, then ask the model to answer using only those passages.

• 1. Load documents and split them into chunks (a few hundred tokens each)
• 2. Create an embedding for every chunk and store it in a vector index
• 3. At question time, embed the question and fetch the top-k closest chunks
• 4. Put those chunks into the prompt and ask the model to answer from them
• 5. Return the answer with references to the source chunks

# Minimal RAG sketch using sentence-transformers + NumPy
from sentence_transformers import SentenceTransformer
import numpy as np

embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunks = ["...chunk 1...", "...chunk 2...", "...chunk 3..."]
vectors = embedder.encode(chunks, normalize_embeddings=True)

def retrieve(question: str, k: int = 2) -> list[str]:
    q = embedder.encode([question], normalize_embeddings=True)[0]
    scores = vectors @ q                      # cosine similarity
    return [chunks[i] for i in np.argsort(scores)[::-1][:k]]

context = "\n\n".join(retrieve("What is the refund window?"))
prompt = f"Answer using only this context:\n{context}\n\nQuestion: What is the refund window?"

Evaluate and improve

• Build a small test set of questions with known answers
• Check retrieval first — if the right chunk isn't retrieved, the answer can't be right
• Tune chunk size, overlap and top-k
• Log prompts and outputs so you can debug failures