Data Science & AI interview questions — with answers

Default values are evaluated once, when the function is defined. def add(item, items=[]) reuses the same list on every call, so items accumulate across calls. Use items=None and create a new list inside the function instead.

A dict stores key-value pairs in a hash table. Python hashes the key to find its slot, so lookups, inserts and deletes are O(1) on average. Keys must be hashable; since Python 3.7 dicts keep insertion order.

A set holds unique, unordered, hashable items and checks membership in O(1) on average; a list is ordered, allows duplicates and checks membership in O(n). Use a set to remove duplicates or test 'is x in this collection?' quickly.

A lambda is a small anonymous function written in one expression: lambda x: x * 2. They are handy as short arguments to sorted(key=...), map(), filter() or Pandas apply(), but named functions are clearer for anything non-trivial.

map(f, items) applies f to every item; filter(f, items) keeps items where f returns True; functools.reduce(f, items) combines items into one value, such as a running product. In modern Python, comprehensions often replace map and filter for readability.

with uses a context manager to set up and reliably tear down a resource. with open('data.csv') as f: guarantees the file is closed even if an error occurs. You can write your own context managers with __enter__/__exit__ or contextlib.contextmanager.

In CPython, the GIL allows only one thread to execute Python bytecode at a time. Threads still help for I/O-bound work (network, disk), but CPU-bound work needs multiprocessing, native libraries like NumPy that release the GIL, or other approaches.

Use threads (or asyncio) for I/O-bound tasks such as calling many APIs, because the GIL is released while waiting. Use multiprocessing for CPU-bound tasks such as heavy computation, because each process has its own interpreter and GIL.

asyncio is Python's library for asynchronous, single-threaded concurrency using async def and await. While one task waits on I/O, the event loop runs others. It is efficient for many concurrent network calls, such as hitting several LLM or web APIs at once.

A class is a blueprint that defines attributes and methods; an object is an instance of that class. __init__ initialises a new object's attributes, and self refers to the instance the method is called on.

Encapsulation bundles data and behaviour and hides internals; inheritance lets a class reuse another's behaviour; polymorphism lets different classes share an interface (e.g. each has a .predict()); abstraction exposes what an object does while hiding how.

A staticmethod takes no automatic first argument — it's a plain function grouped in the class. A classmethod receives the class (cls) as its first argument and is often used for alternative constructors, such as Model.from_config(cfg).

Methods with double underscores, like __init__, __str__, __repr__, __len__, __eq__ and __add__, let your objects work with built-in syntax and functions — printing, len(), comparisons and operators.

__str__ returns a readable, user-friendly string used by print(). __repr__ returns an unambiguous developer-facing string, ideally one that could recreate the object, used in the REPL and debugging. If __str__ is missing, Python falls back to __repr__.

Inheritance lets a child class reuse and extend a parent class. super() calls the parent's method — commonly super().__init__(...) inside the child's __init__ — so the parent's setup still runs. Python resolves methods using the MRO (method resolution order).

An iterable is anything you can loop over (lists, strings, dicts) because it has __iter__. An iterator is the object that produces the values one by one with __next__ and remembers its position. iter(iterable) returns an iterator.

yield turns a function into a generator: it returns a value, pauses the function and keeps its local state, then resumes from the same point on the next request. This produces values lazily without building a full list.

Like list comprehensions but with parentheses: sum(x * x for x in range(10**6)). They yield items lazily instead of creating a full list, which saves memory when you only need to iterate once.

Process it in pieces: pd.read_csv(path, chunksize=100_000) returns chunks you can aggregate one at a time, or read only the columns you need with usecols and efficient dtypes. Python's csv module with a loop also streams row by row.

append(x) adds x as a single element — appending a list adds a nested list. extend(iterable) adds each element of the iterable individually. [1,2].append([3,4]) gives [1,2,[3,4]]; extend gives [1,2,3,4].

Use list(dict.fromkeys(items)) — dict keys are unique and preserve insertion order. list(set(items)) also removes duplicates but loses the original order.

Use slicing with a negative step: text[::-1]. For lists you can also call items.reverse() to reverse in place, or reversed(items) for an iterator.

Slicing extracts part of a sequence with seq[start:stop:step]; start is included, stop is excluded. nums[1:4] gives items 1–3, nums[-3:] the last three, nums[::2] every second item. Slicing a list returns a new list.

Formatted string literals introduced in Python 3.6: f"Revenue: {total:,.2f}". They embed expressions directly inside strings, support format specifiers, and are faster and more readable than % formatting or .format().

list.sort() sorts the list in place and returns None. sorted(iterable) returns a new sorted list and works on any iterable. Both accept key= and reverse=, e.g. sorted(people, key=lambda p: p['age']).

CPython uses reference counting — objects are freed when no references remain — plus a cyclic garbage collector that finds groups of objects referencing each other. Memory comes from a private heap managed by the interpreter.

A virtual environment (venv, conda) is an isolated Python installation with its own packages. It keeps each project's dependency versions separate, avoids conflicts, and makes environments reproducible with requirements.txt or environment files.

A module is a single .py file. A package is a directory of modules (traditionally with an __init__.py) that can be imported with dotted names, like sklearn.model_selection.

When a file runs directly, its __name__ is "__main__"; when it's imported, __name__ is the module name. Code under this check runs only when the file is executed as a script, not on import — useful for tests or a main() entry point.

Optional annotations such as def mean(xs: list[float]) -> float. Python doesn't enforce them at runtime, but they improve readability, editor autocomplete and let tools like mypy catch type errors before you run the code.

A closure is an inner function that remembers variables from its enclosing function even after the outer function has returned. Closures power decorators and function factories such as make_multiplier(3) returning a function that triples its input.

Python looks up names in this order: Local (inside the current function), Enclosing (outer functions), Global (module level) and Built-in. Use global or nonlocal only when you deliberately need to rebind an outer variable.

Pickling serialises a Python object into bytes with the pickle module so it can be saved or sent and later restored (unpickling). It's often used to save trained models. Never unpickle data from untrusted sources — it can execute code.

Use the json module: json.loads() parses a JSON string into dicts and lists, json.dumps() converts Python objects to a JSON string, and json.load()/json.dump() work with files. requests responses also offer .json().

Use the requests library: r = requests.get(url, params=..., headers=..., timeout=10), then r.raise_for_status() and data = r.json(). For POST requests send json=payload. Always set timeouts and handle errors and rate limits.

zip() pairs items from several iterables position by position: list(zip(names, scores)) gives [(name1, score1), ...]. It stops at the shortest input. dict(zip(keys, values)) is a quick way to build a dictionary.

enumerate(items, start=0) yields (index, item) pairs, so you can loop with an index without managing a counter: for i, row in enumerate(rows, start=1):

Broadcasting lets NumPy perform arithmetic on arrays of different shapes by virtually stretching the smaller one, without copying data. For example, subtracting a row of column means from a 2D array centres every column in one vectorised step.

NumPy arrays store items of one type in contiguous memory and run operations in optimised C code (vectorisation), avoiding Python's per-element overhead. Lists store references to separate Python objects of any type.

Measure first with time, timeit or cProfile to find the hot spots. Then vectorise with NumPy/Pandas instead of loops, use the right data structure (sets/dicts for lookups), cache repeated work with functools.lru_cache, and parallelise I/O-bound tasks.

COUNT(*) counts all rows. COUNT(column) counts rows where that column is not NULL. COUNT(DISTINCT column) counts unique non-NULL values.

NULL means unknown, so comparisons with = or <> return unknown, not true. Use IS NULL / IS NOT NULL. Aggregates such as SUM and AVG ignore NULLs, and COALESCE(col, 0) replaces NULL with a default.

COALESCE returns the first non-NULL value from its arguments: COALESCE(phone, mobile, 'Not provided'). It's commonly used to fill missing values in reports or before calculations.

Group by the columns that define a duplicate and keep groups with more than one row: SELECT email, COUNT(*) FROM users GROUP BY email HAVING COUNT(*) > 1.

Number the rows within each duplicate group and delete the extras: WITH d AS (SELECT id, ROW_NUMBER() OVER (PARTITION BY email ORDER BY id) AS rn FROM users) DELETE FROM users WHERE id IN (SELECT id FROM d WHERE rn > 1).

A self join joins a table to itself using aliases. Classic example — list each employee with their manager: SELECT e.name, m.name AS manager FROM employees e LEFT JOIN employees m ON e.manager_id = m.id.

A CROSS JOIN returns the Cartesian product — every row of one table combined with every row of the other. It's useful for generating combinations, such as every store × every date, but can explode in size if used by accident.

A subquery is a query nested inside another. A correlated subquery references columns from the outer query and runs once per outer row, e.g. employees earning more than their department's average. They can often be rewritten as joins or window functions for speed.

IN checks whether a value is in a list or subquery result; EXISTS checks whether a subquery returns any row. EXISTS stops at the first match and handles NULLs more predictably, so it's often preferred for correlated checks. Beware NOT IN with NULLs, which can return no rows.

Window functions compute a value for each row using a set of related rows, without collapsing them like GROUP BY. They use OVER (PARTITION BY … ORDER BY …) — for example running totals, rankings, moving averages and comparing each row to the previous one.

Use SUM as a window function: SELECT order_date, amount, SUM(amount) OVER (ORDER BY order_date) AS running_total FROM orders. Add PARTITION BY customer_id for a running total per customer.

LAG(col) returns the value from a previous row and LEAD(col) from a following row in the window order. They're used for month-over-month change: revenue - LAG(revenue) OVER (ORDER BY month).

Rank inside each group and filter: WITH r AS (SELECT *, ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) AS rn FROM products) SELECT * FROM r WHERE rn <= 3.

Use a window frame: AVG(sales) OVER (ORDER BY day ROWS BETWEEN 6 PRECEDING AND CURRENT ROW). This averages the current row and the six before it.

An index is a separate data structure (usually a B-tree) that lets the database find rows without scanning the whole table, much like a book index. It speeds up reads on filtered, joined and sorted columns but slows writes and uses storage.

A clustered index defines the physical order of rows in the table, so there can be only one (often the primary key). A non-clustered index is a separate structure pointing to the rows; a table can have many.

Too many indexes slow down INSERT, UPDATE and DELETE because each index must be updated. Indexes on low-selectivity columns (like a yes/no flag) rarely help, and functions on indexed columns in WHERE can stop the index being used.

Normalisation organises tables to reduce redundancy and update anomalies by splitting data into related tables. 1NF: atomic values, no repeating groups. 2NF: no partial dependency on part of a composite key. 3NF: no transitive dependency between non-key columns.

Denormalisation deliberately adds redundancy — for example storing customer city on the orders table — to make reads faster and simpler. It's common in data warehouses and reporting, at the cost of more storage and harder updates.

OLTP systems handle many small, fast transactions such as orders and payments, with normalised tables. OLAP systems support analysis over large historical data, usually with denormalised star schemas in a data warehouse.

A warehouse design with a central fact table of measurable events (sales amount, quantity) and foreign keys to surrounding dimension tables (date, product, customer, store). It keeps analytical queries simple and fast.

A view is a saved query that behaves like a virtual table; it runs each time you query it. A materialised view stores the query result physically, so it's faster to read but must be refreshed to stay current.

A stored procedure is a named block of SQL (and procedural logic) saved in the database and executed with a call. It can take parameters, run multiple statements and encapsulate business logic close to the data.

Atomicity (all or nothing), Consistency (the database moves from one valid state to another), Isolation (concurrent transactions don't interfere) and Durability (committed changes survive crashes). They make transactions reliable.

A transaction groups statements into one unit of work: BEGIN, run the statements, then COMMIT to save or ROLLBACK to undo. The classic example is a bank transfer — debit and credit must both succeed or both fail.

Rules enforced by the database: PRIMARY KEY, FOREIGN KEY, UNIQUE, NOT NULL, CHECK (e.g. price > 0) and DEFAULT. They protect data quality at the source.

CHAR(n) stores fixed-length strings padded with spaces; VARCHAR(n) stores variable-length strings up to n characters. Use CHAR for fixed codes like country codes, VARCHAR for most text.

Use CASE: CASE WHEN score >= 90 THEN 'A' WHEN score >= 75 THEN 'B' ELSE 'C' END AS grade. CASE also enables conditional aggregation such as SUM(CASE WHEN status = 'paid' THEN amount ELSE 0 END).

Use conditional aggregation, which works everywhere: SELECT product, SUM(CASE WHEN month = 'Jan' THEN sales END) AS jan, SUM(CASE WHEN month = 'Feb' THEN sales END) AS feb FROM sales GROUP BY product. Some databases also offer PIVOT.

Use the database's date functions to extract parts (EXTRACT or DATE_PART), truncate to a period (DATE_TRUNC('month', order_date)), add intervals and compute differences. Grouping by DATE_TRUNC is the standard way to build monthly reports.

Aggregate by month, then compare with the previous month using LAG: (revenue - LAG(revenue) OVER (ORDER BY month)) * 100.0 / LAG(revenue) OVER (ORDER BY month). Watch out for division by zero and integer division.

Use a LEFT JOIN and keep unmatched rows: SELECT c.* FROM customers c LEFT JOIN orders o ON o.customer_id = c.id WHERE o.id IS NULL. NOT EXISTS is an equally good alternative.

A JOIN combines columns from tables side by side based on a matching condition. A UNION stacks the rows of two queries with the same number and type of columns on top of each other.

DISTINCT removes duplicate rows from the result, which requires sorting or hashing. If duplicates come from a join, fixing the join or using EXISTS is often faster and more correct than adding DISTINCT.

LIMIT/OFFSET (or OFFSET … FETCH) works for small pages, but large offsets get slow because skipped rows are still read. Keyset pagination — WHERE id > last_seen_id ORDER BY id LIMIT 20 — stays fast on big tables.

SQL injection happens when user input is concatenated into a query and changes its meaning. Prevent it with parameterised queries (placeholders), ORMs or query builders, input validation, and least-privilege database accounts.

Recursive CTEs repeatedly join a query to its own result to walk hierarchical or graph data — org charts, category trees, bill of materials — or to generate number and date series.

Many databases support PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY salary). Without it, rank rows with ROW_NUMBER and COUNT and average the middle one or two values.

GROUP BY collapses rows into one row per group. PARTITION BY (inside OVER) divides rows into groups for a window function but keeps every row, so you can show each row alongside its group total or rank.

Clarify the tables, grain and expected output; write the result columns first; build the query step by step with CTEs; handle NULLs, duplicates and ties explicitly; and test the logic on a few sample rows out loud.

Skewness measures asymmetry. In a right-skewed (positive) distribution, like income, the long tail is on the right and the mean is greater than the median. In a left-skewed distribution the tail is on the left and the mean is below the median.

Kurtosis describes how heavy a distribution's tails are compared with a normal distribution. High kurtosis means more extreme values (fat tails), which matters in areas like finance and risk.

A range calculated from sample data that would contain the true parameter in a stated share of repeated samples — for a 95% CI, about 95 out of 100 such intervals. Wider intervals mean more uncertainty; larger samples make them narrower.

The default assumption of no effect or no difference — for example, 'the new checkout page does not change conversion rate'. A hypothesis test asks whether the data gives enough evidence to reject it in favour of the alternative hypothesis.

A two-tailed test checks for a difference in either direction; a one-tailed test checks only one direction (e.g. 'B is better than A'). Two-tailed is the safer default unless a direction was justified before seeing the data.

Power is the probability that a test detects a real effect of a given size (1 − β). It increases with larger samples, larger true effects, lower variance and a higher α. Experiments are usually designed for 80% power.

A t-test compares means. A one-sample t-test compares a mean to a value; an independent two-sample test compares two groups; a paired test compares before-and-after measurements on the same subjects. It's used when population SD is unknown.

To test relationships between categorical variables — for example whether course choice is independent of city — using a contingency table of counts, or to check whether observed counts fit an expected distribution.

Analysis of variance tests whether the means of three or more groups differ, by comparing variation between groups with variation within groups. A significant result says at least one mean differs; post-hoc tests show which.

As sample size grows, the sample mean gets closer to the true population mean. It's why averages from large samples are reliable and why casinos win in the long run.

It updates the probability of a hypothesis given new evidence: P(A|B) = P(B|A) × P(A) / P(B). It's the basis of spam filters, medical test interpretation and Naive Bayes classifiers.

Much lower than 99%. In 100,000 people, about 100 have the disease and ~99 test positive, while ~999 of the 99,900 healthy people also test positive. So only about 99 / 1,098 ≈ 9% of positives are real — the base rate matters.

The probability of an event given that another has occurred: P(A|B) = P(A and B) / P(B). For example, the probability a user buys given they added an item to the cart.

Events where one doesn't change the probability of the other: P(A and B) = P(A) × P(B). Coin tosses are independent; drawing cards without replacement is not.

It models the number of successes in n independent trials with the same success probability p — for example how many of 20 leads convert if each converts with probability 0.1.

It models the count of events in a fixed interval when events occur independently at a constant average rate — calls per hour, website errors per day. Its mean and variance are both λ.

When the sample doesn't represent the population. Examples: surveying only app users about app satisfaction, survivorship bias (studying only successful companies), and self-selection in voluntary reviews.

Simple random sampling, stratified sampling (sample within groups such as cities), cluster sampling (sample whole groups), systematic sampling (every k-th record) and convenience sampling, which is easy but often biased.

Visually with box plots and scatter plots, and statistically with z-scores (|z| > 3) or the IQR rule — values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR. Always investigate before removing: an outlier may be an error or the most important data point.

The range of the middle 50% of the data: IQR = Q3 − Q1. It's robust to outliers and is the basis of the box plot.

Both measure how two variables move together. Covariance depends on units, so its size is hard to interpret. Correlation (Pearson's r) is standardised between −1 and 1, making strength and direction easy to compare.

Pearson measures linear relationships between continuous variables and is sensitive to outliers. Spearman uses ranks, so it captures monotonic (consistently increasing or decreasing) relationships and is more robust.

When predictors are highly correlated with each other. It makes regression coefficients unstable and hard to interpret, though predictions may still be fine. Detect it with correlation matrices or VIF; fix it by dropping or combining features or using regularisation.

It models a target as a linear combination of predictors. Key assumptions: linear relationship, independent errors, constant error variance (homoscedasticity), normally distributed errors for inference, and no severe multicollinearity.

R² is the share of variance in the target explained by the model, from 0 to 1. It always rises when you add features, so adjusted R² — which penalises extra predictors — is better for comparing models.

Running many tests inflates false positives: at α = 0.05, about 1 in 20 tests will look significant by chance. Corrections such as Bonferroni or controlling the false discovery rate (Benjamini–Hochberg) address it.

Trying many analyses, subsets or stopping points until something becomes significant, then reporting only that. Avoid it by pre-registering the hypothesis and metric, fixing sample size in advance and reporting all tests.

A result can be statistically significant but too small to matter — a 0.1% conversion lift with millions of users. Always report the effect size and confidence interval and judge whether the change is worth acting on.

A measure of how large a difference or relationship is, independent of sample size — for example the difference in means, Cohen's d or a percentage lift. It's what decision-makers actually care about.

Decide the baseline rate, the minimum detectable effect, significance level (α, usually 0.05) and power (usually 80%), then use a power calculation. Smaller effects and noisier metrics need much larger samples.

Peeking repeatedly and stopping at the first significant result inflates the false-positive rate. Fix the duration or sample size in advance, or use sequential testing methods designed for continuous monitoring.

A trend that appears in several groups reverses when the groups are combined, because of a confounding variable such as group size. It's a reminder to segment data and think about confounders before drawing conclusions.

A variable that influences both the supposed cause and the effect, creating a misleading association — ice cream sales and drownings both rise with temperature. Randomisation or controlling for confounders addresses it.

Parametric tests (t-test, ANOVA) assume a distribution such as normality. Non-parametric tests (Mann–Whitney U, Wilcoxon, Kruskal–Wallis) make fewer assumptions and work on ranks — useful for small or skewed samples.

A resampling method: repeatedly sample from your data with replacement and recompute the statistic, building an empirical distribution. It gives confidence intervals for almost any metric without strong distribution assumptions.

The standard deviation of a sample statistic, such as the mean, across repeated samples. For the mean it's SD / √n, so it shrinks as the sample grows — that's why larger samples give tighter estimates.

The long-run average outcome of a random variable: the sum of each outcome times its probability. Businesses use it to compare options, such as the expected profit of a promotion.

A method that chooses the parameter values that make the observed data most probable under the model. It underpins logistic regression and many other statistical models.

Frequentists treat parameters as fixed and probability as long-run frequency, using p-values and confidence intervals. Bayesians treat parameters as uncertain, combine prior beliefs with data, and report posterior probabilities and credible intervals.

"If the new design truly made no difference, there would be only a 3% chance of seeing a result this big by luck. That's unlikely, so we think the improvement is real — and here is how big it is likely to be."

It fits a line (or hyperplane) that predicts the target as a weighted sum of features plus an intercept, choosing weights that minimise the sum of squared errors. It's simple, fast and highly interpretable.

It computes a weighted sum of features and passes it through the sigmoid function to get a probability between 0 and 1, then applies a threshold (often 0.5) to classify. It's trained by minimising log loss and is a strong, interpretable baseline.

At each node it tries possible splits and picks the one that best separates the target — maximising information gain (entropy reduction) or Gini impurity reduction for classification, or variance reduction for regression.

An unrestricted tree keeps splitting until it memorises the training data. Control it with max_depth, min_samples_leaf, min_samples_split, pruning, or by using ensembles such as random forests and gradient boosting.

It builds trees one after another; each new tree is trained on the residual errors (gradients of the loss) of the current ensemble, and its predictions are added with a learning rate. XGBoost, LightGBM and CatBoost are popular, fast implementations.

To predict a new point, KNN finds the k closest training points by a distance measure and takes a majority vote (classification) or average (regression). It needs scaled features and becomes slow on large datasets.

An SVM finds the boundary that separates classes with the widest possible margin. The kernel trick lets it learn non-linear boundaries by implicitly mapping data to higher dimensions. It works well on medium-sized, high-dimensional data.

It applies Bayes' theorem to classify, assuming features are independent given the class — rarely true, hence 'naive'. It's fast and works surprisingly well for text classification such as spam detection.

Choose k, place k centroids, assign each point to the nearest centroid, move each centroid to the mean of its points, and repeat until assignments stop changing. It assumes roughly spherical clusters and needs scaled features.

Use the elbow method (plot inertia against k and look for the bend), silhouette scores, and — most importantly — whether the clusters make business sense and are actionable.

Principal Component Analysis reduces dimensions by projecting data onto new, uncorrelated axes (principal components) that capture the most variance. It's used for visualisation, noise reduction and speeding up models; features should be scaled first.

As features increase, data becomes sparse and distances between points become less meaningful, so models need far more data and distance-based methods like KNN degrade. Feature selection and dimensionality reduction help.

Creating better model inputs from raw data — ratios, date parts, aggregates over time windows, text features, interaction terms. Good features often improve a model more than switching algorithms.

One-hot encoding for nominal categories with few values; ordinal encoding when categories have an order; target encoding or embeddings for high-cardinality columns, done carefully inside cross-validation to avoid leakage.

Distance- and gradient-based models (KNN, SVM, neural networks, regularised regression) are sensitive to feature scale. Standardisation rescales to mean 0 and SD 1; min-max normalisation rescales to a 0–1 range. Tree models generally don't need scaling.

When information that wouldn't be available at prediction time sneaks into training — for example a 'refund issued' column when predicting fraud, or scaling the full dataset before splitting. It produces great validation scores and poor real-world results.

Train fits the model, validation is used to tune hyperparameters and choose models, and the untouched test set gives a final, unbiased estimate. Reusing the test set for tuning makes its score optimistic.

Parameters are learned from data (regression weights, tree splits). Hyperparameters are set before training (learning rate, max_depth, number of trees, k) and are tuned with validation data.

Grid search tries every combination of chosen values — thorough but expensive. Random search samples combinations and often finds good settings faster when only a few hyperparameters matter. Bayesian optimisation is a smarter third option.

A table of true positives, false positives, true negatives and false negatives for a classifier. Precision, recall, specificity and accuracy are all calculated from it, and it shows exactly which mistakes the model makes.

The ROC curve plots true positive rate against false positive rate across all thresholds; AUC is the area under it — the probability the model ranks a random positive above a random negative. 0.5 is random, 1.0 is perfect.

With heavy class imbalance, ROC-AUC can look good even when the model produces many false positives. The precision-recall curve focuses on the positive class, so PR-AUC better reflects performance for rare events like fraud.

MAE (average absolute error, easy to explain), RMSE (penalises large errors more), MAPE (percentage error, problematic near zero) and R². Choose the one that matches the business cost of errors.

Don't default to 0.5. Plot precision and recall against thresholds and pick the point that matches business costs — for example the threshold that keeps false alarms manageable while catching most fraud.

Understand why it's missing, then drop, impute (mean/median/mode, model-based) or add a 'was missing' indicator. Fit imputers on training data only. Some models, like gradient boosting libraries, handle missing values natively.

Ensembles combine several models to get better, more stable predictions — bagging, boosting and stacking. Stacking trains a meta-model on the predictions of several base models.

Layers of connected units (neurons) that each compute a weighted sum of inputs and apply a non-linear activation. Stacking layers lets the network learn complex patterns; weights are learned with backpropagation and gradient descent.

The algorithm that computes how much each weight contributed to the error by applying the chain rule backwards from the output layer. These gradients are then used by an optimiser to update the weights.

Non-linear functions applied to each neuron's output — ReLU, sigmoid, tanh, softmax. Without them, a deep network would collapse into a single linear model and couldn't learn complex patterns.

Gradient descent updates parameters in the direction that reduces the loss. The learning rate sets the step size: too high and training diverges or oscillates, too low and it's slow or gets stuck. Adam and SGD with momentum are common optimisers.

In deep networks, gradients can shrink towards zero (so early layers stop learning) or grow huge (unstable training). Fixes include ReLU activations, careful initialisation, batch normalisation, residual connections and gradient clipping.

A regularisation technique that randomly switches off a fraction of neurons during training, forcing the network not to rely on any single path. It's turned off at inference time.

Convolutional neural networks use learnable filters that slide over images to detect edges, textures and shapes, building up to objects. They're used for image classification, object detection and medical imaging.

RNNs process sequences step by step but struggle with long-range dependencies; LSTMs add gates to remember information longer. Transformers use self-attention to look at all positions at once, train in parallel and now dominate NLP.

Starting from a model pre-trained on a large dataset and fine-tuning it for your task — for example a pre-trained image model adapted to classify product photos. It needs far less data and compute than training from scratch.

When one class is much rarer than another. Use suitable metrics, class weights, oversampling (e.g. SMOTE) or undersampling inside cross-validation, collect more minority examples and tune the decision threshold.

Use global importance (permutation importance, feature importances) and local explanations such as SHAP values or LIME for individual predictions, plus partial dependence plots. Simple models like logistic regression are explainable by design.

Save the trained model and preprocessing pipeline, wrap them in an API (e.g. FastAPI) or batch job, containerise with Docker, version the model and data, and monitor latency, errors and prediction quality after release.

Data drift is when input distributions change; concept drift is when the relationship between inputs and target changes. Monitor input statistics and live performance, set alerts, and retrain on recent data when quality drops.

Define churn and the prediction window with the business, assemble historical features (usage, tenure, complaints, payments) without leakage, split by time, start with a logistic regression baseline then try gradient boosting, evaluate with PR-AUC and recall at a useful threshold, explain drivers with SHAP, and plan how the retention team will act on the scores.

An initial investigation of a dataset — shape, data types, summary statistics, distributions, missing values, outliers and relationships between variables — using tables and charts before formal analysis or modelling.

Descriptive: what happened. Diagnostic: why it happened. Predictive: what is likely to happen. Prescriptive: what we should do about it. Most analyst work is descriptive and diagnostic.

Key performance indicators measure progress towards a business goal — conversion rate, revenue per customer, churn rate. Good KPIs are tied to a goal, clearly defined, measurable from reliable data, and owned by someone who can act on them.

An Excel (or Google Sheets) tool that summarises data by dragging fields into rows, columns, values and filters — for example total sales by region and month — without writing formulas. Pivot charts visualise the result.

Automatically highlighting cells based on rules — values above target in green, duplicates in red, colour scales or data bars — so patterns and exceptions stand out in a sheet.

SUMIF adds values that meet one condition; SUMIFS adds values that meet several conditions (e.g. region = South and month = March); COUNTIFS counts rows meeting several conditions. They're the formula-based alternative to pivot tables.

Use Data → Remove Duplicates on the chosen columns, or the UNIQUE() function to return distinct values dynamically. Check first which columns define a duplicate, and keep a copy of the original data.

Rules that restrict what users can enter in cells — drop-down lists, number or date ranges, text length — with optional input messages and error alerts. It keeps manually entered data clean.

A relative reference (A1) shifts when a formula is copied; an absolute reference ($A$1) stays fixed. Mixed references ($A1 or A$1) lock only the column or row — essential for copying formulas across tables.

The data-preparation engine in Excel and Power BI. It connects to sources, then cleans and reshapes data through recorded, repeatable steps — removing columns, splitting, merging, unpivoting and changing types — so refreshes apply the same transformations automatically.

Power BI Desktop for building reports (Power Query for data prep, a data model with DAX, and report visuals), the Power BI Service for publishing, sharing, scheduling refresh and dashboards, and mobile apps for viewing.

A report is a multi-page, interactive analysis built from one dataset in Power BI Desktop. A dashboard (in the Power BI Service) is a single-page canvas of pinned tiles that can come from several reports, used for at-a-glance monitoring.

Filter context is the set of filters applied to a calculation by slicers, visuals and CALCULATE. Row context is the current row when iterating, as in calculated columns or iterator functions like SUMX. Understanding both explains most DAX results.

SUM adds up a single column. SUMX iterates over a table row by row, evaluates an expression for each row (e.g. Quantity × Price) and then sums the results.

Create a measure for the current value, then the prior-year value with CALCULATE([Sales], SAMEPERIODLASTYEAR('Date'[Date])), and divide: DIVIDE([Sales] - [Sales LY], [Sales LY]). It needs a proper, marked date table.

Time-intelligence functions need a continuous calendar table with one row per date, marked as the date table and related to the fact table. It also gives consistent year, quarter, month and week columns for slicing.

Relationships link tables on key columns so filters flow between them. Cardinality describes the link — one-to-many is the norm (one product, many sales), while many-to-many should be used carefully. Filter direction controls which way filters travel.

A Power BI feature that restricts the data each user sees using DAX filter rules on roles — for example regional managers see only their region's sales — while sharing one report.

Import loads data into Power BI's compressed in-memory engine — fast, but refreshed on a schedule. DirectQuery leaves data in the source and queries it live — always current, but slower and with some modelling limits.

Use a star schema, remove unused columns and rows, prefer measures over calculated columns, reduce high-cardinality columns, limit visuals per page, avoid complex bi-directional relationships, and check Performance Analyzer to find slow visuals and DAX.

A clear purpose and audience, the most important KPIs at the top, consistent layout and colours, the right chart for each question, minimal clutter, useful filters, clear labels and definitions, and data people trust.

Line charts for trends over time, bar charts for comparing categories, stacked bars for parts of a whole across groups, scatter plots for relationships, histograms for distributions, and tables when exact values matter. Avoid 3D and crowded pie charts.

Grouping users by a shared starting point — such as signup month — and tracking their behaviour over time, like retention by month since signup. It shows whether newer customers behave better or worse than older ones.

Measuring how many users move through each step of a process — visit → sign up → add to cart → pay — and where they drop off. It points to the steps worth fixing first.

Customer segmentation by Recency (how recently they bought), Frequency (how often) and Monetary value (how much). It identifies best customers, at-risk customers and those to win back.

Confirm the data is correct, then break the drop down by time, region, channel, product and customer segment to locate it; check for tracking changes, outages, pricing or stock issues, competitor actions and seasonality; and quantify each likely cause.

Comparing two versions — say two landing pages — by randomly splitting users and measuring a pre-defined metric. The analyst checks the sample size, runs the significance test, reviews guardrail metrics and explains whether the change is worth rolling out.

Data analysts focus on extracting, analysing and visualising data to answer questions. Business analysts focus on understanding business processes and requirements and translating them into solutions; the roles often overlap.

Extract data from sources, Transform it (clean, join, aggregate) and Load it into a destination such as a data warehouse. Analysts often do lightweight ETL with SQL, Power Query or Python.

A central database designed for analysis that stores cleaned, integrated historical data from many systems, usually modelled as star schemas. Examples include BigQuery, Snowflake, Redshift and Azure Synapse.

Pandas to load, clean, merge and aggregate data; Matplotlib or Seaborn for charts; Jupyter notebooks for exploration; and scripts to automate repetitive reports and pull data from APIs or databases.

Use pd.merge(left, right, on='customer_id', how='left'), choosing inner, left, right or outer like SQL joins. Check row counts afterwards to make sure the join didn't create duplicates.

It splits data into groups by one or more columns, applies an aggregation or function to each group and combines the results: df.groupby('city')['revenue'].agg(['sum', 'mean']).

Identify them with box plots, z-scores or the IQR rule, investigate whether they're errors or genuine extremes, then correct errors, and for genuine values consider medians, capping or analysing them separately — and say what you did.

Documentation listing each table and column with its meaning, data type, allowed values, source and owner. It helps everyone interpret data consistently and speeds up onboarding.

Clarify the business impact and deadline of each, agree priorities with stakeholders, deliver quick wins early, reuse existing reports where possible, and communicate timelines openly.

Use the STAR method — Situation, Task, Action, Result — with a specific example: the question, the data you used, what you found and the measurable effect of the decision it led to.

Access only the data needed, mask or aggregate personal information in reports, follow company policy and applicable law on personal data, store files securely, and avoid sharing raw data by email or personal devices.

A metric is a number you measure (revenue, orders, sessions). A dimension is an attribute you slice metrics by (date, city, channel, product category).

Listen to their reasoning, re-check the data and assumptions together, show the method transparently, test their alternative explanation with data, and agree on what additional evidence would settle it.

Running several attention operations in parallel, each with its own learned projections, then combining them. Different heads can capture different relationships — syntax, references, topic — at the same time.

Attention itself doesn't know token order. Positional encodings (fixed sinusoidal, learned, or rotary embeddings such as RoPE) add order information so the model can tell 'dog bites man' from 'man bites dog'.

Encoder-only models (BERT) read the whole input and are great for classification and embeddings. Decoder-only models (GPT-style) generate text left to right and power most chat LLMs. Encoder-decoder models (T5) map an input sequence to an output sequence, as in translation.

Pre-training teaches a model general language patterns from huge unlabelled text by predicting tokens. Fine-tuning then adapts it to specific behaviour or tasks with smaller, curated datasets — instructions, conversations or domain examples.

Fine-tuning a pre-trained model on many examples of instructions paired with good responses, so it follows user requests instead of simply continuing text.

Reinforcement Learning from Human Feedback: humans rank model responses, a reward model learns those preferences, and the LLM is optimised to produce responses the reward model scores highly. It improves helpfulness and safety. Alternatives such as DPO learn directly from preference pairs.

Designing prompts that reliably produce the output you want: clear instructions, relevant context, examples, a defined output format and constraints. Good practice is to test prompts against a set of cases, not just one example.

Zero-shot gives only an instruction; one-shot adds a single example; few-shot adds several examples of input and desired output. Examples help the model match format and style, especially for unusual tasks.

Asking the model to work through a problem step by step before giving a final answer. It can improve performance on reasoning, maths and multi-step tasks; many newer models also perform this kind of reasoning internally.

Instructions set by the application, not the end user, that define the model's role, rules, tone and output format for the whole conversation — for example 'You are a support assistant; answer only from the provided documents.'

Ways to limit which tokens the model can choose. Top-k samples from the k most likely tokens; top-p (nucleus sampling) samples from the smallest set whose probabilities add up to p. Both trade diversity against coherence, alongside temperature.

Describe the schema precisely, show an example, and use the API's structured-output or JSON mode or tool calling where available. Always validate the result with a schema validator and retry or repair on failure.

A database optimised to store embeddings and run fast similarity search, usually with approximate nearest-neighbour indexes. Examples include pgvector (PostgreSQL), Pinecone, Weaviate, Qdrant, Milvus and Chroma.

Cosine similarity measures the angle between vectors, focusing on direction (meaning) rather than length. Euclidean distance measures straight-line distance. For normalised embeddings they give equivalent rankings; cosine is the usual choice for text.

Splitting documents into smaller passages before embedding. Chunks should be small enough to be specific but large enough to keep context — often a few hundred tokens with some overlap, ideally split on headings or paragraphs. Test sizes against real questions.

Combining keyword search (such as BM25) with vector search. Keywords catch exact terms like product codes and names; vectors catch meaning. Merging both usually retrieves better context for RAG than either alone.

After retrieving a broader set of candidates, a re-ranker model (often a cross-encoder) scores each passage against the question more precisely and keeps the best few. It improves relevance at modest extra cost.

Evaluate retrieval (did we fetch the right passages? recall, precision at k) and generation (is the answer faithful to the sources, relevant and complete?). Use a test set of real questions with expected answers, human review, and LLM-as-judge with care.

Common causes: the right document was never indexed, poor chunking, weak retrieval or no re-ranking, too much irrelevant context, outdated data, or the prompt not forcing the model to stick to the sources. Debug retrieval first.

Low-Rank Adaptation is a parameter-efficient fine-tuning method that freezes the original weights and trains small low-rank matrices added to certain layers. It cuts memory and compute dramatically; QLoRA combines it with a quantised base model.

Storing model weights with fewer bits — for example 8-bit or 4-bit instead of 16-bit — to reduce memory and speed up inference, usually with a small loss in quality. It's how large open models run on modest hardware.

Training a smaller 'student' model to imitate a larger 'teacher' model's outputs, producing a faster, cheaper model that keeps much of the teacher's ability.

Malicious instructions hidden in user input or retrieved content that try to override your system prompt — for example a web page saying 'ignore previous instructions'. Defend by separating instructions from data, limiting tool permissions, validating outputs, filtering inputs and requiring human approval for risky actions.

Crafting prompts that trick a model into ignoring its safety rules. Providers mitigate it with safety training and filters; applications add their own moderation, restricted capabilities and monitoring.

Use the smallest model that meets quality, shorten prompts, cache repeated prompts and results, retrieve fewer but better chunks, stream responses, batch requests, and route only hard queries to larger models.

Receiving the response token by token as it's generated instead of waiting for the whole answer. It makes chat interfaces feel much faster, even though total generation time is similar.

Models that understand or generate more than one type of data — text, images, audio, video. Examples: describing a chart image, answering questions about a PDF with diagrams, or generating images from text.

Diffusion models learn to reverse a gradual noising process: starting from random noise, they denoise step by step, guided by a text prompt's embedding, until a coherent image emerges.

Closed API models are usually the most capable and easy to use, but you depend on the provider and send data to them. Open-weight models can be self-hosted for privacy, control and customisation, but need infrastructure and expertise.

Models can only read a fixed number of tokens at once. Work around it with retrieval (send only relevant chunks), summarising long histories, map-reduce summarisation over long documents, and choosing long-context models when needed.

Measuring output quality — correctness, faithfulness, helpfulness, safety, format. It's hard because answers are open-ended. Use task-specific test sets, automatic checks where possible, human review and carefully calibrated LLM-as-judge methods.

Using a strong LLM with a rubric to grade another model's outputs at scale. It's useful but can be biased (towards longer answers or its own style), so validate it against human ratings.

Checks around the model: input filtering, topic restrictions, output validation (schemas, banned content, PII detection), grounding checks, rate limits and human review for high-risk outputs.

Customer support assistants, internal knowledge search, document summarisation, data extraction from forms and emails, code assistance, content drafting, meeting notes and sales email personalisation — usually with a human reviewing important outputs.

Frameworks that provide building blocks for LLM apps — model wrappers, prompt templates, document loaders, chunking, vector store integrations, retrieval pipelines and agents. Many teams also build directly on provider SDKs for more control.

Minimise what you send, mask or remove PII where possible, check the provider's data retention and training policies, restrict access to logs, comply with applicable data-protection law, and keep sensitive workloads on approved infrastructure.

Making a model's answer depend on specific, verifiable sources — retrieved documents, database results or tool outputs — rather than its general training data, and ideally citing those sources.

Caching LLM responses keyed by the meaning of the query (via embeddings) rather than the exact text, so similar questions reuse a previous answer. It cuts cost and latency for repetitive queries, with a similarity threshold to avoid wrong matches.

Collect and clean documents, chunk them with metadata, embed and index in a vector store, retrieve with hybrid search and re-ranking, prompt the LLM to answer only from the retrieved passages with citations, add access control per user, and evaluate with real employee questions before launch.

It produces fluent text but can be confidently wrong, doesn't know your private data unless you connect it, can be inconsistent, and costs money per use. It works best for drafting, summarising and searching, with humans reviewing important decisions.

Breaking a goal into ordered sub-tasks before or during execution. Approaches include plan-then-execute (make a full plan, then run it) and iterative planning (decide one step at a time and re-plan as results arrive).

Plan-and-execute writes a full plan first and then carries out the steps — efficient and easy to review for predictable tasks. ReAct decides step by step from observations — more adaptive for uncertain tasks but can wander without limits.

Short-term memory is the current conversation and scratchpad in the context window. Long-term memory persists across sessions — facts, preferences, past results — usually stored in a database or vector store and retrieved when relevant.

Summarise earlier steps, keep only relevant tool outputs, store large results externally and pass references, trim tool responses to the fields needed, and split work across sub-agents with their own contexts.

Several specialised agents working together — for example a researcher, a writer and a reviewer — coordinated by a supervisor or by passing messages. It helps with complex tasks but adds coordination cost and new failure modes.

A supervisor agent receives the task, breaks it down, delegates sub-tasks to worker agents with specific tools, collects their results and assembles the final answer. It keeps each worker focused and easier to test.

When a single agent with good tools — or a simple deterministic workflow — can do the job. Multi-agent setups cost more tokens, are harder to debug and can amplify errors, so add them only when specialisation clearly helps.

In a workflow, the developer defines the steps and the LLM fills in parts (classify, extract, draft) along fixed paths. In an agent, the model decides the steps dynamically. Workflows are more predictable; agents are more flexible. Many production systems mix both.

Pausing the agent for human review or approval at key points — before sending emails, making payments, deleting data or when confidence is low. It balances automation with safety and accountability.

Build a test set of realistic tasks and measure task success rate, correctness of the final output, number of steps, cost and latency, tool-call errors and safety violations. Review full trajectories, not just final answers.

Assessing the whole sequence of an agent's reasoning and tool calls — did it choose sensible tools, in a sensible order, without unnecessary or risky steps — rather than only checking whether the final answer was right.

Calling the wrong tool, invalid arguments, infinite loops, giving up too early, hallucinating tool results, losing track of the goal in long contexts, following injected instructions from web pages or documents, and running up costs.

Set a maximum number of steps and a cost budget, detect repeated identical tool calls, require progress checks, and fall back to asking a human or returning a partial result when limits are reached.

Least-privilege tool permissions, allow-lists for actions and domains, input and output validation, approval for irreversible actions, spending limits, rate limits, sandboxed code execution and complete logging for audit.

Agents read untrusted content — web pages, emails, documents — that may contain instructions like 'send the data to this address'. Because agents can act, injection can cause real damage. Treat tool outputs as data, limit permissions and require approval for sensitive actions.

Give an agent only the tools and access it needs for its task — read-only where possible, scoped to specific records or folders — so mistakes or attacks have limited impact.

Tools should return clear, structured error messages. The agent can then retry with corrected arguments, try an alternative tool, or report the problem. Orchestration code should add retries with backoff for transient failures.

Tracing every model call, tool call, input, output, latency and cost in a run so you can debug failures, find slow or expensive steps and monitor quality in production.

Use smaller models for simple steps, cap steps and tokens, cache tool results, keep context lean, avoid unnecessary multi-agent chatter, and monitor cost per task with alerts.

Examples include LangGraph, CrewAI, AutoGen, the OpenAI Agents SDK, the Claude Agent SDK and LlamaIndex agents. Many teams also write a simple loop directly with a provider's tool-calling API for full control.

A library for building agents and workflows as graphs of nodes (steps) and edges (transitions) with shared state. It supports loops, branching, persistence and human-in-the-loop checkpoints.

The information carried between steps — the goal, messages, intermediate results, which steps are done and any variables — that lets the agent resume, branch or hand off work reliably.

Retrieval becomes a tool: the agent decides when to search a knowledge base, can reformulate queries, search several sources and combine results — often called agentic RAG — instead of retrieving once before answering.

An agent that reads a codebase, plans changes, edits files, runs tests or commands in a sandbox, inspects the results and iterates until the task passes. It needs strong sandboxing and review before changes are merged.

An agent that operates a browser or desktop by reading the screen and issuing clicks and keystrokes. It can automate tasks without APIs but is slower and needs careful safety controls.

Good candidates have a clear goal, need several steps or tools, have verifiable outcomes and tolerate occasional mistakes or allow review. Highly regulated, irreversible or simple single-step tasks are often better as fixed workflows.

Having the agent (or a separate reviewer agent) critique its draft or plan against criteria and revise it. It can improve quality but adds cost and can't replace external verification such as tests.

Run the new version against a regression suite of recorded tasks, compare success rate, cost and safety metrics with the old version, review failures, then roll out gradually with monitoring.

A program that exposes tools, resources (data) and prompts through the Model Context Protocol. Clients such as AI assistants connect to it and can then list and call its tools in a standard way.

Give it tools to read calendars, check availability, propose slots and send invites; define rules (working hours, time zones, meeting length); require confirmation before sending invites; handle conflicts and errors; and log every action.

Tools for web search, reading pages and taking notes; a plan step that breaks the question into sub-questions; a loop that searches, reads and records sources; a synthesis step with citations; and limits on steps and domains.

By pausing with a clear summary of what was done, what's needed and why — then waiting for approval or input through a queue, ticket or chat message — and resuming from saved state afterwards.

How much the agent can do without human approval — from suggesting actions only, to acting with approval, to fully autonomous within limits. Start low and increase autonomy as evaluation shows the agent is reliable.

Agents pass data between steps and tools. Validated, schema-based outputs prevent parsing failures and silent errors, and make it easy to check each step programmatically.

Agents that run for minutes or hours across many steps. Challenges include persisting state, recovering from crashes, staying within context limits, controlling cost, and keeping humans informed of progress.

Emerging standards that let agents built by different teams or vendors discover each other, exchange tasks and messages, and collaborate through a common interface, much as MCP standardises tool access.

Lead research and enrichment, support ticket triage and drafting, invoice and document processing, internal knowledge assistants, report generation and data entry across systems — tasks with clear inputs, outputs and reviewable results.

Compare time saved per task, task completion rate, error rate, cost per task and user satisfaction against the manual baseline, and track how often humans must correct or take over.

Python, LLM APIs and prompting, tool and API design, RAG, state management, evaluation and testing, security thinking (permissions, prompt injection), observability, and an understanding of the business process being automated.

"A normal AI chat answers your question. An AI agent is more like an assistant you give a goal to — it figures out the steps, uses apps and data to do them, checks its work, and asks you before doing anything important."