Numbers with addresses: rank is how many indices you need
Click any cell to read its address.
Indices needed
2
Shape
(4, 4)
Numbers held
16
The whole greyscale digit is a matrix. Each number needs a row h and a column w: x[1, 2] = 8. Shape (4, 4) says there are 4 positions along each of the two axes.
Switch between the four buttons and click cells to read their addresses. The number of indices you need to name one number is the tensor's rank, also called ndim, and the shape lists how many positions each axis has. The element count is the product of the shape, so (3, 4, 4) holds 48 numbers. This "rank" differs from the rank of a matrix in linear algebra, which counts independent rows: a 3×4 matrix always has two axes, but its linear-algebra rank can be 0, 1, 2, or 3.
Step 2 of 6: Memory Is One Line
Shape and strides turn an index into a slot in memory
c1
h1
w2
Index
[1, 1, 2]
Strides
(16, 4, 1)
Offset
1·16 + 1·4 + 2 = 22
Value
5
Memory has no rows or channels: the 48 numbers sit in one line. The tensor keeps a shape (3, 4, 4) and strides (16, 4, 1), meaning how many slots to jump when an index goes up by 1. Stepping w moves 1 slot, stepping h skips a row of 4, and stepping c skips a whole channel of 16. So x[1, 1, 2] lives at slot 22. The last index moves fastest, which is called row-major order.
Move c, h, and w and follow the pointer into memory. A tensor is a flat block of numbers plus two short tuples, the shape and the strides. The stride of an axis is the product of the sizes to its right, so shape (3, 4, 4) has strides (16, 4, 1), and x[c, h, w] sits at offset c·16 + h·4 + w. In PyTorch, x.stride() returns these numbers. Each float32 takes 4 bytes, so this tensor uses 192 bytes, while a batch of 32 RGB images at 224×224 holds about 4.8 million values, roughly 19 MB.
Step 3 of 6: Reshape vs Transpose
Same shape, different contents
Rows2
New shape
(2, 6)
Strides
(6, 1)
Data copied?
no
Reshape keeps the 12 numbers in memory exactly where they were and only changes the shape and strides used to read them. Read the grid left to right, top to bottom, and you get 0, 1, 2, … 11 again, the same order as memory. Nothing is copied, which is why reshape is almost free.
Slide Rows to regroup the 12 numbers, then switch to transpose. Reshape never moves data: it only sets a new shape and new strides, so it works exactly when the sizes multiply to the same total. Libraries let you write −1 for one size and infer it, so reshape(2, −1) gives (2, 6). Transpose swaps the strides instead, here from (4, 1) to (1, 4), so reading the result row by row jumps around memory. That is why PyTorch's view() fails on a transposed tensor, while reshape() and .contiguous() make a copy.
Step 4 of 6: Broadcasting
Size-1 axes stretch without copying
Click any sum to see where its two numbers came from. Dashed cells are copies that are never stored.
A, right-aligned
(3, 4)
B, right-aligned
(1, 4)
Per axis
✓ ✓
Result
(3, 4)
A is 3 examples with 4 features; B is one bias per feature. Line the shapes up from the right: 4 matches 4, and B has no first axis, which counts as 1. A size-1 axis is stretched to fit, so every row gets the same bias. This is exactly how y = xW + b adds b to every example. The dashed copies are never stored: the library just reads B's row again (stride 0).
Try each B and click the sums. NumPy-style broadcasting lines shapes up from the last axis: each pair of sizes must match, or one of them must be 1, and a missing leading axis counts as 1. The stretched copies are never stored, because the library reads the same numbers again with a stride of 0. The (3, 1) + (3,) case is a common silent bug, for example a (batch, 1) prediction minus a (batch,) target inside a loss, which gives a (batch, batch) matrix instead of an error.
Step 5 of 6: Element-wise vs Matrix Multiply
One pairs cells, the other sums away an axis
Operation
W shape
Click an output cell to see how it is computed.
Rule
(m,k) @ (k,n) → (m,n)
Result
(2, 2)
Multiply-adds
12
Each output is a dot product: row 0 of x times column 1 of W, multiplied term by term and summed. The shared axis, the 3 features, is used up by the sum and disappears, so (2, 3) @ (3, 2) gives (2, 2): one score per example per output. That is a dense layer. Each input number is reused once per output column, which is why GPUs are so good at it.
Pick an operation and a shape for W, then click output cells. Matrix multiplication sums over the shared axis, so (m, k) @ (k, n) gives (m, n) and costs m·k·n multiply-adds. Element-wise operations need matching or broadcastable shapes and cost one operation per element, so they are usually limited by memory speed, while a matmul reuses each input many times and is limited by arithmetic. Einstein notation writes both: torch.einsum("ik,kn->in", x, W) is the matmul, and "ij,ij->ij" is the element-wise product.
Step 6 of 6: Tensors in a Real Model
A batch of images through one dense layer
Layout
Batch2
Image side4px
Input numbers
96
Input memory (fp32)
384 B
W + b params
490
Every step so far shows up in one tiny classifier. The batch is a 4-axis tensor (2, 3, 4, 4), flatten is a reshape to (2, 48), @ W is a matmul that uses up the pixel axis, and + b broadcasts one bias row across the batch. Change the batch: only the first axis moves. W and b stay the same size, because the same weights serve every image. Switch the layout and the flattened numbers come out in a different order, so weights trained for one layout give nonsense on the other.
Change the batch, the image size, and the layout. PyTorch orders image batches as (N, C, H, W) and TensorFlow defaults to (N, H, W, C), so the same image flattens into numbers in a different order. A dense layer on a flattened 224×224 RGB image needs about 1.5 million weights for 10 outputs, which is one reason image models use convolutions instead. Transformers use (batch, sequence length, embedding size), and attention scores add a heads axis.
What is a tensor in deep learning?
In deep learning, a tensor is a multi-dimensional array of numbers with a fixed shape. A scalar has zero axes, a vector one, a matrix two, and higher-rank tensors have three or more. Tensor operations are the reshapes, element-wise arithmetic, broadcasts, and matrix multiplications that transform these arrays.
Every input, weight, activation, and gradient in PyTorch, TensorFlow, or JAX is a tensor. A batch of images is a 4D tensor, a batch of token embeddings is a 3D tensor, and attention weights are another 4D tensor. Most bugs in model code are shape bugs, so reading and predicting tensor shapes is one of the most practical skills in deep learning.
Key ideas
A tensor is an n-dimensional array, and its shape lists the size of each axis.
The number of elements is the product of the sizes in the shape.
Reshaping changes the shape without changing the elements or their order.
Broadcasting stretches size-1 axes so tensors of different shapes can combine element-wise.
Element-wise operations need matching or broadcastable shapes; matrix multiplication needs the inner sizes to match.
Image batches are 4D tensors, and their axis order differs between frameworks.
Common questions
What is the difference between a NumPy array and a PyTorch tensor?
Both are n-dimensional arrays with nearly the same indexing and broadcasting rules. A PyTorch tensor can also live on a GPU and can record operations for automatic differentiation. torch.from_numpy and tensor.numpy() convert between them, sharing memory when the tensor is on the CPU.
What does squeeze and unsqueeze do?
squeeze removes axes of size 1, so (1, 3, 1, 4) becomes (3, 4). unsqueeze inserts a size-1 axis at a chosen position, so a (3, 4) tensor becomes (1, 3, 4). They are mainly used to set up broadcasting or to add a batch axis to a single example.
What is the difference between torch.mm, torch.matmul, and the @ operator?
torch.mm only multiplies two 2D matrices. torch.matmul, which @ calls, also handles vectors and batches: for 3D inputs of shape (b, n, m) and (b, m, p) it performs b separate matrix products. It broadcasts the batch axes too, which is how attention multiplies queries and keys for every head at once.
How do you debug tensor shape errors?
Print .shape at each step and compare it with the shape you expect, writing the expected shapes as comments. Most errors come from a missing batch axis, swapped axes after a transpose, or unintended broadcasting. Libraries such as einops make shapes explicit in the code, for example rearrange(x, "b c h w -> b (h w) c").
Related explainers
Linear Algebra— Vectors, matrices, and the language of data