Documenting an undocumented chip: the TensorCore of TPU v5e

The aim of this post is to be an unofficial reference for the TensorCore of Google's TPU v5e. It presents the instruction set of this core, both what each instruction does and how many cycles it takes.

The data in this document has been collected experimentally by running probes on a TPU. We do not claim that this document is exhaustive or perfect, but it is a fair overview of our understanding of the core.

This document does not cover the SparseCore, later TPU versions, or topologies of multiple TPUs.

This work is not affiliated with Google.

Methodology

In our previous post we presented our work building a simulator for TPU v5. This simulator allowed us to deepen our theoretical understanding of TPUs. It let us confirm the communication primitives between a server and an attached TPU. It also allowed us to start deciphering the instruction set of TPUs for both versions 5 and 7. However, it still had its limits: precise semantics and latencies of the instruction set can only be recovered by running micro-kernels on a physical device.

The natural next step was to use the simulator to write a custom driver for TPUs. This driver is minimal: its only purpose is to connect to a TPU and run small binary probes on the TensorCore. These probes are assembled using an in-house TPU assembler. The driver exposes the same low-level performance counters as one would access when using the official driver.

Once the driver worked, we built a small harness for LLM agents to explore the TensorCore of v5e in depth. Given a question, the LLMs would write a sequence of probes: small programs, each meant to learn one microarchitectural detail of the TPU. The results of the probes were recorded and used to close the gap between our software simulator and a real TPU. We steered this semi-automatic exploration fairly heavily as LLMs tends to go toward complexity. This was especially valuable to prevent the simulator code from becoming a tangled mess of one-off and brittle fixes.

This work focused on the TPU v5e chip for cost reasons: we are unfunded and first wanted to check our methodology on a cheaper device.

The TensorCore at a glance

The chip

A v5e chip has one TensorCore and 16 GiB of HBM. The TensorCore can't read HBM directly: data reaches the core only by DMA, into its vector memory (VMEM), its scalar memory (SMEM) or its instruction memory (IMEM). The host loads a program with two DMA commands sent through the firmware: host memory to HBM, then HBM to IMEM. The host talks to the chip over PCIe. The core runs at 1.5 GHz. Every cycle count in this post is a core cycle.

The core

Some words we use throughout:

The core has a scalar side and a vector side.
Every cycle the core fetches one bundle from IMEM. A bundle is a group of instructions meant to start together. Its scalar instructions are the bundle's scalar part, the others its vector part.
A bundle has slots, one per instruction. Each slot takes only some kinds of instructions.
The vector side issues a vector part: it starts all of its instructions in the same cycle.
A vector register is 8 sublanes of 128 lanes, one 32-bit word per lane.
A guard is the predicate register that decides whether an instruction runs.
The slower units take work by a push and give their result back by a pop (see below).
A granule is 32 bytes, the unit DMA lengths are counted in.
A sync flag is a 32-bit counter used as an atomic. A DMA can credit a sync flag: it adds to the flag the number of granules it moved.
A stall is an ordinary wait: an instruction is held until what it needs is ready.

When a program breaks a rule, the core ends in one of these ways:

it faults: it stops at the bundle that broke the rule and reports a fault to the host;
it hangs: part of the core waits forever for something that never comes;
it stops: it goes idle in the middle of the program and no fault is reported.

An instruction is refused when the assembler or the chip rejects it.

The scalar side runs the control flow, computes addresses and does basic arithmetic. It has two scalar slots, 32 scalar registers of 32 bits, 14 predicate registers, SMEM and the sync flags.

The vector side does the computations, on vector registers of 8 sublanes × 128 lanes × 32 bits. It has four vector ALU slots, loads and stores between the registers and VMEM, and three kinds of slower units: 4 matrix units (MXU), 3 cross-lane units (XLU) and one unit for transcendentals (EUP).

Push, then pop. The slower units don't give their result back in the instruction that starts them. An instruction pushes the work to the unit, and the result goes into the unit's result queue. A later pop moves the oldest result from that queue into a vector register.

From fetch to issue. When a bundle is fetched, its scalar part runs at once and its vector part goes into the vector queue. The queue has 20 slots, and every fetched bundle takes one. A bundle gives its slot back 12 cycles after its vector part issues. If it has no vector part then it gives it back 4 cycles after its fetch. When no slot is free the fetch waits. The vector side issues the vector part of the oldest bundle in the queue as soon as all of its instructions can start.

One consequence of this design is that the scalar side runs ahead of the vector side by up to 20 bundles.

sfence makes the scalar side wait until every vector part in the queue has issued. vpop.v2s waits only for the value it pops.

The scalar side can't read vector registers. To get a vector value, the vector side pushes element 0 of a register with push_v2s, and the scalar side pops it with vpop.v2s (see Pops and the result FIFOs). For more data, a DMA can copy VMEM into SMEM.

Floating-point behavior

A denormal operand is read as a zero of its own sign.
A result that would be denormal is written as a zero of the result's sign.
Rounding is to nearest, ties to even.
An overflow gives ±inf.
NaNs produced by operations are always 0x7fc00000: a NaN operand's payload and sign are dropped, and 0 · inf, inf − inf and a signaling NaN all give 0x7fc00000. The exception is veup.push: it passes a word through the EUP unchanged, NaNs included.
max and min return NaN if either operand is NaN. max(+0, −0) is +0 and min(+0, −0) is −0, in either operand order.
The ceiling or floor of a denormal is a zero of its sign.
The float compares follow IEEE.

Bundles

A bundle is 512 bits and has 14 slots:

Slots	Holds
`s0`, `s1`	scalar instructions: arithmetic, compares, branches, SMEM loads and stores. A DMA takes both.
`valu0` to `valu3`	vector ALU instructions.
`vex0`, `vex1`	pushes to the matrix and cross-lane units
`vres0`, `vres1`	pops from the matrix units, the cross-lane units and the EUP
`vld0` to `vld2`	vector loads. `vld2` can hold one misc instruction instead.
`vst`	the one vector store

The slots also share six immediates and four scalar register ports. These ports are how instructions (both scalar and vector) refer to a scalar register. The scalar registers are read at fetch, before the vector part goes into the vector queue.

Most instructions take a fixed number of cycles, given in the tables. No padding is needed between dependent instructions: if one reads a register before its value is ready, the core waits. A program is fast when dependent instructions are far enough apart that it doesn't.

Predicates. Every slot has a guard. A slot is considered empty if it is guarded by p15, "never".

State

	How many	Each
scalar registers `s0`..`s31`	32	32 bits
predicates `p0`..`p13`	14, plus `p14` (true) and `p15` (false)	1 bit
vector registers `v0`..`v63`	64	4 KiB: 8 sublanes × 128 lanes
mask registers `vm0`..`vm15`	16	4 bits per 32-bit element, one per byte
sync flags	512	32 bits, a done bit, a public bit
SMEM	1 MiB	32-bit words
VMEM	128 MiB	rows of 512 bytes, the size of one sublane
IMEM	4 MiB	65,536 bundles of 64 bytes
HBM	16 GiB	reached only by DMA

Sync flags are the TPU version of mutexes and atomics. A bundle can wait for a sync flag to reach a given value. DMA instructions credit sync flags to tell the core they are done. The core has 512 flags and the host 128.

Reading the instruction tables

Each section starts with a table of its instructions. Two rows, one from the scalar side and one from the matrix unit:

Instruction	Slot	Cycles	What it does
`sadd.s32 sD, sX, sY\|imm`	s0, s1	1	Wrapping add.
`matmul.{f32r,bf16,bf8,if8bf16}[.lgmr.msr{a,b}] uU, vS [, vmsk=vmK]`	vex0, vex1	125 / 131	Multiply each matrix row the register holds by the GMR.

Instruction: the instruction syntax in our unofficial assembler.
- a|b means the operand can be either: sY|imm is a scalar register or an immediate. The description calls it by its first name.
- {a,b} means pick one: matmul.bf16 and matmul.f32r are both valid.
- […] is optional: matmul.f32r u0, v3 is valid, and so is matmul.f32r u0, v3, vmsk=vm1.
Slot: the bundle slots that can hold the instruction. s0, s1 means either scalar slot.
Cycles: how long until the result can be used, from the destination register or by a pop.
What it does: the semantics with known corner cases.

Operand names:

sD, sX, sY: scalar registers;
vD, vS, vT: vector registers;
vmK: mask registers;
pN: predicates;
uU: a cross-lane or matrix unit;
imm: an immediate;
sY.bcast: scalar sY copied into every element.

Scalar side

Instructions

Instruction	Slot	Cycles	What it does
Integer and bitwise
`sadd.s32 sD, sX, sY\|imm`	s0, s1	1	Wrapping add.
`ssub.s32 sD, sX, sY\|imm`	s0, s1	1	Wrapping subtract; computes `sY` − `sX`.
`sadd.ovf sD, sX, sY\|imm`	s0, s1	1	Add as `sadd.s32`. On a signed overflow the core faults.
`ssub.ovf sD, sX, sY\|imm`	s0, s1	1	Subtract (`sY` − `sX`) as `ssub.s32`, with the same overflow fault as `sadd.ovf`.
`sand.u32 sD, sX, sY\|imm`	s0, s1	1	Bitwise and.
`sor.u32 sD, sX, sY\|imm`	s0, s1	1	Bitwise or.
`sxor.u32 sD, sX, sY\|imm`	s0, s1	1	Bitwise xor.
`sshll sD, sX, sY\|imm`	s0, s1	1	Shift left; an amount of 32 or more gives 0.
`sshrl sD, sX, sY\|imm`	s0, s1	1	Logical shift right; an amount of 32 or more gives 0.
`sashr sD, sX, sY\|imm`	s0, s1	1	Arithmetic shift right; an amount of 32 or more shifts by 31, leaving only the sign.
`smax.u32 sD, sX, sY\|imm`	s0, s1	1	Unsigned max.
`smin.u32 sD, sX, sY\|imm`	s0, s1	1	Unsigned min.
`smov sD, sY\|imm`	s0, s1	1	Copy a register or an immediate.
`sclz sD, sY\|imm`	s0, s1	1	Leading zeros.
`smul.u32 sD, sX, sY\|imm`	s0	2	Low half of the integer product.
`smulh.u32 sD, sX, sY\|imm`	s0	2	High half of the integer product.
Float
`smax.f32 sD, sX, sY\|imm`	s0, s1	1	f32 max; a NaN operand gives NaN.
`smin.f32 sD, sX, sY\|imm`	s0, s1	1	f32 min; a NaN operand gives NaN.
`sceil.f32 sD, sY\|imm`	s0, s1	1	Float ceiling.
`sfloor.f32 sD, sY\|imm`	s0, s1	1	Float floor.
`smul.f32 sD, sX, sY\|imm`	s0	2	f32 product.
`sadd.f32 sD, sX, sY\|imm`	s1	2	f32 add.
`ssub.f32 sD, sX, sY\|imm`	s1	2	f32 subtract.
`scvt.f32.s32 sD, sY\|imm`	s0, s1	2	Convert s32 to f32, rounding to nearest even.
`scvt.s32.f32 sD, sY\|imm`	s0, s1	2	Convert f32 to s32, rounding to nearest even; out of range saturates, NaN gives 0.
Compares and predicates
`scmp.{eq,ne} pN, sX, sY\|imm`	s0, s1	2	Compare `sX` = `sY` (`eq`) or `sX` ≠ `sY` (`ne`), into a predicate register. These two take no type suffix.
`scmp.{gt,ge,lt,le}.{s32,u32} pN, sX, sY\|imm`	s0, s1	2	Compare `sX` >, ≥, < or ≤ `sY`, signed (`s32`) or unsigned (`u32`), into a predicate register.
`scmp.{eq,ne,gt,ge,lt,le}.f32 pN, sX, sY\|imm`	s0, s1	2	The same six compares on f32, into a predicate register.
`scarry.u32 pN, sX, sY\|imm`	s0, s1	2	Carry out of an add, into a predicate.
`spred.or pN, pX[!], pY[!]`	s0, s1	2	OR of two predicates, each one optionally negated.
`sinfnan.f32 pN, sX`	s0, s1	2	Test for inf or NaN, into a predicate.
Divide and result FIFOs
`sdiv[.push_quotient\|.push_remainder] sX, sY\|imm`	s0	9	Unsigned divide; pushes quotient and remainder to the divide FIFO. On the chip the two push forms also push both words.
`spop.drf sD`	s0, s1	1	Pop the divide FIFO.
`vpop.sfrf sD`	s0, s1	1	Pop the sync-flag FIFO.
`vpop.v2s sD`	s0, s1	1	Pop the vector-to-scalar FIFO.
SMEM
`sld.smem sD, smem[sX\|imm\|sX + imm]`	s1	6	Load one SMEM word, at the address in sX, at an immediate address, or at sX plus an immediate.
`sst.smem sX, smem[sY\|imm]`	s1	1¹	Store one SMEM word, at the address in sY or at an immediate address.
Control flow
`sbranch.{rel,abs} label\|imm`	s0, s1	—²	Relative or absolute branch; conditional only through predication.
`sbranch.ind sX`	s0, s1	—²	Branch to the address in a register; conditional only through predication.
`scall.{rel,abs} label\|imm, link=sD`	s0, s1	—²	Relative or absolute call; conditional only through predication.
`shalt`	s0, s1	—	Halt once every queued vector part has issued.
`shalt.yield`	s0, s1	—	Halt as `shalt`, reported to the host as yielded.
`shalt.yield.cond`	s0, s1	—	Conditional yield; on a one-core v5e it never yields and does nothing.
`sdelay imm[, dest=d]`	s0, s1	32 × imm + d	Stall the scalar side for a given number of cycles.
`sfence`	s0, s1	12³	Wait until every bundle in the vector queue has been issued.
Counters and hidden registers
`srdreg sD, lcc.{lo,hi}`	s0, s1	1	Low or high half of the 64-bit core cycle counter; it counts stall and wait cycles too.
`srdreg sD, gtc.{lo,hi}`	s0, s1	1	Low or high half of the 64-bit global time counter, an 800 MHz clock.
`srdreg sD, {yield,tag,tracemark}`	s0, s1	1	The yield, tag or tracemark register.
`set_tag sY\|imm`	s0, s1	1	Write the tag register.
Core to core
`spush.sccf sY\|imm`	s0, s1	—	Push onto the scalar core-to-core FIFO; faults on a one-core v5e.
`spop.sccf sD`	s0, s1	—	Pop the scalar core-to-core FIFO.

¹ An sld.smem issued in the next bundle reads the newly stored value.

² See Branches, calls and delay slots below.

³ On top of the time until every queued vector part has issued.

Details

Predication

Every slot carries a guard, written (pN): one of the predicate registers p0 to p13, p14="always" or p15="never". The two scalar slots can also invert it, written (pN!). A slot whose guard is false has no effect. There is no separate conditional branch as they are represented using a guarded unconditional branch (in (p1) sbranch.abs loop, loop is only taken if p1 is true).

Predicates are written only by the compares, sinfnan.f32, scarry.u32 and spred.or, and are ready 2 cycles later.

Since each operand can be negated and p15 always reads false, spred.or can be used to implement other operations.

Wanted	Instructions	Why
p3 = p1	`spred.or p3, p1, p15`	p1 \| false
p3 = !p1	`spred.or p3, p1!, p15`	!p1 \| false
p3 = !(p1 & p2)	`spred.or p3, p1!, p2!`	!p1 \| !p2, by De Morgan
p4 = p1 & p2	`spred.or p3, p1!, p2!` `spred.or p4, p3!, p15`	the line above, then its negation

When a predicate is used as a guard (as in (p4) sadd.s32 s5, s5, 1) the predicate needs to be available on issue. A guard in the bundle right after the one that writes its predicate stalls for one cycle.

scmp.lt.s32 p4, s1, s2
(p4) sadd.s32 s5, s5, 1 // <- this will stall 1 cycle

This is not the case for spred.or.

scmp.lt.s32 p1, s1, s2
spred.or    p3, p1!, p2!      // no stall

Branches, calls and delay slots

A branch changes the program counter 4 bundles later. The 4 bundles right after it always run. A taken sbranch.ind costs one more cycle than the other branches. A taken branch inside the delay slots of another taken branch faults the core.

A call saves its return address in the register named by link: scall.abs f, link=s10 saves it in s10. There is no return instruction; a function returns with sbranch.ind s10.

The return comes back one bundle early: to the call's last delay slot, not to the bundle after it. That bundle has already run once, so it runs a second time. In the trace below, the call is at 48, its delay slots are 49 to 52, and it saves 52. After the return, 52 runs again, then 53.

Pops and the result FIFOs

The scalar side can't read the other units' results directly. Three FIFOs bring values to it, each with its own pop:

the divide FIFO: each sdiv pushes its quotient, then its remainder; spop.drf pops them. It holds 16 words, enough space for 8 divisions.
the sync-flag FIFO: read_sync, read_sync_done and read_sync_public_access push a flag's value or one of its bits. vpop.sfrf pops them. It holds 16 values.
the vector-to-scalar FIFO: push_v2s pushes element 0 of a vector register (sublane 0, lane 0). vpop.v2s pops it. It holds 128 words. It is the main way for the scalar side to get a value from the vector side.

A pop takes the oldest value and costs 1 cycle once the value is there. A pop whose value is still on its way (the associated push was issued before the pop but the instruction is not yet retired) waits until the value arrives. A pop of an empty FIFO with nothing on its way faults the core.

sdiv pushes its results 9 cycles after it issues, read_sync 13 cycles after, and push_v2s 61 cycles after.

Divide

Only slot 0 has a divider. The divide is unsigned. Dividing by zero gives a quotient of all ones and the dividend as remainder. sdiv.push_quotient and sdiv.push_remainder push both the quotient and remainder as sdiv does.

SMEM

SMEM is the scalar side's own memory: 256 Ki 32-bit words (1 MiB), addressed by word. Only slot 1 can issue a load or store to SMEM. There's no concept of banks or ports with SMEM. A loaded value is ready 6 cycles after the load. A load in the bundle after a store reads the updated value.

Arithmetic specific to the scalar side

sinfnan.f32 is true for an infinity or a NaN, of either sign, and false otherwise, denormals included.

Scalar register used as a destination twice in one bundle

Two scalar slots writing the same scalar register in one bundle fault the core. Two writes to the same predicate in one bundle leave the AND of the two values.

Vector ALU

Instructions

Instruction	Slot	Cycles	What it does
Integer and bitwise
`vadd.s32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Add `vS` and `vT`, element by element; overflow wraps.
`vsub.s32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Subtract, element by element: computes `vT` − `vS`; overflow wraps.
`vmul.u32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Multiply `vS` and `vT`, element by element, keeping the low 32 bits of each product.
`vand.u32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Bitwise and of `vS` and `vT`, element by element.
`vor.u32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Bitwise or of `vS` and `vT`, element by element.
`vxor.u32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Bitwise xor of `vS` and `vT`, element by element.
`vshll.u32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Shift each element of `vS` left by `vT` bits; 32 or more gives 0.
`vshrl.u32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Shift each element of `vS` right by `vT` bits, filling with zeros; 32 or more gives 0.
`vashr.s32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Shift each element of `vS` right by `vT` bits, filling with its sign bit; 32 or more leaves only the sign.
`vpopcnt vD, vS`	valu0..3	2	Count the bits set in each element.
`vclz vD, vS`	valu0..3	2	Count the leading zero bits of each element; 32 for 0.
Float
`vadd.f32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Add `vS` and `vT` as f32, element by element.
`vsub.f32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Subtract as f32, element by element: computes `vT` − `vS`.
`vmul.f32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Multiply `vS` and `vT` as f32, element by element.
`vmax.f32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	The larger of `vS` and `vT`, as f32, element by element; a NaN operand gives NaN, and +0 is above −0.
`vmin.f32 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	The smaller of `vS` and `vT`, as f32, element by element; a NaN operand gives NaN, and −0 is below +0.
`vrelux vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Clamp to [0, `vT`]: max(min(`vS`, `vT`), 0); a zero result is +0.
`vclamp vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Clamp to [−`vT`, `vT`]; a negative `vT` gives +0, and a zero the clamp produces is +0.
`vceil.f32 vD, vS`	valu0..3	2	Round each f32 element up to a whole number.
`vfloor.f32 vD, vS`	valu0..3	2	Round each f32 element down to a whole number.
Converts
`vcvt.f32.s32 vD, vS`	valu0..3	2	Convert s32 to f32, rounding to nearest even.
`vcvt.s32.f32 vD, vS`	valu0..3	2	Convert f32 to s32, rounding to nearest even; out of range saturates, NaN gives 0.
`vcvt.bf16.f32 vD, vS`	valu0..3	2	Convert f32 to bf16, rounding to nearest even, into the low half; the high half is all ones.
`vcvt.hf16.f32 vD, vS`	valu0..3	2	Convert f32 to f16, rounding to nearest even, into the low half; overflows to inf.
`vcvt.bf8.f32 vD, vS`	valu0..3	2	Convert f32 to bf8 (e5m2), rounding to nearest even, into the low byte; overflows to inf from 61440.
`vcvt.if8.f32 vD, vS`	valu0..3	2	Convert f32 to if8 (e4m3, bias 11), into the low byte; saturates to 0x7f, as if8 has no inf.
`vcvt.{hf16,bf16,bf8,if8}.f32.sr vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Convert f32 to f16, bf16, bf8 or if8, rounding stochastically on `vT`.
Packs and unpacks
`vpack.bf16 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Convert two registers to bf16 as `vcvt.bf16.f32` does, then pack them compressed.
`vpacki.bf16 vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Convert two registers to bf16 as `vcvt.bf16.f32` does, then pack them interleaved.
`vpack.{16,8,4,2,1} vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Pack the low N bits of every 2N-bit field of two registers (N = 16, 8, 4, 2 or 1), compressed.
`vpacki.{16,8,4} vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	The same, interleaved. There is no `vpacki.2` or `vpacki.1`.
`vunpack.{bf16,s16}.iN vD, vS`	valu0..3	2	Widen the low (N = 0) or high (N = 1) half of every word in place, bf16 to f32 or s16 to s32.
`vunpack.{bf16,hf16,s16}.slN vD, vS`	valu0..3	2	Widen 16-bit packed rows 8N to 8N+7 (N = 0, 1), bf16 or f16 to f32, s16 to s32.
`vunpack.{bf8,if8,s8}.slN vD, vS`	valu0..3	2	Widen 8-bit packed rows 8N to 8N+7 (N = 0..3), bf8 or if8 to f32, s8 to s32.
`vunpack.s4.slN.s8 vD, vS`	valu0..3	2	Widen 4-bit packed rows 32N to 32N+31 (N = 0, 1) to 8-bit packed rows, sign-extended.
`vunpack.s2.slN vD, vS`	valu0..3	2	Widen 2-bit packed rows 64N to 64N+63 (N = 0, 1) to 4-bit packed rows, sign-extended.
`vunpack.s1.slN vD, vS`	valu0..3	2	Widen 1-bit packed rows 128N to 128N+127 (N = 0, 1) to 2-bit packed rows, sign-extended.
Sublane moves
`vsrot vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Rotate each lane's sublanes: element (s, l) takes `vS`'s element ((s + `vT`) mod 8, l). `vT` is read per element, low 3 bits only.
`vsperm vD, vS, vT\|sY.bcast\|imm`	valu0..3	2	Gather within each lane: element (s, l) takes `vS`'s element (`vT` mod 8, l). `vT` is read per element, low 3 bits only.
`vlaneseq vD`	valu0..3	2	Each element's flat index, `128 × sublane + lane`.
Moves and selects
`vmov vD, [vS,] vT\|sY.bcast\|imm`	valu0..3	1	Copy a register, an immediate or a broadcast scalar.
`vsel.vmskK vD, vS, vT\|sY.bcast\|imm`	valu0..3	1	Per byte: `vT` where the mask bit is set, `vS` where it is clear.
`vsel.nvmskK vD, vS, vT\|sY.bcast\|imm`	valu0..3	1	Per byte: `vS` where the mask bit is set, `vT` where it is clear.
Compares and mask builders
`vcmp.{eq,ne,gt,ge,lt,le}.s32 vmK, vS, vT\|sY.bcast\|imm`	valu0..3	1	Compare signed `vS` =, ≠, >, ≥, < or ≤ `vT`, element by element, into a mask register.
`vcmp.{eq,ne,gt,ge,lt,le}.f32 vmK, vS, vT\|sY.bcast\|imm`	valu0..3	1	The same six compares on f32, into a mask register.
`vcmp.{tlt,tle}.f32 vmK, vS, vT\|sY.bcast\|imm`	valu0..3	1	Compare total-order f32 `vS` < `vT` (`tlt`) or ≤ `vT` (`tle`), into a mask register.
`vcmp.carry.u32 vmK, vS, vT\|sY.bcast\|imm`	valu0..3	1	Unsigned carry out of `vS` + `vT`, into a mask register.
`vclassify vmK, vS, vT\|sY.bcast\|imm`	valu0..3	1	Test each element's float class against the class bits in `vT`, into a mask register.
`vinfnan.f32 vmK, vS`	valu0..3	1	Test each element for ±inf or NaN, into a mask register.
`vbytenez vmK, vT\|sY.bcast\|imm`	valu0..3	1	Set mask bit k where byte k of `vT` is nonzero.

Details

Packs and unpacks

A vector register holds 8 sublanes of 128 32-bit words. A word can also hold two 16-bit values, four 8-bit values, and so on. We call packed row a row of 128 values within a sublane. With 16-bit values a sublane holds 2 packed rows, with 8-bit values 4, and so on.

A pack instruction merges two registers vS and vT into one by halving the width of every value. The first kind of pack is a compressed pack that stacks the packed rows of the two inputs. The second kind of pack is an interleaved pack that, as its name indicates, interleaves them.

An unpack goes the other way: it widens the values of one register, vS. A slN unpack takes block N of the packed rows, as many as the result holds, and widens each, in order. In the figures, vS is colored by block, and each result cell keeps the number and the color of the packed row it came from.

The illustrations above show the behavior as a property of the bit width and not the type. In practice the type matters for field widening (unpacking):

signed integers are sign-extended;
bf16 becomes the top half of the f32;
f16 is converted to f32 as an IEEE half;
bf8 is read as e5m2, and if8 as e4m3 with bias 11;
the narrow integers are sign-extended into their wider field: vunpack.s1 turns a 1 into 0b11.

Packing looks at the type only on vpack.bf16 and vpacki.bf16: each field is read as an f32 and the conversion to bf16 follows the same logic as vcvt.bf16.f32. Other pack instructions operate on bits.

Sublane moves

vsrot rotates within each lane. Writing s for a sublane index and l for a lane index, vD[s][l] = vS[(s + (vT[s][l] & 7)) & 7][l].

vsperm permutes within each lane. Writing s for a sublane index and l for a lane index, vD[s][l] = vS[vT[s][l] & 7][l].

vlaneseq takes no input. It generates a vector holding every number from 0 to 1023. Writing s for a sublane index and l for a lane index, vD[s][l] = 128 * s + l.

Converts

Each convert keeps what fits the target format and drops the rest. For the float formats, the sign and exponent carry over (re-biased) and the dropped part is the low end of the f32's 23-bit fraction:

vcvt.bf16.f32: bf16 is the top half of an f32. It keeps the top 7 fraction bits and drops the low 16.
vcvt.hf16.f32: f16 is IEEE binary16, with a 5-bit exponent. It keeps the top 10 fraction bits and drops the low 13.
vcvt.bf8.f32: bf8 is e5m2, the top byte of an f16. It keeps the top 2 fraction bits and drops the low 21.
vcvt.if8.f32: if8 is e4m3 with exponent bias 11. It keeps the top 3 fraction bits and drops the low 20.
vcvt.s32.f32: keeps the integer part and drops everything below the binary point.

The dropped bits are used to round the number. Line the dropped bits up at the top of a 23-bit field, padding with zeros on the right; call that frac. For bf16, frac is the low 16 bits of vS shifted left by 7.

The .sr forms (vcvt.bf16.f32.sr vD, vS, vT, and the same for f16, bf8 and if8) take the threshold from vT & 0x7fffff. When frac is above the threshold the convert rounds up; at or below it, it rounds down.
The plain forms behave the same with a threshold of half, 0x400000, except that a value exactly at half rounds to nearest even.

vclassify

vclassify is true where vS's class is one of the bits set in vT. Quiet and signaling NaNs are alike, but the sign of a NaN counts:

Bit	Class
`0x001`	negative NaN
`0x002`	−inf
`0x004`	−normal
`0x008`	−denormal
`0x010`	−0
`0x020`	+0
`0x040`	+denormal
`0x080`	+normal
`0x100`	+inf
`0x200`	positive NaN

So 0x102 means "infinite, either sign".

Masks

A vector register has 8 sublanes, each of 128 32-bit elements. Instructions often need to work on a subset of a vector register. A mask says which part. A mask register has the same shape as a vector register: 8 sublanes of 128 4-bit entries. Each bit stands for one byte of the associated 32-bit element. This allows masks to work with packed vectors.

The two mask builders below, vmkmask and vmksublane, run in the vector ALU slots. The mask ALU, which combines masks, runs in the misc slot.

Instructions

Instruction	Slot	Cycles	What it does
`vmkmask vmK, imm`	valu0..3	1	Build a sublane × lane rectangle from an immediate.
`vmksublane vmK, imm`	valu0..3	1	Build a band of sublanes, starting and ending at a byte, from an immediate.
`vmsk.mov vmD, vmK`	misc	1	Copy a mask register.
`vmsk.not vmD, vmK`	misc	1	Invert every bit of a mask register.
`vmsk.and vmD, vmK, vmL`	misc	1	Bitwise and of two mask registers.
`vmsk.or vmD, vmK, vmL`	misc	1	Bitwise or of two mask registers.
`vmsk.xor vmD, vmK, vmL`	misc	1	Bitwise xor of two mask registers.
`vmsk.pack vmD, vmK, vmL`	misc	1	Merge two masks into one mask for 16-bit data packed by `vpack`.

Details

vmkmask

vmkmask builds a rectangular mask. Its immediate gives the rectangle's first and last sublane and its first and last lane:

vmksublane

vmksublane selects a band of sublanes across all 128 lanes. Unlike vmkmask, the band can start and end in the middle of a sublane, at a byte.

To say where, number the bytes of one lane from top to bottom: sublane 0 holds bytes 0 to 3, sublane 1 bytes 4 to 7, and so on down to byte 31. The immediate gives the first and the last byte of the band, both included. Every lane gets the same band.

Vector loads and stores (VMEM)

Each vector register is made of 8 sublanes of 128 words. VMEM is made of rows of 128 words, so one vector register fills 8 rows of VMEM, one row per sublane.

Every load and store takes an optional sublane mask sm=imm|sM. The plain and .base forms also take an optional stride=imm|sS, and every store an optional element mask vmsk=vmK. The table leaves these out.

Instructions

Instruction	Slot	Cycles	What it does
Loads
`vld vD, vmem[imm]`	vld0..2	1	Load 8 rows into vD, sublane `k` from row `imm + stride × k`. Masked-off sublanes read 0.
`vld.base vD, vmem[sX + imm]`	vld0..2	1	The same, with a scalar register added to the address.
`vld.shuf vD, vmem[imm], shuffle=imm\|sX`	vld0..2	1	Load each sublane from its own row, one of the 16 rows starting at `imm`.
`vld.shuf.base vD, vmem[sX + imm], shuffle=imm\|sY`	vld0..2	1	The same, with a scalar register added to the address.
`vld.indexed.N vD`	vld0..2	1	Gather through IARN (N = 0 or 1): each element from row `IARN + sublane`, same lane.
Stores
`vst vS, vmem[imm]`	vst	1⁴	Store vS, sublane `k` to row `imm + stride × k`.
`vst.base vS, vmem[sX + imm]`	vst	1⁴	The same, with a scalar register added to the address.
`vst.shuf vS, vmem[imm], shuffle=imm\|sX`	vst	1⁴	Store each sublane to its own row. Two selected sublanes naming the same row hang the core.
`vst.shuf.base vS, vmem[sX + imm], shuffle=imm\|sY`	vst	1⁴	The same, with a scalar register added to the address.
`vst.indexed.N vS, vmem[imm]`	vst	1⁵	Scatter through IARN (N = 0 or 1): each element to row `imm + IARN + sublane`, same lane.
Index registers (N = 0 or 1)
`set_raw_iar_N vS`	vst	1⁶	Copy vS into IARN as it is.
`set_lane_iar_N vS`	vst	1⁶	Copy sublane 0 of vS into all 8 sublanes of IARN.
`set_sublane_iar_N vS`	vst	1⁶	Copy vS into IARN, each word multiplied by 32.
Through the store slot
`push_v2s vS, vmem[imm]`	vst	61⁷	Send element 0 of vS (sublane 0, lane 0) to the scalar side, through the vector-to-scalar FIFO. The `vmem` operand is ignored.
`set_prng vS`	vst	1⁸	Seed the random-number generator from vS.

⁴ A vld of a row it wrote waits until 10 cycles after it.

⁵ Every later vld, whatever rows it reads, waits until 8 cycles after it.

⁶ A vld.indexed behind it waits until 8 cycles after it; a plain vld doesn't wait.

⁷ Until the value can be popped by vpop.v2s.

⁸ The PRNG stays busy for 3 cycles.

Details

Addressing modes

Contiguous. By default, rows are contiguous: sublane i = VMEM[base + i].

Strided. Given a stride, sublane i = VMEM[base + i * stride].

Shuffled. The shuffle operand packs eight 4-bit offsets, one per sublane. sublane i = VMEM[base + shuffle[i]]. Offsets go from 0 to 15, so a shuffled access reaches the 16 rows from base.

Indexed. This does an indirect load or store according to an IAR. In pseudo notation this maps to (sublane i)[e] = VMEM[IAR[i][e] + i][e]. An IAR is similar to a vector register but is only used for indexed loads and stores. There are only two IARs. These instructions implement gathers and scatters. Only the low 18 bits of IAR[i][e] + i count: VMEM has 2^18 rows, and a larger row number wraps around.

Masks

Load and store instructions take a sublane mask sm. This 8-bit mask switches off some sublanes: a load returns 0 in a masked-off sublane, and a masked-off sublane takes no port. Stores can also take an element mask vmsk that writes only a subset of the vector register to VMEM. A 32-bit store looks at one mask bit per word, the bit of the word's lowest byte, and writes the whole word or none of it.

Ports: how many loads and stores fit in a cycle

VMEM has 32 ports. Row r maps to port r mod 32. Each port can read or write one row per cycle. A load or store touches up to 8 rows at a time. Reading or writing two different rows on the same port in one instruction makes the core hang, so a program must avoid it. A load may name the same row in several sublanes (load coalescing); a store may not.

A bundle might carry up to three loads. A port with a pending load makes later loads on it take longer. A bundle of loads costs as many cycles as the busiest port has loads (1, 2 or 3), whatever kind of load they are: plain, strided or shuffled.

Write queues

Each port has a write queue. A store's rows reach their ports' write queues 6 cycles after the store issues. A queue holds 7 rows. A port writes its oldest queued row in every cycle no load reads that port: reads take priority over writes. When a store's row arrives at a full queue, it takes the cycle from the loads instead: the head is written and the new row joins. A store is also held at issue while any of its ports has 14 rows not yet written, counting the queue, the 6-cycle pipe, and the row written the cycle before.

Cross-lane units (XLU)

The XLU's job is to move data across lanes: it transposes matrices held in one or more vectors, rotates, broadcasts and permutes lanes, and reduces each sublane to one value. There are three XLUs. Each operation can pick one through uU.

Instructions

The cycle counts below are for unit 0.⁹

Instruction	Family	Slot	Cycles	What it does
`xlane.add uU, vS`	reduction	vex0, vex1	115	Compute the sum of the 128 lanes of each sublane, then broadcast it across that sublane.
`xlane.max uU, vS`	reduction	vex0, vex1	115	Compute the maximum of the 128 lanes of each sublane, then broadcast it across that sublane.
`xlane.min uU, vS`	reduction	vex0, vex1	115	Compute the minimum of the 128 lanes of each sublane, then broadcast it across that sublane.
`xlane.maxidx uU, vS`	reduction	vex0, vex1	115	Index of the max lane of each sublane; a tie goes to the highest lane.
`xlane.minidx uU, vS`	reduction	vex0, vex1	115	Index of the min lane of each sublane; a tie goes to the highest lane.
`lanerot imm\|sX, uU, vS`	mover	vex0, vex1	114¹⁰	Rotate the 128 lanes.
`lanebcast imm\|sX, uU, vS`	mover	vex0, vex1	114¹⁰	Broadcast one lane to all lanes.
`permute uU, vS`	mover	vex0, vex1	114¹⁰	Gather lanes within each sublane, using the pattern latched by `setpcr`.
`setpcr.{lanes,sublanes,bytes} uU, vS`	pattern latch	vex0, vex1	—¹¹	Latch one lane pattern used by every sublane (`.lanes`), a lane pattern per sublane (`.sublanes`), or a source lane per byte (`.bytes`).
`xpose.start.W uU, vS`	transpose	vex0, vex1	163^10,12	Start a new transpose operation.
`xpose.cont uU, vS`	transpose	vex0, vex1	163^10,12	Push a new vector into the current matrix being transposed.
`xpose.end uU, vS`	transpose	vex0, vex1	163^10,12	End the current transpose operation.
`xpose.W uU, vS`	transpose	vex0, vex1	163^10,12	Start and end a transpose in one push.
`xpose.seg.W uU, vS`	transpose	vex0, vex1	163^10,12	Segmented transpose in one push: every `W` lanes of the vector are their own matrix.
`xpose.seg.start.W uU, vS`	transpose	vex0, vex1	163^10,12	Start a new segmented transpose.
`xpose.binary.vy uU, vT`	transpose	vex0	171	Transpose a second 16-bit matrix alongside the 16-bit push in vex1 of the same bundle, interleaving the results.
`vpop.xpose vD, uU`	pop	vres0, vres1	1¹³	Pop the oldest result of a cross-lane unit.

⁹ Unit 1 is the same. On unit 2, reductions and movers are 24 cycles sooner (91, 90, 98 packed) and transposes 11 sooner (152; 160 for 16-bit).

¹⁰ Number for the 32-bit variant. The 16-bit variants (lanerot.packed, lanebcast.packed, permute.packed; xpose.start.p16.W, xpose.p16.W, xpose.seg.p16.W, xpose.seg.start.p16.W) are 8 cycles slower.

¹¹ No result. The next mover on the same unit (permute, lanerot, lanebcast) can start the cycle after a setpcr.lanes, and 8 cycles after a .sublanes or .bytes one.

¹² For a stream: the first result, counted from the first push, with the pushes back to back. Each later result is ready 8 cycles after the one before (16 for 16-bit).

¹³ A pop holds the unit's output port for 8 cycles (one sublane per cycle), so one unit gives at most one pop every 8 cycles.

Details

Matrix and "stream"

The unit of computation of the XLU is a vector: 8 matrix rows of 128 32-bit values, one per sublane, or 16 matrix rows of 128 16-bit values, two packed rows per sublane. A matrix is represented by a stream of vectors. To transpose a 128 × 128 matrix of 32-bit entries, we start a transpose with xpose.start.128, push 14 xpose.cont and finish with an xpose.end: 16 pushes of 8 rows. The transposed matrix gets written vector by vector to the xpose queue, one vector every 8 cycles: 16 pops of 8 rows. The 16-bit variant is similar but output vectors come once every 16 cycles: a 128 × 128 matrix of 16-bit entries is 8 pushes and 8 pops.

Depth of the `xpose` queue

The queue backing vpop.xpose holds 49 results, on each of the three units. A transpose stream reserves room for all its result rows when it starts. The width of the transposed matrices is the instruction's W. A stream waits until there's enough room in the queue for its results, which in practical terms makes the core hang.

xpose.binary.vy

xpose.binary.vy goes in vex0 of the same bundle as a 16-bit xpose instruction in vex1, on the same unit: an xpose.start.p16.W, xpose.p16.W, xpose.cont or xpose.end of a 16-bit stream. If the stream's first push carries one, every later push must too; if it doesn't, none may.

By default a 16-bit transpose can only emit an output vector register every 16 cycles. If xpose.binary.vy is used then two 16-bit matrices are interleaved and we get one output vector register every 8 cycles.

Segmented transpose

In the table above W must be a multiple of 8 (16 for a 16-bit transpose). xpose.start.W treats the first W lanes of each vector as one matrix of width W, and ignores the other lanes. With xpose.seg.start.W, each vector holds 128 / W independent matrices, all transposed at once.

Permutation patterns and `setpcr`

Each unit has its own setpcr latch. A second setpcr replaces the first entirely, pattern and form, and a permute doesn't use it up. A unit that was never latched gathers with an all-zero pattern: lane 0 of each sublane, broadcast. setpcr and permute on the same unit in one bundle hang the core: the vpop.xpose behind them never gets a result.

Instruction family and performance

The family matters for timing. On a given unit, two ops of the same family can follow each other every 8 cycles (16 for packed movers). Two ops of different families cost more, depending on the unit and the families. Transposes are the slowest to follow, as subsequent instructions have to wait for the transposer to walk the stream: a reduction or a mover waits 82 to 105 cycles more past the walk, depending on the unit.

The table below gives, for each unit, how many cycles after the first op the next op on the same unit can start, when the two are back to back. Each table row is the first op, each column the next one. walk is max(W, 8 × pushes) for the transpose ahead (16 × pushes for a packed one). Where the units differ, a cell reads unit 0 / unit 1 / unit 2.

first ↓ / next →	reduction	mover	packed mover	pattern latch	transpose
reduction	8	20 / 25 / 24	20 / 25 / 24	20 / 25 / 24	41 / 58 / 41
mover	42 / 42 / 41	8	8	8	34 / 53 / 38
packed mover	50 / 50 / 49	16	16	16	42 / 61 / 46
pattern latch	42 / 42 / 41	1 (`.lanes`), 8	1 (`.lanes`), 8	—	34 / 53 / 38
transpose	walk + 105 / 88 / 105	walk + 102 / 82 / 97	walk + 102 / 82 / 97	walk + 102 / 82 / 97	walk

Summation order of `xlane.add`

Each group of 8 adjacent lanes gets added pairwise in a reduction tree. The resulting 16 partial sums then get added one after the other.

Matrix units (MXU)

Instructions

Instruction	Slot	Cycles	What it does
`matpush.{f32r,bf16,bf8,if8bf16,u8,s8,u4,s4}.msr{a,b}[.t] uU, vS [, vmsk=vmK]`	vex0, vex1	2 for f32r, else 4¹⁴	Push one register of that type into staging register msra or msrb. With `.t`, the matrix rows go in as columns. `f32r` is f32 data, which the unit rounds to bf16 on the way in.
`lmatreg.gmr.msr{a,b} uU`	vex0, vex1	1	Copy msra or msrb into the GMR. Empty msra or msrb.
`lmatreg.lmr.msr{a,b} uU`	vex0, vex1	33¹⁵	Copy msra or msrb into the LMR. Empty msra or msrb.
`matmul.{f32r,bf16,bf8,if8bf16}[.lgmr.msr{a,b}] uU, vS [, vmsk=vmK]`	vex0, vex1	125¹⁶ / 131¹⁷	Multiply each matrix row the register holds by the GMR. `.lgmr.msrX` first does `lmatreg.gmr.msrX`.
`matmul.{u8,s8,u4,s4}[.lgmr.msr{a,b}] uU, vS [, vmsk=vmK]`	vex0, vex1	115¹⁶ / 121¹⁷	The same for the integer types.
`matmul.lmr[.lgmr.msr{a,b}] uU, width=N`	vex0, vex1	125¹⁶ / 131^17,18	Multiply the next 8 rows of the LMR by the GMR. After N rows (8, 16, 32, 64 or 128) it starts again from the first.
`vpop.mxu vD[, uU]`	vres0, vres1	1	Pop the oldest result register of a matrix unit.
`c2c.move uU, vS`	vex0, vex1	—	Core-to-core move of a register; unreachable on a one-core v5e.

¹⁴ Until the rows are in the staging register. A .t push takes twice as long.

¹⁵ Until the lmr register is fully filled. lmatreg.lmr copies the 128 rows of the staging register into the LMR, 4 rows a cycle.

¹⁶ Until the first result is pushed to the mxu queue. Latencies for unit 0 or 1.

¹⁷ Until the first result is pushed to the mxu queue. Latencies for unit 2 or 3.

¹⁸ With a float LMR. With an integer one, 115 / 121, like the integer multiplies.

Details

How the matrix units work

An MXU has one job: compute A · W, where A and W are two matrices of size up to 128 × 128 (256 deep for the 4-bit types). Our reading is that it has a 128 × 128 grid of multipliers that holds the values of W. A is streamed through the grid row by row, and the rows of the result are pushed onto the unit's mxu result queue.

MXUs

A TensorCore has 4 independent MXUs. They are identical, except that units 2 and 3 take 6 cycles longer to give their first result.

Internal registers

Each MXU has 4 internal registers:

GMR holds the matrix W that every matmul multiplies by.
msra and msrb are staging registers. Vector registers are pushed into them with matpush, and lmatreg then copies one of them to the GMR or the LMR. With two of them, you can load the next matrix into one while the unit multiplies with the other.
LMR is loaded from msra or msrb by lmatreg.lmr.

Types

A vector register is 8 sublanes of 128 lanes, one 32-bit word per lane. Lane l always holds column l. The type sets how many numbers share a word, and so how many matrix rows one register holds:

Type	Numbers per word	Rows per register	Where row r is
f32r	1	8	sublane r
bf16	2	16	sublane r / 2; the low half if r is even, the high half if odd
bf8, if8bf16, u8, s8	4	32	sublane r / 4, byte r mod 4
u4, s4	4 bytes, each holding two numbers	32 rows of bytes	sublane r / 4, byte r mod 4

A register given to matpush is that many rows of W (with .t, that many columns), and a register given to matmul that many rows of A.

A GMR loaded by lmatreg.gmr puts the unit in one of three modes: float (every float type), 8-bit integer (u8, s8) or 4-bit integer (u4, s4). A multiply runs only in its GMR's mode, and matmul.lmr in its LMR's. Signedness can differ between the two sides. Any other multiply computes nothing, so no rows get pushed to the mxu queue, and a vpop.mxu waiting for them hangs the core.

The staging registers are stricter: all the pushes into one must have the same type, except that f32r and bf16 pushes can be mixed.

Floats are rounded to bf16 (to nearest even) on the way in. Accumulation is done in f32.

Integers accumulate exactly in s32. In u4 and s4, each byte holds two numbers, one in its low 4 bits and one in its high 4 bits. The multiply pairs low with low and high with high: a W byte (w0, w1) meeting an A byte (x0, x1) adds w0·x0 + w1·x1 to the sum. So each result adds up 256 products instead of 128.

Push order and transposes

The staging register is a shift register of 128 rows. matpush.<type>.msr{a,b}[.t] pushes a vector register to it. The first push into an empty staging register decides how the pushes are done. With .t the push is done transposed and the vectors are pushed as columns. Without .t they are pushed as rows. Each push goes in at the top and moves everything already there. Past 128 rows the oldest rows drop out. If a push is done in the other orientation (plain or .t), or with a type that can't be mixed with the first, the push never completes and the core hangs.

An lmatreg resets the orientation and type.

Masks

A mask can be passed using vmsk. This will zero every byte whose mask bit is clear. It adds no latency. It is there for matrices smaller than the register: masking the part of a register that is outside the matrix makes it count as zeros, so whatever was left in it doesn't get into the sums.

Floating-point behavior

Two exceptions to the floating-point rules above: a result that would be denormal is +0 whatever its sign, and a zero sum is +0 even when every product is −0.

Systolic array

The MXU is based on a systolic array: a grid of identical cells that only talk to their neighbours. The weights are loaded first and stay in place: cell (k, j) keeps W[k][j]. The input tensor A flows through the grid. Each cycle, every cell takes a partial sum from the cell above and a value of A from the cell on its left. It adds the value times its weight to the sum, then passes the sum down and the value to the right. A[i][k] enters the grid k cycles after A[i][0], so it reaches cell (k, j) in the same cycle as the partial sum for out[i][j].

The MXU is a variant of a systolic array where each cell takes 8 consecutive products at a time instead of one. Within a cell, the first 4 products are summed with 3 bits more precision than f32 and rounded to f32, then the last 4 the same way; the two partial sums are added together, then added to the sum coming from the cell above.

For integer multiplies we couldn't recover the order of the additions as integer sums are associative and commutative.

Result queue

Results are always 32 bits, f32 for floats and s32 for integers, one per word. A multiply gives one result row per row of A, so matmul.f32r and matmul.lmr push one result register, matmul.bf16 two, and the 8-bit and 4-bit types 4.

Results go into the unit's queue one vector register at a time, and vpop.mxu takes them out oldest first. The queue has room for 48 slots. A register of float results takes 2 slots and a register of integer results 1. We do not know why: both hold 32-bit values. Said another way, the queue holds up to 1.5 128 × 128 matrices of f32 results, or 3 of s32 results, at a time.

A multiply's room is reserved when it issues. If there isn't enough room left, the multiply waits, and so do the vector instructions behind it, including the pops that would make room: the core hangs. Pop the earlier results before issuing a multiply that wouldn't fit.

How fast

The latency in the table is the time until the first result. After that, the unit goes through 1 row of A per cycle for the float types and 2 for the integer types:

Type	Rows of A per multiply	Cycles per multiply
f32r	8	8
bf16	16	16
bf8, if8bf16	32	32
u8, s8, u4, s4	32	16

So a 128-row A goes through in 128 cycles (floats) or 64 (integers), plus the latency.

At one row of A a cycle, an MXU does 128 · 128 multiply-adds a cycle. Four MXUs at 1.5 GHz give 4 · 128 · 128 · 2 · 1.5·10⁹ ≈ 197·10¹² flops, which matches Google's official number. The integer path's 2 rows a cycle doubles that to Google's published 394 TOPS for int8.

The LMR

The LMR can be used instead of A to compute A · W using matmul.lmr: push A with .t, copy it to the LMR with lmatreg.lmr, and each matmul.lmr multiplies the next 8 rows of A by the GMR. For example, with W in v0 to v15 and A in v16 to v31, 8 f32 rows per register:

for g = 15 down to 0:             # W, last rows first
    matpush.f32r.msra u0, v(g)
lmatreg.gmr.msra u0               # GMR = W
for g = 15 down to 0:             # A with .t, last rows first
    matpush.f32r.msrb.t u0, v(16+g)
lmatreg.lmr.msrb u0               # LMR = A
for i = 0 to 15:
    matmul.lmr u0, width=128      # rows 8i to 8i+7 of A · W
    vpop.mxu v(32+i), u0

This gives the same result, bit for bit, as streaming A with matmul.f32r u0, v(16+i).

Copying a staging register filled without .t to the LMR transposes it: the same code without .t on A's pushes computes transp(A) · W. As we can only observe the results, we cannot tell whether the transpose happens when lmatreg.lmr copies the matrix in or when matmul.lmr reads it.

Each call to matmul.lmr gives the next 8 rows of the result. width says how many rows of A it goes through before it starts again from row 0.

width=128: calls 1 to 16 give rows 0 to 7, 8 to 15, ..., 120 to 127 of A · W. Call 17 gives rows 0 to 7 again.
width=32: calls 1 to 4 give rows 0 to 7, 8 to 15, 16 to 23 and 24 to 31. Call 5 gives rows 0 to 7 again. Rows 32 to 127 of A are never used.

width is 8, 16, 32, 64 or 128. It only sets where the walk wraps: 8 and 128 cost the same.

The first matmul.lmr after lmatreg.lmr fixes width until the next lmatreg.lmr. A matmul.lmr with another width hangs the core.

A matrix that was never loaded counts as all zeros: with no LMR, or no GMR, every result is 0.

Transcendentals (EUP)

Instructions

Instruction	Slot	Cycles	What it does
`vrsqrt.f32 vS`	valu3	6	1/√`vS`, from the chip's own tables.
`vexp2.f32 vS`	valu3	6	2^`vS`, from the chip's own tables.
`vlog2.f32 vS`	valu3	6	log2 `vS`, from the chip's own tables.
`vtanh.f32 vS`	valu3	6	tanh(`vS`), from the chip's own tables.
`vsigmoid.f32 vS`	valu3	6	sigmoid(`vS`) − 0.5, equal bit for bit to tanh(`vS`/2)/2.
`vrecip.f32 vS`	valu3	6	1/`vS`, from the chip's own tables.
`veup.push vS`	valu3	6	Pass a register through the EUP unchanged.
`vpop.eup vD`	vres0, vres1	1	Pop the oldest EUP result.

The latencies above are counted until the result is available in the EUP queue.

Details

EUP queue

EUP functions push their results to the EUP queue. The queue holds 16 results.

Special cases

vtanh returns exactly ±1 for |x| ≥ 8 and x itself for |x| < 2⁻⁷. vsigmoid returns exactly ±0.5 for |x| ≥ 16 and x/4 for |x| < 2⁻⁵. A NaN result of a function always comes out as 0x7fc00000 except for veup.push which passes a NaN through unchanged.

How the EUP computes its functions

The EUP does not compute its functions exactly. It approximates each one in three steps.

1. Reduce the input. An identity leaves each function a small range to compute:

vexp2: split x into an integer part i and a fraction f. 2ⁱ is just an exponent, so only 2^f, for f between 0 and 1, is left.
vlog2: write x = m · 2^e. The answer is e + log₂ m, so only log₂ m, for m between 1 and 2, is left.
vrecip: the same split gives 1/x = 2^−e / m, so only 1/m is left.
vrsqrt: the same split gives 1/√x = 2^−e/2 / √m. Halving e needs an even e, so there are two tables, one for even e and one for odd.
vtanh: tanh is odd, so only positive x matters. Above 8 the answer is 1 and below 2⁻⁷ it is x, so the table covers 2⁻⁷ to 8.
vsigmoid: no table of its own. It returns tanh(x/2) / 2, which is sigmoid(x) − ½.

2. Pick a segment. The top bits of the reduced input pick a segment, and a per-function table gives three integers for that segment.

3. Evaluate. The three integers describe a quadratic that gets interpolated to give the final answer.

Accuracy

For inputs between ½ and 2, the functions are good to 14 to 20 bits at the median. This is enough precision for bf16 and f16 but is well short of f32's 24 bits. vlog2 has the fewest correct bits: 11.4 at worst and 14.2 at the median. vtanh and vsigmoid are not always monotonic, especially for larger inputs.

DMA

DMA instructions copy data from one memory to another: this core's VMEM, SMEM and instruction memory, the chip's HBM, and the host's memory across PCIe. The program issues a transfer and carries on. When the transfer is done, it credits a sync flag, and the program learns that the copy is done by waiting on that flag.

A DMA moves data in granules of 32 bytes, 8 words of 4 bytes. Every length is a number of granules, whatever the two memories are, SMEM included. A VMEM row, the size of one sublane of a vector register, is 512 bytes or 16 granules, and a whole vector register is 128 granules.

Instructions

Instruction	Slot	Cycles	What it does
`dma.simple src, dst, len=sL\|imm, flag=sF\|imm` `[, sync_mode=1, upper=sU, thread=1, trace=1, relaxed=1]`	s0 and s1	varies	Copy `len` granules from `src` to `dst`, then credit `flag`.
`dma.strided src, dst, len=sL\|imm, flag=sF\|imm,` `run=sR, dst_stride=sD, src_stride=sS` `[, sync_mode=1, upper=sU, thread=1, trace=1, relaxed=1]`	s0 and s1	varies	The same copy, in strided runs.
`dma.general src, dst, len=sL\|imm,` `override=sC, dest_flags=sF, src_flag=sG` `[, stride_info=sI\|imm, strided=imm, sync_mode=1, upper=sU, thread=1, trace=1, relaxed=1]`	s0 and s1	varies	Cross-device DMAs.

A bundle holds at most one DMA. A DMA instruction spans both s0 and s1.

The operands:

src, dst: the source and destination addresses, each in its memory.
len: the length in granules.
flag: the sync flag the transfer credits.
upper: the high 32 bits of a host address.
thread: 0 or 1. On VMEM → HBM and HBM → VMEM, thread=1 makes the transfer low priority. It changes nothing else.
trace: 0 or 1. trace=1 lets the DMA write its records to the hardware trace. Data, credit and timing are unchanged.
relaxed: refused on every copy. A fill accepts it, and nothing visible changes.
run, dst_stride, src_stride, override, dest_flags, src_flag, stride_info, strided are explained below.

Details

Memories

Memory	Address counts	Aligned to	Size	Used as
VMEM `mem.vmem`	granules	16 granules	128 MiB	source and destination
HBM `mem.hbm`	granules	16 granules	16 GiB	source and destination
SMEM `mem.smem`	4-byte words	any word	1 MiB	source and destination
IMEM `mem.imem`	granules	8 granules (4 bundles)	4 MiB	source and destination
host `mem.host`	bytes	none (low 5 bits dropped)		both, not host → host
TensorCore 1 `mem.tc1.*`	Refused: a v5e chip has one TensorCore, so there is no TensorCore 1.
SparseCore N `mem.scN.*`	Refused: these are a SparseCore's `spmem`, `smem`, `imem` and `timem`, and a v5e has no SparseCore.
all TensorCores' VMEM `mem.vmem_all`	Refused, even though this core is the only TensorCore on the chip.
memory id 3, no name `vmem`, `vmem.noncore`	Refused. The assembler has no name for it.

A misaligned address stops the core. A transfer that runs past the end of the memory stops the core.

Host addresses count bytes and are 64 bits wide, too wide for one scalar register. The address register holds the low 32 bits and upper=sU the high 32 bits.

A host address doesn't need to be aligned: the transfer starts at address & 0xffffffffffffffe0.

Not every direction is possible:

	to VMEM	to SMEM	to HBM	to IMEM	to host
from VMEM	runs	runs	runs	runs¹⁹	runs
from SMEM	runs	runs	runs	runs¹⁹	runs
from HBM	runs	runs	runs	runs¹⁹	runs
from IMEM	runs	runs	runs	runs	runs (`dma.general`)
from host	runs	runs	runs	refused¹⁹	refused

¹⁹ The copied data must be valid bundles. If not, the core stops.

Lengths

len is 0, a multiple of 16 granules (whole 512-byte rows), or 1:

A length of 0 moves nothing. The flag keeps its value but its done bit is set.
A length of 1 moves a single granule, 8 words, and only between SMEM and HBM: SMEM → SMEM, SMEM → HBM, HBM → SMEM or HBM → HBM. When the length is 1, every alignment is valid.

Source and destination address

The source and destination addresses passed to a dma.general or a dma.simple can be either a vector register or a scalar register. In the case of a vector register the DMA only reads element 0 of the register.

dma.strided can't: both its addresses always come from scalar registers.

Strided transfers

dma.strided copies len granules as a series of runs. Everything in this section counts granules (32 bytes), except run, which counts rows of 16 granules:

len: the number of granules to copy, in total.
run: the length of one run, in rows. run = 2 is a run of 32 granules.
src_stride and dst_stride: the number of granules from one run's start to the next one's, on the source and on the destination. They can be negative.

A negative stride walks backward from the start address:

A zero src_stride copies one run again and again:

A zero dst_stride writes every run onto the same rows:

In pseudocode, where src and dst are the two addresses in granules and the memories are arrays of granules:

run_len = run · 16          # granules in one run
k = 0
while k · run_len < len:
    # strides are signed 32-bit numbers
    s = src + k · src_stride
    d = dst + k · dst_stride
    # the last run can be cut short
    n = min(run_len, len − k · run_len)
    if s or d breaks the alignment rules of Memories: stop the core
    for i in 0 .. n − 1:
        dst_mem[d + i] = src_mem[s + i]
    k = k + 1
flag += len

SMEM is the other exception to counting in granules: its addresses count words. Its strides still count granules, so a stride of 1 moves 8 words.

General transfers

TPU chips in a slice are wired together by ICI, the inter-chip interconnect. dma.general is the one DMA that can send data across it, into another chip's memory; dma.simple and dma.strided stay on this chip or go to its host. As we worked on a single TPU v5e chip, we have not seen a remote transfer land.

dma.general adds three things to dma.simple: another chip as the destination, separate flags for the two ends, and a stride descriptor of several levels.

Override

override is a 32-bit register that can change a dma.general at run time without changing the program. Bits 16 to 28, except bit 23, are masks: each one is XORed into the matching field of the instruction: the source memory and core, relaxed, sync_mode, and the destination memory and core. The DMA then runs as if the instruction had encoded the result. The other bits are taken as they are. Bit 29 turns an SMEM-to-SMEM copy into a single 4-byte write. Bits 0 to 13 hold what we take to be a chip id, used only when bit 14 is set; with bit 14 clear the transfer stays on this chip or goes to its host. Bits 15 and 23 do nothing, alone or with thread, trace, relaxed or sync_mode set. Our reading is that bit 14 sends it over ICI to chip chip_id, and that the masks let one compiled kernel be pointed at different memories or chips at run time.

Sync flags

A dma.general takes two flag operands, both registers. dest_flags is a flag word like dma.simple's, naming up to two flags, credited once the data has landed. src_flag is a single flag. Our reading is that on a remote transfer dest_flags are the receiving chip's and src_flag the sending chip's.

Strides

dma.general can step through memory with up to three nested strides. They sit in SMEM, in a stride descriptor the program writes before the DMA. strided says how many levels there are, between 0 and 3. stride_info is the SMEM word where the descriptor starts. It must be a multiple of 16 or the core faults. Each level is three words: source stride, destination stride and count. Strides count granules.

In pseudocode, for three levels, with the descriptor holding S0 D0 c0 | S1 D1 c1 | S2 D2 c2:

run_len = c2 · 16           # granules in one run
copied = 0
for i0 = 0, 1, 2, ...:      # until len granules are copied
    for i1 in 0 .. c0 − 1:
        for i2 in 0 .. c1 − 1:
            s = src + i0 · S0 + i1 · S1 + i2 · S2
            d = dst + i0 · D0 + i1 · D1 + i2 · D2
            n = min(run_len, len − copied)
            copy n granules from s to d
            copied = copied + n
            if copied == len: done

Filling instead of copying

All three DMA instructions take an optional field sync_mode=1, which turns the copy into a fill: the source register's value is written, unchanged, to every 32-bit word of the destination. The source memory isn't read, but the value must still be a valid source address: a multiple of 16 for a VMEM or HBM source, anything for an SMEM source. A fill to or from the host is refused.

relaxed=1 is refused on every copy. A fill accepts it, and it changes nothing visible there.

The sync flag

A DMA adds len to its flag, so the wait for a 16-granule copy is sync_greater_equal flag[n], 16.

A long transfer doesn't credit its flag all at once: it credits it about every 2048 granules (64 KiB). The done bit is set once, at the end of the transfer.

The add wraps at 32 bits, and mixes freely with the core's own add_sync and with other DMAs on the same flag.

The flag operand names one flag, flag=flag[6], or two, flag=(flag[6], host.flag[7]). The transfer credits both. Our reading is that this is mainly useful for tracing: one sync flag can be used for synchronization within the TensorCore, and the other one used to transfer the DMA progress back to the host. A credit to an invalid sync flag stops the core.

Ordering with vector code

A DMA is written in slots s0 and s1, and it runs in two halves. At fetch, like any scalar instruction, it reads its scalar registers: addresses, length, flag, upper. It then joins the vector queue and issues from it like any vector bundle.

The last sync flag credit is done with a bit of delay compared to when the transfer is done. The specific time is not fully deterministic and depends on the source and destination memories, but as a rule of thumb has been observed to be roughly 150 cycles. We have a weak suspicion that the DMA sync flag credit mechanism is done on a 1150 MHz clock but do not have much evidence and explanation for it.

Accesses to scalar memory using sld and sst always win against a pending DMA. The story is more subtile for vector memory access using vld and vst. The dma engine has an arbitration mechanism to avoid being fully starved off from core-issued vector memory accesses.

In practice busy polling the memory manually to detect the end of a DMA is a bad idea as it will slow the DMA.

What a transfer costs

DMA timing is less regular than the rest of the core. All times here are from the issue to a wait on the flag that returns, which adds about 17 cycles to the transfer itself.

Granules per cycle on long transfers:

	to VMEM	to SMEM	to HBM	to host
from VMEM	~15.8	1.0	~15.8²⁰	~0.3
from SMEM	1.0	0.5	1.0	not measured
from HBM	~15.8	~0.97	~6.3	~0.3
from host	~0.3	not measured	~0.3	refused

²⁰ 15.8 for the first 0.7 to 1.4 MiB, then about 13.6.

From the above:

Transfers between VMEM and HBM move about one VMEM row (16 granules) a cycle. With the TensorCore's 1.5 GHz clock this is about 755 GB/s, close to the HBM's 819 GB/s.
Transfers with an SMEM end move one granule a cycle.
HBM → HBM moves 6.3 granules a cycle each way.
HBM refreshes every 3.9 µs. A DMA hitting the chunk of addresses being refreshed in HBM will wait for 300 to 400 cycles.
VMEM → HBM starts at 15.8 granules a cycle and drops to about 13.6 after 0.7 to 1.4 MiB. We can't explain this drop.
The host link caps each direction on its own: writes to the host reach 14.74 GB/s and reads 14.15 GB/s. That is roughly 90 to 94% of a PCIe Gen3 x16 or Gen4 x8 link.

Queues and limits

DMAs go through a network of queues before being executed as shown below. Each final queue runs its transfers one after another in issue order, except when they are prioritized. Each stage passes its oldest DMA on when the next stage has room. A DMA issue might stall if the queues are full.

We couldn't pin down how deep these queues are and are not confident they really are queues (hardware pipelining can easily be mistaken for queueing!). The number of entries a DMA takes depends on the instruction (simple, strided, or general with one or more levels of strides), on where it goes and comes from, and on the address range. Host DMAs, for example, are counted differently from on-chip ones in ways we don't understand.

Transfers in different queues run at the same time. Two transfers in different queues can still slow each other down where they use the same memory:

VMEM can move about 24.4 granules a cycle across concurrent DMAs.
HBM is capped at about 16 granules a cycle across concurrent DMAs, and at about 13.7 when reads and writes mix.
SMEM is often the bottleneck as it can move at most one granule a cycle.

Priority

All three DMA instructions take an operand thread=0|1. On VMEM → HBM and HBM → VMEM only, thread=1 makes a transfer strictly low priority: a later thread=0 transfer in the same queue goes first, and the thread=1 one waits, with no limit, for as long as thread=0 transfers keep coming, even if it has already started. On every other path thread does nothing.

Misc

Instructions

Instruction	Slot	Cycles	What it does
Writing a flag: flag `n` of this core
`set_sync flag[n], sY\|imm`	misc	3²¹	Set a sync flag.
`add_sync flag[n], sY\|imm`	misc	3²¹	Add to a sync flag.
`set_sync_done flag[n], sY\|imm`	misc	3²¹	Set the flag and clear its done bit. The value is optional.
`add_sync_done flag[n], sY\|imm`	misc	3²¹	Add to the flag and clear its done bit.
`set_sync_done_inverted flag[n], sY\|imm`	misc	3²¹	Set the flag and set its done bit.
`add_sync_done_inverted flag[n], sY\|imm`	misc	3²¹	Add to the flag and set its done bit.
`set_public_access flag[n], sY\|imm`	misc	1	Set a flag's public-access bit.
Waiting on a flag
`sync_equal flag[n], sY\|imm`	misc	2²²	Hold the vector side until the flag = the value.
`sync_not_equal flag[n], sY\|imm`	misc	2²²	Hold the vector side until the flag ≠ the value.
`sync_greater_than flag[n], sY\|imm`	misc	2²²	Hold the vector side until the flag > the value.
`sync_greater_equal flag[n], sY\|imm`	misc	2²²	Hold the vector side until the flag ≥ the value.
`sync_less_than flag[n], sY\|imm`	misc	2²²	Hold the vector side until the flag < the value.
`sync_done flag[n]`	misc	2²²	Hold the vector side until the flag's done bit is set.
`sync_not_done flag[n]`	misc	2²²	Hold the vector side until the flag's done bit is clear.
`sync_*.yield …`	misc	2²²	The yieldable form of each wait above, with the same operands.
Reading a flag
`read_sync flag[n]`	misc	1²³	Push a flag's value to the sync-flag result FIFO.
`read_sync_done flag[n]`	misc	1²³	Push a flag's done bit to the sync-flag result FIFO.
`read_sync_public_access flag[n]`	misc	1²³	Push a flag's public-access bit to the sync-flag result FIFO.
Atomics: flag `n` of any core, on any chip
`atomic.write flag[chip:core:n], sY\|imm`	misc	46²⁴	Write the flag.
`atomic.write.done flag[chip:core:n], sY\|imm`	misc	46²⁴	Write the flag and clear its done bit.
`atomic.write.done.inv flag[chip:core:n], sY\|imm`	misc	46²⁴	Write the flag and set its done bit.
`atomic.add flag[chip:core:n], sY\|imm`	misc	46²⁴	Add to the flag.
`atomic.add.done flag[chip:core:n], sY\|imm`	misc	46²⁴	Add to the flag and clear its done bit.
`atomic.add.done.inv flag[chip:core:n], sY\|imm`	misc	46²⁴	Add to the flag and set its done bit.
Holding the vector issue
`delay imm`	misc	N + 2	Hold the vector issue N + 2 cycles, N being a small immediate count.
`vdelay imm\|sX`	misc	N + 1	Hold the vector issue N + 1 cycles, N being the operand's value.
`vst_fence`	misc	1	Vector store fence.
Trace, interrupts and random numbers
`trace imm\|sX, imm\|sY`	misc	—²⁵	Append a hardware trace record.
`set_tracemark imm\|sX`	misc	—²⁵	Set the trace record's mark.
`core_interrupt imm`	misc	—²⁵	Raise programmable interrupt N mod 4 of this core's bank; the program runs on. A halt raises ProgramCompletion instead.
`read_prng vD`	misc	1	Read the PRNG state.
`vrandom vD`	misc	1	Draw a register of random words.
Core-to-core FIFO
`vpop.ccrf vD`	vres0, vres1	—	Pop the core-to-core vector FIFO.

²¹ A read_sync or a wait issued within 3 cycles of a flag write is held until those 3 cycles have passed.

²² Vector-issue cycles when the condition already holds. Otherwise stall until the condition holds.

²³ The value can be popped with vpop.sfrf 13 cycles after the read_sync.

²⁴ For a flag of this same core: the cycles from the atomic's issue until a read_sync reads the new value.

²⁵ No result.

Details

Sync flags

There are two ways to read a flag. read_sync copies its value into a queue that the scalar side has access to. A wait (sync_greater_equal and the other sync_* instructions) holds the program until the flag reaches a value.

A sync flag is either public or private, and starts public. Both the software driver and a TPU instruction can set that property.

A private flag can be read from software but its value cannot be set. A public flag can be read and set from software. A host write to a private flag is dropped, and a running core stops at once. Later runs don't start until a PCI function reset, which also makes every flag public again.

A vpop.sfrf gets a read_sync's value 13 cycles after the read_sync; a pop issued sooner waits for it. The sfrf queue holds 16 values.

Waits

A wait whose condition already holds costs 2 cycles. Otherwise the vector side stalls until the condition is true. In the meantime the scalar side runs on until it is blocked on the vector side.

Atomics

Atomic instructions allow accessing sync flags from other cores and chips atomically.

A single v5e chip has only one TensorCore, so atomics are of little use there. An atomic write to a flag of this core can be read back 46 cycles later.

An atomic names its flag with three fields, written chip:core:n in the table above. In the instruction they are packed into one 32-bit word, an immediate or a register:

Bits	Field
0–13	the flag number
14–16	the core selector
17–31	the chip

Selector	Core
0	`RESERVED_SELF`
1	`NON_CORE`
2, 3	TensorCore 0, 1
4 to 7	SparseCore 0 to 3

Overflow of `add_sync`

An add_sync whose signed 32-bit sum overflows still writes the wrapped sum, then the core stops. A DMA credit that overflows a flag does the same.

Delay vs vdelay

delay and vdelay have the same purpose: to hold the vector issue for a number of cycles. delay takes only a small immediate count, while vdelay takes either a register or a larger immediate, at the cost of using one of the bundle's immediates.

vpop.ccrf

Transfers a vector from one TensorCore to another. It is backed by one queue per TensorCore. Its latency and precise semantics are unknown, as a TPU v5e has only a single TensorCore.

yield instructions

We were unable to pinpoint how .yield forms differ from non-yield forms.

Documenting an undocumented chip: the TensorCore of TPU v5e

Methodology

The TensorCore at a glance

The chip

The core

Floating-point behavior

Bundles

State

Reading the instruction tables

Scalar side

Instructions

Details

Predication

Branches, calls and delay slots

Pops and the result FIFOs

Divide

SMEM

Arithmetic specific to the scalar side

Scalar register used as a destination twice in one bundle

Vector ALU

Instructions

Details

Packs and unpacks

Sublane moves

Converts

vclassify

Masks

Instructions

Details

vmkmask

vmksublane

Vector loads and stores (VMEM)

Instructions

Details

Addressing modes

Masks

Ports: how many loads and stores fit in a cycle

Write queues

Cross-lane units (XLU)

Instructions

Details

Matrix and "stream"

Depth of the xpose queue

xpose.binary.vy

Segmented transpose

Permutation patterns and setpcr

Instruction family and performance

Summation order of xlane.add

Matrix units (MXU)

Instructions

Details

How the matrix units work

MXUs

Internal registers

Types

Push order and transposes

Masks

Floating-point behavior

Systolic array

Result queue

How fast

The LMR

Transcendentals (EUP)

Instructions

Details

EUP queue

Special cases

How the EUP computes its functions

Accuracy

DMA

Instructions

Details

Memories

Lengths

Source and destination address

Strided transfers

General transfers

Override

Sync flags

Strides

Depth of the `xpose` queue

Permutation patterns and `setpcr`

Summation order of `xlane.add`

Overflow of `add_sync`