DL0068 Computational Graphs

A computational graph is a directed acyclic graph whose nodes are operations or stored values and whose edges are the tensors flowing between them. Deep learning frameworks materialize this graph while running the forward pass: PyTorch records it dynamically on every iteration (define-by-run), while early TensorFlow built the whole graph statically before execution. Its central use is reverse-mode automatic differentiation: every node carries a known local derivative, so the chain rule becomes a mechanical backward sweep that produces every parameter’s gradient in a single pass, whose cost is roughly 2-3 times the forward pass and does not grow with the number of parameters being differentiated. The same data structure also drives memory planning such as activation checkpointing, whole-graph compiler optimizations like kernel fusion in XLA or torch.compile, device placement, and deployment export to formats such as ONNX.

(1) Structure and Evaluation: nodes are ops or values and edges are tensors; the forward pass evaluates nodes in topological order and caches every intermediate, because the backward sweep needs those values to compute local derivatives.
(2) One Sweep, All Gradients: each node multiplies its incoming adjoint by its local Jacobian and passes the result to its inputs, summing contributions at fan-out; a single reverse traversal yields gradients for all parameters, which is why backprop scales to billion-parameter models.
(3) Dynamic vs Static Graphs: PyTorch’s define-by-run graph follows arbitrary Python control flow and debugs like ordinary code; static graphs (TensorFlow 1, exported ONNX) trade that flexibility for ahead-of-time whole-graph optimization and portable deployment.

Figure 1: Forward pass of with : each op consumes its input tensors, emits one value, and caches it for the backward sweep.

The backward sweep traverses the same graph in reverse. Starting from the seed , each node applies the chain rule locally: the square node returns , the subtraction forwards to and to , the addition copies its adjoint to both and , and the multiplication swaps operands, delivering and . No node needs global knowledge of the loss; correctness of the whole sweep follows from composing local derivatives edge by edge. The figure shows and to make the mechanics uniform, but a framework skips those branches when the input and target carry requires_grad=False, so only and are actually materialized. When a value feeds several nodes (fan-out), its adjoint is the sum of the contributions along each outgoing edge, which is exactly the multivariate chain rule.

Figure 2: The same graph in reverse: each node multiplies the incoming adjoint by its local derivative, so one sweep accumulates and at the leaves.

The graph pays for itself well beyond gradients. Since it records exactly which intermediates feed the loss, frameworks can plan memory: reverse mode must retain cached activations, so training a depth- network costs activation memory, and gradient checkpointing cuts this to by storing only boundary activations and recomputing each segment during the backward pass, at the price of roughly one extra forward evaluation. A static snapshot of the same graph lets compilers fuse ops, pick kernel layouts, and place tensors across devices, and exporters freeze it into a portable artifact. The trade-off is flexibility: a define-by-run graph is rebuilt every iteration, so data-dependent if-statements and loops simply work, while a static graph must capture control flow symbolically, which is why TensorFlow 2 defaulted to eager execution and recovers graph benefits selectively through tf.function tracing.

Mathematical Formulation:

Where:

• is the scalar loss; are the weight, input, bias, and target scalars of the running example.
• is the output value of node , and is its adjoint, the gradient of the loss with respect to that node’s output.
• are the nodes that consume ; each child contributes its incoming adjoint times the local partial , and the results sum, which is the multivariate chain rule.