DL0188 Liquid Neural Networks

A Liquid Neural Network is a recurrent model whose hidden state is defined by an ordinary differential equation in continuous time rather than by a fixed-stride update rule, and whose effective time constant is computed from the current input and state instead of being frozen at training time. The canonical instance is the Liquid Time-Constant (LTC) network, where a neuron leaks toward a resting value at rate and is simultaneously driven by a nonlinearity that also appears in the denominator of the effective time constant, so each neuron can behave like a fast detector on one input and a slow integrator on another. The trained parameters are static after training; what is “liquid” is the input-dependent time constant and synaptic drive, which makes the system a family of dynamics selected on the fly rather than one fixed linear-time-invariant filter. Because the state is a function of physical time, the elapsed interval is an explicit argument to a numeric solver, which is what lets LNNs consume irregularly sampled sensor streams without imputation. Against this, a gated RNN applies the same learned transition once per token at an implicit constant stride, and a Transformer has no recurrent state at all: it mixes tokens by attention over the whole prefix, paying prefill and carrying a linearly growing KV cache.

(1) Continuous-Time State: the hidden vector is the solution of an ODE evaluated by a solver, so the model interpolates between observations instead of skipping to the next index.
(2) Liquid Time Constant: varies per neuron per step, giving input-dependent timescales from one fixed parameter set.
(3) Elapsed Time Is An Input: unequal gaps between samples are handled exactly by integrating over , which matters for event streams, medical records, and asynchronous sensors.
(4) Sparse Auditable Wiring: Neural Circuit Policies wire LTC neurons in a sensory, inter, command, motor hierarchy, and a 19-neuron command layer has flown real drones and driven real cars.
(5) Cost Scales With Solver Steps: compute is for solver steps per observation, with no KV cache, but the recurrence is inherently sequential.
(6) Provable Boundedness: the LTC construction bounds both the state and , so the dynamics cannot blow up the way an unconstrained neural ODE can.

Figure 1: One nonlinearity does two jobs. The same supplies the drive toward the resting potential and sets the effective time constant, and the solver spans whatever the sensor happened to deliver, so the grey internal steps, not the sequence index, determine the compute bill.

The contrast with a fixed-weight RNN is easiest to see through the timescale. An LSTM or GRU also has a forget gate that modulates memory retention, but the gate acts on a fixed discrete stride: the model learns one retention profile for “one step” and has no representation of how much real time a step covered. If your samples arrive at 3 ms and then at 400 ms, the gated RNN sees two identical steps unless you hand it the gap as an extra feature and hope it learns the exponential relationship. The LTC instead solves for the exponential, because enters the integrator directly. The contrast with a Transformer is structural rather than temporal. Attention is a stateless global mixer: it can reach any earlier token in one hop, which is why it dominates language, but it has no notion of physical duration beyond whatever positional scheme you inject, and its cost grows quadratically with context while its memory grows linearly. An LNN keeps a fixed-size state regardless of horizon, which is exactly what an embedded controller running at 50 Hz for hours needs, and exactly the wrong trade for retrieving a fact from 200,000 tokens of text.

Mathematical Formulation:

Where:

• is the continuous-time neuron state and the sensory input at that instant.
• is a bounded nonlinearity (typically a sigmoid of an affine map) with static trained parameters ; it appears twice, as the drive and inside the time constant.
• is the per-neuron leak time constant, the resting potential the drive pulls toward, and the supremum of that bounds the fastest achievable dynamics.
• is the liquid (effective) time constant; the bracketing inequality is what guarantees the state stays bounded and the solver stays stable.
• The last line is the fixed-weight RNN baseline: depends on step index only, with applied identically at every step and no anywhere in the update.

Figure 2: Left, the whole point of the denominator: a strong input collapses toward , so one neuron spans a wide band of timescales. Right, the same input pulse through a fixed leaky integrator and an LTC cell with identical ; the liquid cell charges roughly five times faster while driven and relaxes more slowly once the drive is removed, behaviour a single fixed-weight filter cannot produce.

The practical objection to LNNs is the solver. Every observation costs evaluations of , stiff dynamics push up, and the whole recurrence is sequential, so training cannot be parallelized across time the way attention or a convolutional scan can. Closed-form Continuous-time (CfC) networks answer this by replacing the numeric integration with an analytic approximation of the LTC solution, keeping the input-dependent timescale while deleting the solver from the graph, at reported speedups of one to five orders of magnitude in training and inference over solver-based LTCs.

Closed Form And Cost:

Here are separate learned heads of and acts as a time-dependent gate that interpolates between two candidate states as elapsed time grows, which is the closed-form stand-in for exponential relaxation. Note that has no term in and no cache: an LNN controller at can run indefinitely in constant memory, while the Transformer term grows with every token retained.