Tolerance in a perceptual space

A tolerance slider is a promise that one number means one thing. Measured as a distance in RGB it does not, and the picture where it fails is the ordinary one: a coloured object with a lit side and a shaded side.

The number that is not a number

Every colour-replacement tool has a slider marked tolerance, and almost all of them compute it the same way: take the colour you sampled, take each pixel, treat both as points in a cube whose axes are red, green and blue, and measure the straight-line distance between them. It is a reasonable-looking definition. It is also the reason those tools feel unreliable, and the failure is not subtle once you know where to look for it.

sRGB is a storage encoding. Its three axes were chosen so that eight bits per channel covers the range a display can produce without visible banding, and the transfer function that squeezes them is a compression curve, not a model of the eye. A step of thirty counts near black crosses a very large perceptual gap; the same thirty counts near white crosses almost nothing. So a tolerance of thirty is two settings at once — a tight one in the highlights and a loose one in the shadows — and in a photograph those two regions are the same object.

Thirty counts of RGB, at three points on the same axis
Sampled value	Same value plus 30	Roughly how different they look
a dark grey near 20	near 50	obviously two different greys
a mid grey near 120	near 150	a clear step
a light grey near 210	near 240	barely separable

The consequence in practice is a mask that grabs half the shadow side of an object and none of its lit side, or one that selects the object correctly and also the whole of a dark background. Widening the slider fixes one end and ruins the other, which is what makes the control feel like it is fighting you.

What a perceptual space changes

OKLab is a different set of axes over the same colours, chosen so that equal distances in the space correspond roughly to equal differences to a person looking at them. It is not perfect and it does not claim to be; what it is, is uniform enough that a fixed distance means a comparable amount of colour wherever it is measured. Converting an image into it costs one pass — about twenty-five milliseconds per megapixel — and the result is cached beside the original bytes for as long as the picture is open, so every subsequent slider move is a read rather than a conversion.

Once you are in that space, a second thing becomes available that RGB does not offer at all: the three quantities you actually care about separate cleanly. Lightness is one axis. The other two can be read as a polar pair — chroma, which is how much colour is present, and hue, which is which colour it is. Those are three independent questions, and this tool exposes them as three independent tolerances rather than collapsing them into one distance.

Why three sliders rather than one

Consider a red car in front of a brick wall on grey tarmac, with someone in a red jacket standing beside it. The paint and the brick share a hue and differ in chroma. The paint and its own shadow share a hue and a chroma and differ only in lightness. The paint and the jacket differ slightly in all three. Selecting the car and nothing else is a statement about all three axes at once, and there is no single scalar distance that expresses it. A combined tolerance has to trade the three against each other, and whichever trade it picks will be wrong for some ordinary picture.

The cost is that three sliders are more to learn than one, and that is a real cost rather than a rhetorical one. The mitigation here is the coverage figure: it sits under the sliders, updates as they move, and converts an abstract question — is this tolerance right — into an observable one. When a number jumps from four percent to thirty-one percent on one small drag, something large has joined the selection, and you can see that without understanding anything about colour spaces at all.

What the band actually looks like

Each tolerance is not a threshold. Full weight is held out to sixty percent of the stated tolerance and then falls away on a smoothstep to zero at the tolerance itself, and the three weights are multiplied. A hard threshold would produce a mask with a staircase on every gradient in the picture, and a photograph is made almost entirely of gradients; the feathered shoulder is what stops a recoloured surface acquiring a cut-out edge where its own paint meets its own shadow.

The hue term carries one further adjustment, and it is worth knowing about because it explains a behaviour that otherwise looks like a bug. The hue angle of a nearly grey pixel is arithmetic noise: at a chroma of four thousandths the angle swings through the whole wheel on rounding alone. A strict hue test on such a pixel would either admit every grey in the frame or reject the pale edges of the thing you are selecting. So the measured hue distance is scaled by how far above the chroma floor the weaker of the two colours sits, which hands the job of separating grey from colour to the chroma tolerance — the term that can actually measure it.

The practical reading of all of this is short. Sample in the middle of the region, not on a highlight. Open lightness first, because it is what gathers a surface and its shade. Close chroma when something large arrives, because it is what excludes a background. And watch the coverage figure rather than the picture, because the picture will look plausible long after the selection has stopped being one.

Back to the swatches

Every pixel is remapped locally — this page has no server to send to.