When nine rises are enough

The design under examination cuts a band at nine rises, evenly spaced in the logarithm of the rise, always including both ends and the handover itself. It was chosen for a stated reason: sixteen cut stems a rise at a hundred rises a band is hours of work for a quantity the design predicts to be constant, and a constant is tested at the extremes and at the crossing.

Then the quantity turned out not to be constant, and the sweep at full resolution said so. The design has been treated as the villain of that story since. It is not, and the second band is what shows it.

Two questions, and they come apart #

There are two things a sampling design can be asked. Does anything change? and how much, and where? The nine-rise design answers the first correctly on both bands and the second correctly on only one.

On the golden 8/13 band it says yes and there are nineteen changes; it finds five. On the Lucas 7/11 band it says no and there are none; it finds none. So it has never been wrong about whether, and it was wrong about how many by a factor of nearly four.

That split matters because the two questions were asked in different essays. The first was asked by the design that built the six bands and the second only arose once the first had been answered.

What was actually claimed at nine rises #

It is worth going back to the wording, because the criticism has to be aimed precisely. The nine-rise sweep reported that the survivor at a fixed offset is not constant across a band, naming three offsets on the golden band that change.

That claim is true. What it also did was quote the changes’ positions in its own steps — two to four sweep steps below the crossing — and that reads as a measurement and is not one, because one of its steps is sixteen rises.

So the failure was a units problem #

A position quoted in sampled steps is a position quoted in the sample. At full resolution the changes sit 8 to 58 rises below the crossing rather than two to four of anything, and the three offsets that change cross at three different rises.

That correction is worth more than the numbers on either side of it, and it is not a criticism of nine rises. It is a criticism of quoting a distance in units the instrument does not resolve.

And a resolution problem #

The other thing nine rises could not do is see a feature narrower than its own step. The golden band’s changes are thirteen islands one to three rises wide with gaps of 1 to 48 rises, and one step of the nine-rise design is sixteen rises.

A sample whose step is sixteen rises hits an island about one time in eight. It hit two of thirteen, which is what the arithmetic predicts, and hitting two of them with a run of the other family between them produced the alternating reading that made the whole thing look like a period.

The same arithmetic run forwards says something better about the design than the criticism does. Thirteen islands averaging two rises cover about a fifth of the 126, so nine sampled rises miss every one of them only about one time in eight — which is to say the design was 87 per cent likely to detect that the row is not uniform, and it did detect it, twice over.

So nine rises was never short of detecting the structure. It was short of describing it, and the two are separated by a large margin rather than a small one. Getting from “there is something here” to “the islands are one to three rises wide” is not a matter of a few more samples: it needs adjacent rises, which is the full sweep, because a width of one rise cannot be measured by anything that steps more than one rise.

That gives the design ladder three rungs rather than two. Nine rises says whether anything changes and is right about it on both bands. Roughly thirty would put two or three samples inside a speckled stretch and would say that the changes are scattered rather than single — a real improvement for a third of the cost of the full sweep. Every rise is what measures a width, and nothing cheaper does.

The nine-rise design was chosen for the first of those and used for the third. That is the whole of the error, and it is an error of reading rather than of budget.

What the second band adds #

On the Lucas 7/11 band there are no islands. There is nothing narrower than the sample’s step because there is nothing at all, and the sample and the sweep agree exactly: none and none.

That is a case the criticism could not have anticipated and it changes what the criticism can say. The design is not unreliable; it is reliable exactly when the thing it samples has no structure below its step, and it has no way of knowing whether that is the case.

The band it is right about is uniform in every cell that holds an answer, which is a strong statement precisely because it was made at full resolution and could have been made at nine rises and meant almost nothing.

Which is the general form of the problem #

That last sentence is the uncomfortable part and it is not special to this design. A sample cannot report on structure finer than its own resolution, and it cannot report that it has failed to, because failing to see something and there being nothing to see produce the same output.

So was this sample enough? is not a question a sample can answer. It is a question answered by running the finer measurement, which is what makes the finer measurement worth its cost even when it agrees.

The cost of finding out #

Cutting a band whole is 1,612 to 1,890 cut stems and thirty-eight to sixty-seven minutes. Cutting it at nine rises is about a tenth of that. So the finer measurement costs roughly ten times the coarse one, and it was run twice.

Both were worth running and for different reasons, and the ten-to-one ratio between them is what makes the middle rung of the ladder above worth having at all. The first changed several numbers the collection had been quoting; the second changed none and bounded a claim, which is the return a null result buys and is usually undersold.

Where the sample was right about the ends #

One more thing the nine-rise design got right on both bands, and it is the thing it was built for. The coarse end of the golden band keeps 8 at every offset and the fine end keeps 4 at the three offsets that ever change.

Both are confirmed at full resolution. The design was built to test a constant at its extremes and at the crossing, and at the extremes and the crossing it is right on both bands.

And about the handover #

The claim the thread rests on is that no change of surviving family sits at the rise where the two contact steps change places. The nine-rise design says so on all six bands and the full sweeps say so on the two they cut.

That is a claim about a single rise, which is a rise every version of the design visits. A sample that always includes the point of interest can be trusted about the point of interest, and that is not a coincidence — it is why the design includes it.

Designing a sample for what it will be asked #

The lesson that generalises is about the second question rather than the first. A design built to test is this constant? answers that well and answers where does it change? badly, because the two need different resolutions.

Nine rises were chosen against the first question and the answer to it produced the second question immediately. That is the ordinary sequence and the ordinary mistake is to answer the second with the instrument built for the first.

What a step of sixteen rises is #

It is worth making the step concrete because nine rises sounds like a lot. The golden band spans a factor of 1.28 in the rise across 126 steps of two parts in a thousand each; nine cuts across that is a step of about three per cent in the rise.

Three per cent is enough to step clean over a band on some rungs — that is the argument for sweeping a band at a ratio rather than an absolute amount in the first place. Inside a band it is enough to step over an island thirteen times out of fifteen.

The narrow bands are still sampled #

Four of the six bands have only ever been cut at nine rises, and two of those are too narrow for the question to arise: the Lucas 3/4 band is sixteen rises wide, so its nine cuts are a step of under two rises and are nearly a full sweep already.

The other two — the 3/5 band at 70 rises and the Lucas 4/7 at 86 — are sampled at steps of eight and ten rises. Those are the cases where the same undercounting could be happening and nothing has looked.

Neither wrecks at its coarse end, because a coarse enough stem cannot be wrecked by a single removal, so a full sweep of either would be measuring over a shorter stretch than its width suggests.

What it would cost to close that #

The 3/5 band is 70 rises and the Lucas 4/7 is 86, so cutting both whole is about 2,300 cut stems and roughly fifty minutes. That would take the sample of fully cut bands from two to four.

It would also change what can be said. Two cases support one band has a transition region and one does not; four would support a rate, and a rate over four bands is still small but is a different kind of statement.

A sample that includes its own ends #

There is one design feature worth defending explicitly. The nine rises always include both ends of the band and the handover, which means three of the nine are placed rather than spaced.

That is why the ends are right on both bands and why the handover claim is safe. A design that spaced nine rises evenly and let them fall where they fell would have been worse at the questions it was built for and no better at the ones it was not.

What the two bands say about the design’s future #

The design is still the right one for a band that has not been cut whole, because it is a tenth of the cost and it answers the question it was built for. What has changed is what may be quoted from it.

Positions are out: a change located between two sampled rises is located to within one step and should be quoted that way. Counts are out: a count of changes from a sample is a lower bound. Ends and crossings are in.

The comparison that makes it a measurement #

None of this would be sayable from the sweeps alone. It is sayable because the nine-rise design was re-run on both bands from the same table, so its output and the full output are the same code reading the same stems.

That is the same discipline as keeping the six original lattices inside the larger table when the slot design was extended: a comparison between two designs is worth nothing if the two designs are also two implementations.

What the design cost in the other direction #

There is a symmetric error the design did not make and it is worth crediting. It did not report a change it had not seen. The five changes it found are five of the nineteen that are there; none of them is spurious.

That is not automatic. A design that reads a noisy quantity at nine points and calls each difference a change would manufacture changes out of a stable row, and this one does not, because the quantity it reads is a family label rather than a number with an error on it.

So the design’s errors run one way. It undercounts and it misplaces; it does not invent.

A label is not a measurement with error #

That deserves a sentence of its own, because it is what makes a nine-rise design tolerable at all. Each cell of these sweeps is the answer to which family did this cut leave standing? and the answer is one of a small set of lags rather than a number.

A label read at nine points is nine correct labels. A number read at nine points is nine numbers with error bars and a temptation to draw a line through them. The first is the safer thing to sample and the sweeps here are of the first kind.

Where the same argument has already bitten #

This is the third instrument in the collection to be caught quoting a result in its own units. A residual was read against the window it was measured over; an onset was reported at the last organ of a run, which a run length produces whether or not the condition holds; and now a position was quoted in sampled steps.

All three have the same shape: a quantity that looks like a property of the object and is partly a property of the instrument. None of the three was found by a check; all three were found by moving the instrument setting and looking again.

The cheapest habit that would have caught it #

Vary the setting once. Not a sweep of settings, not a sensitivity analysis — one alternative value, run once, compared.

Every one of the three would have been caught by that. It costs whatever the measurement costs, once, and it is the difference between reporting a number and reporting a number with a claim about the instrument attached to it.

What a reader should carry #

That the coarse design was right about everything it was built to be right about, and that its one bad number was a position quoted in its own steps.

And that whether nine rises are enough is not a property of nine rises. It is a property of what is being sampled, it differs between two bands on the same ladder, and the only way to find out is to stop sampling.

What the picture at the top shows #

Two pairs of bars, one pair per band. In each pair the upper bar is how many changes of surviving family the nine-rise design finds and the lower is how many all the rises find.

On the Lucas band both bars are nothing. On the golden band the upper is five and the lower is nineteen, and the gap between them is one step of sixteen rises meeting thirteen islands one to three rises wide.

The one line #

The nine-rise design finds none of none on the Lucas band and five of nineteen on the golden one: it has never been wrong about whether a band’s answer changes, and it undercounts by a factor of nearly four where the changes are narrower than its own step.

Which case a band is in cannot be read from the sample, so the finer sweep is what tells anybody whether the coarser one was enough — and on one of the two bands it was.