When starting from a cached state, we cannot assume that the initial
particle ordering is consistent with previous time-steps.