The TreeSet was removing most overlays (except the ones I tested with) because they were equal to the comparator.