You usually set the custom sort-order manually by dragging thumbnails. However, when you perform an operation that automatically moves thumbnails (for example, Move To Top, Find, etc.) the custom sort-order will be modified by PMIO. In this case, if a custom order already exists for the current location, you will be prompted with a warning and given the option to cancel the operation. This is to protect your carefully made custom order from accidental replacement.
You can delete the custom sort-order by choosing Delete Custom Sort Order from the File menu.