As an indexer's-gallery fan I understand the 2 points of view. A possible solutions would be:
Option 1. Let the users choose between "infinity scroll" or "pagination" (with their pros and cons)
Option 2. Offer 2 files.gallery to download (the standard one, and/or the pagination version)
I had galleries with millions pictures and pagination solved the load problem on the servers. So, in this situation the feature choice-sort-frontend is not relevant for admins, simply there is only 1 sort by default for everyone and that's all.
In case of @Karl add this feature in the gallery...
If pagination option is selected by customer, then:
1- Specify number of images per page = XXX
2- Specify sort list by default = date-desc (choice between: "name-asc,name-desc,date-asc or date-desc")
3- And finally 'Sort-Options' have to be hidden/disabled from files-gallery-header-Frontend, since it is not necessary anymore for the visitor.