Working with a customer’s laptop which was set up exactly as my own, their iCloud Photos folder was talking ten minutes to sort by date. Mine took about one second!
I found a fix, which is to use a different date to sort on.
Originally, the sort column was just ‘Date‘, like this:

I added another column by right-clicking on the Heading Bar, added ‘Date Created‘, unticked ‘Date‘ and dragged Date Created to where Date had been.

I won’t tell you how long it took me to work it out – let’s just say that it’s a relief to have finally resolved the issue.