It’s not browser cache – it uses PHP sessions which clear automatically when your server decides.
In this case, it’s better to either:
a) Turn off random order
b) Turn off pagination
Getting the two to work together better might be possible with some extensive coding, but even then I’m not 100% sure it’s possible.