With recent changes, the pathway between refetching (on request) and
refetching (on requirement due to unpopulated files) was combined.
Unfortunately this pathway also added a forced invalidation, which
should not have been applied to the second case.
Closes https://github.com/ppy/osu/issues/19365.