After a bit of research, it turns out Windows 10 (7 and 8 aswell) tries to guess what the content of the folder is and assigns special “rules” to it to optimize sorting and view settings.
For some reason, the “Downloads” folder likes to be categorized as a “Pictures” folder (I’m guessing because people store a lot of JPGs in it). The problem with this view is that it tries to generate thumbnails for all the files in this folder, even if they’re not pictures.
The fix is to ensure the folder settings for “Downloads” is optimized for “General items”. This should have an instant effect (the next time you add files to the folder since the thumbnails are otherwise cached anyway).
Unfortunately, this change doesn’t seem to always stick as Windows may change the option as you download more files, so you may have to check this setting once in a while.