I mentioned this on irc but figured a more permanent discussion topic is better. Physfs apparently is not very thread efficient. It has a very simple big lock basically. You can see Methieu Ropert's talk about it at CppCon.
CppCon video: https://www.youtube.com/watch?v=TcuPIVKNSN0
Blog post: https://mropert.github.io/2020/07/26/threading_with_physfs/