Silk has unique characteristics as a material, and in a pre-plastic pre-industrial world with relatively few viable fabric materials (and all of them profoundly labor-intensive by modern standards) it would have been valuable regardless.
Obviously its rarity, social connotations, and mysterious origins had a huge effect on its value. But like gold, its characteristics alone are enough to cause people to go through the trouble to acquire it initially, enough for those other factors to take over.
IMO we shouldn't have started wearing clothes to begin with, it was a bad call. If my ancestors could have just held off on that I'd have nice thick fur right now.
I mean… if you stick to the areas that mimic the early hominid climate you don’t really need them. Lots of modern tropical tribes wear functionally zero clothing.
Obviously its rarity, social connotations, and mysterious origins had a huge effect on its value. But like gold, its characteristics alone are enough to cause people to go through the trouble to acquire it initially, enough for those other factors to take over.