If you're going to use IE6 as the baseline that pretty much every browser was breaking the web relative to it, including later versions of IE, weren't they?
I don't recall many websites at the time only being effectively usable for perf reasons in Chrome, but my memory is a bit hazy. Certainly, there were fewer very JavaScript heavy sites, and most of HTML5 hadn't been created yet. That meant there were fewer opportunities to build web apps that required fast JS. Also, Chrome wasn't as fast as it is now.
But, yes, if people were building apps then that said "only in Chrome" then that would imply that to some degree Chrome was breaking the web. Fortunately, in that case, the other browsers caught up quickly and resolved that tension.