This headline and blog post strike me as disingenuous. The newspapers know how to use robots.txt. Clearly they want to be indexed; they just want to micromanage exactly how their content is presented by search engines. Let's criticize the newspapers' actual position, not a strawman.
> they just want to micromanage exactly how their content is presented by search engines.
They also want to be paid for ads on google's SERP. In some versions of this demand, they want revenue whenever their site is a result on said SERP. (The theory is that said ads are valuable because their site is a result on said SERP.) In others, they also want a revenue share from SERP pages that don't mention their site. (In that case, the theory is something like "the possibility of finding our content is why folks use Google".)
