A couple of things. First, I used the phrase "debit card number" to basically mean, "they didn't have my physical card--but they did have the information to be able to use it."
Second, this happened circa 2004-2005. In the US, at the time, all card use was magnetic strip swipes. Debit cards occasionally used a pin at terminals equipped to do so; but, it wasn't a requirement and most use didn't involve the pin. As an example: a grocery store terminal at the cash register might require your pin; but, if you paid the bill at a restaurant, it would get swiped like any Visa/MC.
At the time, in an online setting, asking for additional information like CVV or using address for AVS wasn't standardized so not all merchants required anything beyond a card number and expiration date.
In that era, nearly everyone I knew had experienced a debit card or credit card stolen and used in this manner--and those with debit cards paid the extra price of actually losing our money for a time.
Debit cards in the US are still mostly swipe + PIN or chip. With the ATM there's daily limits of withdrawal amount but that's it. Maybe some issuers have a 2FA feature but none of the banks I've used have.
When contactless payment was new in NL, it was possible to pay up to € 25 without PIN, which set of all sorts of alarm bells in my head. € 25 isn't much, but the idea that payments could be done without my PIN felt deeply wrong, so I disabled it. I only really started doing contactless payment when it was supported by the banking app on my phone, which always asks for PIN.
That said, PIN isn't terribly secure either; for a common 4-digit PIN, there are only 10,000 possibilities. Too many to test by hand, but trivial to automate if you have access to the algorithm. And for the first PIN cards, the data was on a trivially readable magnetic strip. So if you have the algorithm and the contents of that mag strip, it's trivial to just run through all possibilities. That algorithm was a closely guarded secret, but still, security by obscurity isn't great, so later they replaced the mag strip with a chip that doesn't spill its contents so easily.
