There is a karma threshold for downvoting, you likely haven't reached it yet. It's in the FAQ.
However, as to HN culture, typically anyone complaining about downvotes will be further downvoted. It usually does not matter what the rest of the comment says, if you complain about downvotes, HN will give you a lot more of them. I like to think that it is a way for the community to remind individuals that these are all imaginary points anyway and to not be attached to one's karma number anymore than one is attached to the number of the nearest speed limit sign.
However, as to HN culture, typically anyone complaining about downvotes will be further downvoted. It usually does not matter what the rest of the comment says, if you complain about downvotes, HN will give you a lot more of them. I like to think that it is a way for the community to remind individuals that these are all imaginary points anyway and to not be attached to one's karma number anymore than one is attached to the number of the nearest speed limit sign.