Yes! We have no bananas
It’s really hard to answer a question asked with a negative qualifier without completely restating what you mean. If I say simply, “No,” my answer is unclear. Technically, a negative answer means whatever I said was, in fact, what I really meant. Only a positive answer would mean what I said (or how it was interpreted) was not what I meant. The positive answer would be an agreement.
The problem is answering appropriately sets up a constructive nightmare. Either way I answer appropriately, it sounds as though I contradict myself:
“Yes, that is not what I mean.” (Yes is followed by the negative verb is not.)
“No, that is exactly what I mean.” (No is followed by the positive verb is.)
It’s not a hard-and-fast rule that following a positive with a negative verb (or vice versa) is always wrong or against the rules; it just makes for an awkward sentence. And technically speaking, it’s not precisely correct to say instead, “No, that’s not what I meant,” because the answer to the question asked would be one of the two options mentioned. It’s a confusing sort of double negative.
Of course, there are two issues here: One is the question is asked awkwardly, and the second is the answer is difficult to accurately and correctly state.
In the case of the first issue, the solution is a simple one: Avoid negative verb constructions in questions. Instead of asking if someone is not annoyed, ask if the person is annoyed. Instead of asking if the weather is not sunny, ask if it is cloudy (or sunny). The answers are going to be the same, but it makes it infinitely easier for your audience to respond appropriately and without further question.
The second issue is trickier because if you are on the receiving end of such a question, you must be sure you understand what is truly being asked of you and the person asking the question understands your answer correctly. The best way to accomplish this is to restate the question as part of the answer (“Yes, that’s not what I mean”) or structure your answer to make yourself easily understood without a positive or negative qualifier (“I’m not annoyed,” or “It’s cloudy”).
And if someone you know asks negatively-framed questions all the time, make a copy of this article and slip in his or her mailbox. Maybe you won’t have to think through your answers so carefully if your questioner takes my advice.