I get this face all the time, its the Stop taking pictures mom! face. This particular shot, I was just getting the settings fine tuned and she had had enough! From the Berkelee Jazz festival.
I can identify with your problem. I have two adult boys and a husband who either club together to do their best to spoil any portrait I might attempt with silly behaviour, or get extremely irritated. I have given up with them.
I do remember that face (and worse) from my kids. Once, when my daughter was particularly annoyed, my son said to her, "It's only a camera, not a gun."