Well life or reality is stranger than fiction for sure.
What happened to your brother? Does he still have this mental problem? He tired to scare you and probably didn't realize the real affect of it who knows. Still, very bizarre behaviour.
That incident must have scarred you for life but I feel there might be just more to that than that- some deeply rooted problem. Because the fact that you are still afraid of the dolls because of that incident still feels superficial. Perhaps some hypno therapy is in order to get the real/underlying problem out in the open. They say these sort of problems typically lie in the unconscious mind and so if you are aware, then it shouldn't cause any problems - not to that point anyway but you are having problems. So maybe there's something more - something on the unconscious level.
I love watching horror movies but unfortunately they affect me and causes nightmares so I chose not watch those anymore.