|This is generally the point when your brain 'splodes.|
I've played horror games at conventions, but they were never particularly scary. Maybe it's the atmosphere of being in a room with a hundred other gamers with everyone talking, rolling dice, and hamming it up. I've always preferred to run my own games with the lights down. Or, even better, keep the lights off, with candles here and there to provide a more "organic" form of illumination. Put some creepy music on at a low volume (always preferred the Hellraiser score, myself) and it takes most of the work out of getting your players in the mood.
But I digress.
The scariest game I've ever played was one of my girlfriend's (now wife's) Call of Cthulhu games. There are three such games that stick out in my mind, and two of them were genuinely frightening. I'll focus on the one I recall the best, which was set in 1920's San Francisco. At least, I think it was. Anyway, there were two players in the game. One of us was an educated fellow, sort of a scholar. I, on the other hand, was playing a federal agent.
Our first investigation began as a missing persons case, or so we assumed, involving a little girl. We canvased the neighborhood where she'd been seen, talked to folks about her. No one could say where she lived, but they'd seen her here and there. In fact, one of them had called the case in, hoping we'd locate her and find out where her home was.
Eventually, we did find her. Her name was Madeline, but she was a bit odd. Once we'd spoken with her the first time, she sort of disappeared (not before our eyes, but she definitely wasn't there anymore). We did some research, and it turned out a girl about her age and description went missing near the turn of the century. It couldn't have possibly been her, since she was around eight or nine years old. Or could it be? Old photographs we dug up looked pretty much like she did, right down to the clothes she wore.
Long story short, she'd been taken away by some kind of extradimensional thing or being, which she referred to as "mommy." Apparently, there were lots of "mommies" where it had taken her. In order to get into the realm where this entity lived, you had to do a little hopscotch thing and recite a strange rhyme filled with words no sane man should ever hear, let alone speak. We didn't believe her, but when she showed us--and vanished right in front of us--there was a bit of sanity loss.
My character, as sentimental as he was, felt that he had to rescue this poor innocent girl from whatever evil had taken her away from her family all those years ago. This meant doing "the Hopscotch" (as Madeline called it) and reciting the rhyme she'd used. The creepiest part of the game was crossing that threshold and seeing what lay beyond it.
We did end up saving her, and our sanities suffered for it. My character ended up adopting Madeline. She was a strange little girl, probably insane, but she was also young and there was a chance she'd recover. My character made life as good for her as possible, but she never was what people of the '20's would call "normal." She was even a useful reference during future investigations, though my character was hesitant to go to that well too often.
That, in a nutshell, is the scariest game I've ever played in.
My wife doesn't run games anymore, for which I'm eternally saddened. She had a great grasp of storytelling, and she was probably the most internally consistent GMs I've ever had. Occasionally I bug her and whine about it, but she hems and haws. She doesn't really enjoy GMing, so I don't push the issue. I have her as a player, and that'll have to be enough.