Your child wants the witches and the ghosts and the creaky old house — and absolutely does not want to be frightened. Any single one of them does the job. The rule I held to here is simple: nothing on this shelf will follow your child to bed. They get the shivery mood, the black cats, the fog, the vampire, and then they get to close the book and be completely fine.
Any of these five on its own delivers the goods, so pick by your child's age and what kind of evening you are having. Room on the Broom is the easiest place to start if you want a spot to land: a warm witch, a rhyme your child will chant with you, and a dragon who is gone in two pages. Gustavo, the Shy Ghost is the same safety in a different key, all Day of the Dead pinks and oranges, with a ghost whose only real problem is that he is too shy to say hello. Creepy Carrots! is the one that does the best impression of a horror film and then reveals itself as a joke about a rabbit outwitted by vegetables. Bunnicula gives an older child a whole vampire mystery where the victims are salad. The Spiderwick Chronicles: The Field Guide is the furthest this shelf goes, a creaky house with a spiteful little faery in the walls, still more mischief than menace. That is the ceiling here, deliberately. I left two famous books off, and the reason is at the end of the heads-up.