I am not an expert on these things but I think you may be confusing "Greatest game of all time" with "most popular game of all time". I think Zelda and Metroid are so popular because they have such simple story lines. Complex story lines are often distracting, whether it be movies books or games. Look at adventure games like Siberia and Dreamfall or even Myst. Great games if you like that sort of thing but to complex to the point of boring to the majority of gamers.
I think Zelda was so great because the story is straight forward and people can relate to it. And it had a variety of challenges.
Also I do not think you cannot emphasize enough that Zelda was designed for teenagers with gameplay that could be enjoyed by their younger siblings and that could also enjoyed as these teenagers grew up. Most kids games are discarded at a certain age. Zelda has incredible longevity. Another thing about games like Zelda is that they seem to be interesting only to people who played them as kids. Young adults do not seem interested in these games so I think the success of Zelda has a lot to do with successful age group marketing. Being available on a handheld was a big plus. Being acceptable to parents is also a huge plus. Game developers don't seem to notice that Nintendo has made a fortune with games that parents will buy. Zelda has certainly benefited from that.
I think that most adult games like Psychonauts should be considered niche games and they will never appeal to a mass market like Zelda.
Gameplay should be simple I think. I remember trying to play the Spiderman game and thinking that the swinging was like some kind of weird physics test. If gameplay is to complex then people get irritated. The point of a game is to be able to kill the monster with 10 hits from your sword. 5 hits... the game is to easy, 15 hits the game is to hard and I have better things to do!