that's weird.
i know that it is possible to use vl 5.9's iwconfig with passphrase ("s:<password>") because i used it myself (i'm currently on zenwalk, though). did you try it with s: ? or did you read the manpage and decided right away that you needed a hex key?
how did you enter the hex key - with "-" or without?

setting the essid is not really required, if there's only one wlan around, it will try to connect with this one.