Sorry to hear about your problems.
If you give him negative feedback and he gives it back to you, you can post a response to his negative which can include an e-mail address or URL. So you can tell your story fully if you want to.
There's an asymmetry about feedback that works to your favor. Even if you get a negative feedback as a buyer, this is unlikely to impact your ability to buy other items in the future, as long as it's a relatively isolated case. Some sellers may reject your bid if you have a lot of negative feedbacks, but will probably give you a chance to explain. Everybody knows that there are lousy sellers out there, and I don't think anybody will hold a negative from one of them against you.
But, a negative feedback on a seller, especially a recent one, IS a big deal and will definitely affect their business. Most buyers will look at the seller's feedback history before they even consider making a bid. If the negative feedback is he took an inordinately long time to deliver and was rude about it too I suspect most potential buyers will immediately pass. They aren't going to write to him and ask what happened (and the only thing he can claim is that your comment isn't really what happened which nobody will believe).
If the seller sells a lot, this will be a pretty big deal for the 4-6 months until the link to the item/auction times out.
I think you can make an argument for both negative and no feedback and it's just sort of how you feel. Of course, you can lay this situation out for him - if he didn't realize that your feedback will matter more than his before, he will certainly get the message now. If you aren't trying to change the outcome of the transaction (e.g. get some money back from him) then it's probably not worth all the work.
I doubt that he will lay negative feedback on you first (his best outcome is probably no feedback mutually). If he does try to put a negative feedback on you, then you should response by dropping a nuke on him - Seller ripped me off - I paid in 6 minutes and never received the item. He can't comment anymore other than to say that you're a liar and you've just shut this account down for good.
Good luck,
David Fung