Help, I'm stuck. Abandon or Be friends?

Now theres a friend at school who I've known since reception. He has for the past year being a complete to me, slagging me off calling me gay! At first i took it as a joke and just shrugged it off. But now its really pissing me off. I'm not gay, at all. He is a complete retard, he is the dumbest child I've ever known (no exaggeration) we just took our sats and he had to be taken into a special cl room and given the ******* answers! He's a complete reject. Basically as a human being he is a giant failiure. Yet he still seems to think that he can just slag me off. Last time i got pissed off with him I said so much stuff to him it made him cry!!! But there can be moments when he is safe and a really good friend. What do I do about him. Abandon or Stay friends?