1st, 2nd, 3rd.

Doing some of my piling-up-fast readings and I just realized that I never questioned why the Third World is called the Third World. I've always thought that the Third World is the poor countries and the not-Third World countries are the rich ones. So if this is my definition, and the difference between a Third World and a non-Third World rests on a simple binary, then why is it called the Third World? For it to be the third World, then there must be a First and Second World, right? It's not simply a matter of rich vs poor then?