Also, if you decide to buy a memory card just for the PS2, then by all that's holy do NOT buy the Interact Memory Manager Plus one. I just got reamed by those f@ckers. You have to insert the Memory ...
I got my fist PS2 last week (never had a PS1 either and only save my games on the hard drive in my Xbox, so I'm new to this memory card thing). I bought one of those yello Nyko PS2 memory cards for it ...