When someone walks away, who's hurts the most?

I sitting here listening to a new person I had never heard before. I just love iTunes don't you?
Her name is A Fine Frenzy and you can buy her CD and its great stuff, If you like stuff that makes you think.
Anyway she is the giveaway song this week and I listening to a song called "almost lover"
and all I could think was, who hurts the most when it doesn't work out? I am married now
but I remember walking away from someone and never looking back. I met him again a few
years later and he was with the same women I had taken him away from. I knew when we started talking I could of done it again. Did I hurt him? Oh I bet.. Did I hurt? Well sure he was my high school love. Now I can't remember if it was love or lust. Now that I look back and see what I could of done different I am not sure it would of worked out anyway.

I know this is a bit off what I usually talk about, but I was just wondering when you break up with someone who hurts more you or him? Is it even pain or do we break up for reasons that we look back and wonder what the hell was I thinking? I had him/her in the hand and we could of had a life and I turned away from out future. Why do we not want to take the chance after the biggest break up in our lives hurts so bad you don't know if you could ever go through it again?
Its more questions than answers, but the next time you walk away, at least be willing to try again and don't lock your heart up just because it got broken once. You don't want to be the one who lives alone with house full of cats do you?