Put it this way, our ears haven't developed much since say, 10,000 (probably a lot longer then that actually) years ago. Back then, the loudest noises you were going to hear was some shouting, loud animals once in a while, and who else knows. Basically...our ears simply are not designed to hear 120+ DB noises like rock concerts, airplanes, the whole shabang. Sure, some people are probably more sensitive then others, but everyone is prone to losing some hearing, and everyone WILL lose some hearing over their lifetime because of all the additional noise we hear, especially if you're subjecting yourself to incredibly loud noises. Some will just notice it more then others. In fact, everyone human on the earth pretty much has what is known as tinnitus (ringing in the ear, also a sign of damage) because of all this additional noise we hear that our ears weren't designed for. As I said before, some people will be affected more and some will notice it more then others. In the end, I'd definitely recommend wearing some ear plugs. Your ears simply were not designed to endure the stress you're putting them through, there's no shame in throwing some plugs in.
I got tinitus years ago after scuba diving. I had trouble decompressing my ears and got a horrible infection. So, have great respect for ear protection. And I don't scuba dive anymore.
I don't usually wear earplugs at concerts, though I do stay away from the house speakers. If I stand too close to them, it actually hurts, especially during drum solos. Instead I try to stand near where the sound mixers are. The volume level isn't too bad there. And I figure, if that's where they're mixing the sound, that's where the sound is best.
I was reminded about this thread over the weekend. I went down to BamaJam. Yonder Mountain and Govt Mule were there. I noticed a guy with a baby in a backpack. He was standing on the front row, right in front of the house speakers. Eventually, one of the policemen made him move. He was pretty mad ("This is bullshit, man!"), but his kid will probably appreciate it in years to come.

