At what age does life start going downhill?

Looking back, even in elementary school, everyone had differences in studies, sports, looks, or speech/ability.
It seems like if you live too casually when you're young, it starts causing headaches later on.