People call me sir.... I'm not that old yet!!! Am I?
It might just be them being polite. I sometimes call young people sir and ma'am. I never considered it might be a weird thing to do. Although if your age starts with a 3, then yes you are that old yet.
Most of the women I have encountered do not care to be called Ma'am, unless they are old school southern and quite matronly. Rare birds around here. Most female strangers of any age addressed by another stranger do not mind as much being called "Miss" in a respectful way.
Try it the next time you open a door or pick up something for some female stranger. Really, really, really, don't call them Lady. That will often backfire.