Who do you call sir [or ma'am] when you are out of uniform?

In uniform most service members know who to salute and who to call sir or ma'am. Out of uniform some service members salute higher ranking officers while others show their respect for those they personally know and some do their best to avoid eye contact with higher ranking people.
Some of us have been trained to address people older than us as sir or ma'am but this seems to be becoming rarer.
Who do you address as sir or ma'am when you are off duty or a veteran?