When an E-4 calls me "hey" or "hey man" knowing that I am an officer, should I bother to correct him?

Oh Hell Yeah.
While it is OK to be friendly, the discipline and respect must be maintained.

You don't have to hit this too hard, but, the next time he does it take him aside and say "it's either sir or (insert your rank here), even if you are off duty".

OUR ARMED FORCES ARE ALLWAYS PREPARING FOR WAR-TIME CONDITIONS. WHEN THE SOLDIER IN CHARGE ISSUES AN ORDER TO A SUBORDINATE DURING COMBAT, NO SLOW, LAX OR INFORMAL RESPONSE CAN BE TOLERATED. BATTLES, WARS AND LIVES ARE AT STAKE.

I would say depends on context and situation. I worked on a flightline without my top on alot, especially while turning wrenches. I saw alot of faces and didn't know everyone's rank.

CAPTAIN.?
Had I Been You, I;d Probably Have To Kick'Him Through The Goal Posts Of Life.
THEN Stomp His-az

It's your duty to uphold good order and if I heard a E-4 calling a officer hey man and the officer did nothing . I would be chewing you both out .

Unless respect for your rank means nothing.

If the SPC knows that your are an officer than absolutely, he is disrespecting you and your rank.

This is a potential storm in a glass of water. It is outside of work and he showed you respect in front of others. You can ask him politely to address you in a different manner outside of work. But the key here is to show that E-4 why you are an officer. You both are service members 24/7. If you demand respect, you do not belong in the officer corps and if you don't correct him, you don't belong in the officer corps.

That would never ever happen in the USMC.