When an E-4 calls me "hey" or "hey man" knowing that I am an officer, should I bother to correct him?

HEY MAN. F#&$k no !!!. You a Officer not a lower enlisted Joe, Like him.
Regardless of infractions correction's most be made.
Naturally the degree of the infarction should reflect the degree of of the correction's.

I never did that to any one with a higher rank than me.
Even I was friends with the Higher NCO or Officer.
The most casual greeting. Was hi Boss. Not hey man.
This is what is wrong at the base level no respect for the simplest of protocol or rank.
The other side of this is NCO'S ,And some officers do not hold the line because they are friends or they are afraid of the Political bullshit.
Regardless of the causes. Rules and regulations are there for a reason.
No order in the ranks.

As an enlisted man I say yes you do correct him, respect is still deserved no matter whether you're in uniform or not.

You must set this soldier straight. As he will most likely do it to other’s. Be the example to him. Be the leader that you are and should be…

Is this even a question? A soldier is a soldier 24/7; in or out of uniform. Respect is a value that a soldier needs to live by. However, if he does not respect you he still is required to show respect to your rank. Maybe you should discuss this with the First Sergeant. Ask him how he would handle an E4 calling him Hey, man?
SGM Francis J Brzyski (ret)

Your a soldier 24/7 and as such military customs and courtesies should be used in and out of uniform. Do you have to salute while in civilian attire? No. But a soldier should always address a officer as sir. Behavior such as this becomes accepted behavior and then normal behavior and therefore deteriorates the standard.

Let that slip by Sir and what does he ignore next? Sleeping on guard duty? Ignore an order during a firefight? The soldier you correct today could be the one that saves lives later. Not to sound off the wrong way but apparently your NCO's are failing you if you even asked this question. I would love to know if your 1SG oe CSM knew one of the Officers in the unit had to even ask this.

YES. Disrespect cannot be allowed in the military. And respect must be a two way street.

Absolutely and on the spot. By doing nothing you're telling them and everyone waiting this is acceptable behavior, setting the standard. Moral courage can be a rough situation; however, by setting standards you'll earn respect from the individual and those watching. Look at it this way, you're helping the individual by preventing future behavior where they may run across the wrong person ending their career so you're doing a good thing by educating them now.

Obviously he did not know your name or rank and wanted to get your attention. Ask a higher ranking officer you know.

You have to keep the standard. Have a talk with him. Explain the situation, if he knows enough to call you sir while in uniform, he will understand. I bowled with a friend who happened to be be an NCO. When we were bowling and in social non military occasions it was first name basis, however any time military were present (even in a social situation) I was Sir or Colonel. Not everyone can do that.