When an E-4 calls me "hey" or "hey man" knowing that I am an officer, should I bother to correct him?

If you have to respect his rank he should respect yours.

We know that "Sir" is the correct way to have addressed you. If it is a Very casual situation, I will sometimes address an officer as "L.T." or "Captain", to still acknowledge my respect for his position. If he wants me to formally address him "by the Book", then I just shift gears, comply, and carry on (but it's never been an issue).

Depending where the, "outside work" was ... on or off base ... would determine how I'd handle such a casual attituded. AS AN NCO, THIS WOULD BE MY REACTION: ON base ... whether "outside work," or not, it would be on-the-spot; but casual. "Specialist, 'hey man,' isn't appropriate. Be careful who you use such a casual attitude with." Depending on his reaction (or her's?), would determine if any additional correction would be required.
OFF base ... I'd wait until we were back on the job and I'd say pretty much the same thing.
All my guys/gals knew that, "Hey Sarge," was appropriate; and fellow NCO's would be, "Hey Mac." I figure youthful casualness like that isn't necessarily disrespect, BUT id does require a subtle response. To be that casual WITH AN OFFICER though is asking for more of a comeback. In the Navy, an officer (Physician) was chewing my butt and I said, NOT intending it to be TO him, "Oh man ...." and wow ... my butt SITLL burns! LOL Taught me a good lesson about what you say in an officer's presence, or more specifically, TO an officer.

Why is this even a question. Hell yes you correct that SPC. It does not matter that you are in town and out of uniform. If you do not want to create a scene then pull him/her off to the side and chew that ass..but for god sakes man do not let it go.

Yes