Yes, it would be better to not have him coach you.
If he keeps coaching you, you are going to become more and more entranced by what he does.
You may reach a point where you start telling yourself that it may not be such a bad idea to entertain a relationship with him.
However, any kind of relationship with him would only be physical or casual in nature.
Yes, you are making sense as far as not wanting to give him hope through continuing your workouts with him.
Indeed, if you were to continue your workouts with him, he may use those sessions as his opportunity to keep making moves on you.
Again, giving in to his advances would only lead to a long distance relationship that is casual in nature.
This is the kind of relationship that you are not looking for.
Hence, it would be wise to not have him as your coach any longer.