What will you call the teacher who teaches you English - English teacher or teacher of English?
You may say that both are correct. When you say so, I definitely side with you. But while teacher of English creates no ambiguity, English teacher is ambiguous. An English teacher may either be a teacher who teaches English or a teacher who is an Englishman or woman.
You should always try to avoid the uses of ambiguous words.