As a mother, do I have the right to deny a person to have contact with my daughter?

You are a great mother. Find out through Social Services who can help you with this situation. Do not let the child's father talk to you that way. His only concern should be his child. He should not have anyone with him that you do not approve of. How dare he. Do not let him stress you. I'm getting mad just thinking about what he said! Take good care.