A common tactic among those who find themselves losing an argument with a vegan is to attempt to undermine their opponent's moral authority by asking, "Who are you to judge?!" In this way, they hope to put an abrupt end to what may be an uncomfortable conversation, or, failing that, at least to change the subject. And though the tactic itself generally amounts to nothing more than a facile and transparent diversion, the question is worth treating seriously and should begin by defining the key term, “to judge”. Thankfully, that work has already been done for us! As it happens, there's an entire industry devoted to defining words. The Oxford Dictionary of American English defines “to judge” simply as, “to form an opinion or conclusion about”. So, the original question could be re-stated as, “Who are you to form an opinion?!” to which one reply might be, “Who are you to decide who gets to form an opinion?” But, as is often the case, taking the question at face value can be a more effective means of exposing the vacuousness of your interlocutor's position.
Who am I to judge, you ask? In short, I am a moral agent and a rational human-being capable of making the important distinction between right and wrong, good and bad, selfish and benevolent. Judgement forms the basis of all legal systems and all codes of morality and yet, the words "Don't judge me!" have become the first resort of many people who don't want their behavior questioned by others. But certainly, the majority of people wouldn't hesitate to "judge" a person who, for instance, abused puppies or poisoned children. It is nothing if not judgmental to say that abusing puppies and poisoning children are bad and if any person guilty of such behavior had nothing to say in his own defense but "Don't judge me!" we would rightly dismiss his insistence that we are not entitled to assign a moral value to his actions. Of course we are. And the same applies to those who support the exploitation, oppression, and slaughter of animals. I have as much right to assign a moral value to that abominable practice as you have to assign a moral value to abusing puppies and poisoning children, or, for that matter, that anyone has to assign a moral value to feeding the poor and aiding the sick. (It's funny how no one ever objects to being “judged” when the “judgement” is favorable.)
Starting from first principles, if we accept as axiomatic (as I think we must) that oppression, exploitation, and cruelty are bad, then it naturally and inevitably follows that we shouldn't support or perpetuate these practices. And if we allow the words “Don't judge me!” and "Who are you to judge?" to provide a moral safe haven for those who choose to support the oppression and exploitation of, and cruelty to the world's billions of non-human animals used for food, clothing, experimentation, and entertainment, then we must grant the same impunity to those who abuse puppies, poison children, or for that matter, commit any other atrocity whatsoever. To grant this impunity would be to surrender to the worst kind of anarchy and moral cowardice. We could establish and enforce no laws and impose no standard of behavior on society. That is not a world in which I care to live, regardless of who feels "judged" as a result. There are many things much more important than not upsetting other people and two of those things are speaking on behalf of the innocent and defenseless and calling things by their true names.
Your argument (in my judgment) is invalid.