If you think those in power will blithely allow their position to wither away without lashing out with violence, then the voluntarist movement ought to surrender their rights immediately. The capacity to deliver death and destruction is the only basis of power in this world. The social, political, religious/moral authority are but collective lies and delusions used to hide the truth regarding the real legitimacy, upon which authority derives.
Sovereign individuals derive their autonomy from their ability to inflict pain, upon those who would wrest away their freedom. Without the will to kill, voluntary interaction is not possible, since voluntary interaction occurs between equals, not between master and slave. A slave has no will to kill, even to live; and would-be masters will kill, even for trivial reasons. Only killers can choose pacifism and fair exchage; others are condemned to it. The rights and freedom of individuals must be wrested from the clutches of the government and social institutions that seek to enslave men, by any means necessary.
Only those worthy of freedom can exercise their rights; the unworthy muck, readily surrenders their freedom, content to allow others to dictate their rights to them. In order to create a voluntarist society of sovereign individuals, the unworthy must be cleansed, along with those that would restrict the rights of man. For freedom, everything, even morality, must be sacrificed.