swivels

Hey Guys; Wondering about removing the stock front swivel and using the lug for a quick release swivel from Uncle Mikes. Mine is the tactical adjustable/folding stock. Can I just pry the ends of the stock swivel out of the lug or will that damage the stock/lug.