Should be able to remove added ProjectReferences by specifying --undo argument.
ProjectReference
--undo