Add removeReferencesTo for removing specific refs

This allows for a less blanket approach than nuke-refs, targetting specific
references that we know we don't want rather than all references that we don't
know we want.