emacs-irony: Remove server directory

This contains the build files and is not needed for irony-server.
Previously, the wrong directory was removed, adding elpa/ gets thje
correct one.