xorg-server: fixup build after #29039

The expression uncarefully overrode whole nativeBuildInputs,
so after the pkgconfig move it got removed and the build failed.
1.19.3 doesn't need these lines anyway...