glib: fix compilation on SmartOS/Illumos (close #12477)

Recent illumos includes a linux-incompatible `inotify.h` header, which configure detects: compilation fails.
Also, a newer `dtrace` on SmartOS fails creating the probes ELF linkable object (with `dtrace -G`). Disable for now.

Remove old configure option `--disable-modular-tests`.