Revert "gcc, binutils: Get rid of 32-bit ARM configure flag exception"

This commit breaks native armv7l-linux builds. Revert it until it can
be root-caused. This reversion does not affect other platforms or
cross-compiling.

This reverts commit 0f5c8046.