CI: skip armhf, x86 build on stable branches
Don't try to build for arches where we don't have a binary repository on stable branches.
Do not cherry pick to v21.03, because there the sections of the config were simply deleted (12f614e9). This patch is in master, so we don't have to do that when branching the release after v21.03.