CI: test_unreferenced_files: skip gcc only

Change it to skip gcc only and clarify why this was done for future reference. From the discussion in !3541 (merged).