The crew neck is the usual culprit. A crew neck is the neckline of an undershirt, and worn on its own it still reads as one. Open a jacket over it and the eye goes straight to bare collarbone.
Nothing about that garment is wrong. It is doing the job it was drawn for, which is sitting under something else.
Knit structure is the second failure. Flat single jersey has no surface to look at, so it leans entirely on colour, and a plain surface in a plain colour reads as plain. Texture is what buys a knit top the right to be worn alone.
Hems come third. A straight hem with side vents is drawn for a shirt to hang loose over jeans. Tuck it and the side seams gather at the waistband. Leave it out over tailored trousers and it cuts a line across the widest part of the hip.
Sheen is the last one, and the hardest to explain until you see two shirts side by side. Filament yarns bounce light back in one direction. Spun yarns scatter it. Under an office ceiling light, one looks like sportswear and the other does not.