I totally agree as well. In fact yesterday I asked the same question of many members here. Suffice it to say if they stay they will be turned off.
actually; it works opposite of what you are saying. a speaker cone sitting in a room (in or out of a cabinet) is a passive radiator and will absorb certain frequencies. so you want to remove them as they act as absorbers (unless you test for it and prefer the effect).
OTOH when you power the cone it stiffens and is less likely to change the sound as it has resistance. so if you leave the subs in the room power them up but turn them down. they will then have minimal effect. for zero effect remove them.
it's possible that a heavy cone like a sub won't move much unpowered, but i'm getting into areas I don't know about. it could be a variable.