What is wrong: The caps and coils in a passive xo don't "absorb current" or produce heat. They are not resistors, they are reactive devices and the end result of speaker level vs line level xo will be exactly the same. It might be best to think of it as removing the lower frequency load from the amp.
It is true the values of components used in line level are much lower, but the issue is the signal is much more susceptible to corruption vs speaker level. I suggest you try both and see for yourself. Passive components at line level can have major effects on the sound, much moreso vs speaker level. I've built my entire system, including tube preamp, amp and speakers... simply changing a single resistor, cap or trafo in the signal path can have major effects. Done at speaker level the effects are far less.
The damping factor may be higher if a passive xo is not in place but in many cases the overall result is not as ideal, especially if you are using a SET amp with high output impedance. Once DF is beyond a certain point increasing it has little effect. The passive xo is not only there to create a crossover slope, it is also there to shape response and create an even impedance for the amp to see. This is more important vs having a direct connection to the amp as it will allow for smoother frequency response from the system. Also, passive xo frequently contain step and notch filters that are intrinsic to the speaker and drivers used. An example is removing the breakup nodes of a ceramic cone driver using a notch filter and baffle step correction.
If you're trying to create a simple slope to remove bass from a single driver then a line level xo makes sense as it may simply require a smaller value coupling cap, so in this case you're actually reducing the complexity of the system as well as the overall parts count. This is good. OTOH, if you have a 3+ way speaker that uses hard-coned drivers with steep slopes a line level xo is a poor choice. If you have a multi-way traditional horn system the drivers typically benefit from impedance and response correction and a line level xo is a poor choice.
As far as you not having another amp for the woofers I thought you said you're looking into a mosfet amp for that? Of course you have to biamp if you want to remove bass duties from a SET amp, otherwise no bass!