Security concerns with walletpassphrase

I have some security concerns with walletpassphrase:

If it stores the passphrase for some unit of time. ie. so you can send funds, surely if the server is compromised, within that timeframe, the attacker can obtain the passphrase in its raw format? If I have somehow misunderstood, ie. perhaps the raw passphrase isn't easily accessible (how?), an attacker could still empty the wallet within that timeframe seeing as the wallet is in an unencrypted state.

All roads are leading me to use bitcoin libraries that allow me to create transactions and sign them using encrypted private keys that are only unencryptable at the time of signing. I feel way more comfortable using my node for transaction broadcasting / indexing capabilities. Rather than leveraging its in-built wallet.