You could define an additional binding that sends unicode characters to the terminal, e.g.:
{
"key": "cmd+backspace",
"command": "workbench.action.terminal.sendSequence",
"when": "terminalFocus",
"args": {"text":"\u0015"} // ^u
},
Then, the binding for the editor should have the when clause "when": "!terminalFocus"
.
I know this is a bit awkward because you need to define everything twice…