As per here, you should be able to disable that with the python.terminal.activateEnvironment
setting:
Tip : To prevent automatic activation of a selected environment, add
"python.terminal.activateEnvironment": false
to yoursettings.json
file (it can be placed anywhere as a sibling to the existing settings).