As per here, you should be able to disable that with the python.terminal.activateEnvironment setting:
Tip : To prevent automatic activation of a selected environment, add
"python.terminal.activateEnvironment": falseto yoursettings.jsonfile (it can be placed anywhere as a sibling to the existing settings).