I suspect disabling memory overcommit would break other applications that depend on it, but it’s worth a try if it’s critical to avoiding OOMs for this specific application.
Note: I’m not an expert with cgroups, so this is based on how I assume cgroups work:
I don’t think a cgroup would save you here; a memory cgroup only says “invoke the OOM killer if the applications in the container exceed N bytes of memory allocated”, but it doesn’t then protect those applications from being reaped if an application outside of the cgroup triggers the OOM killer, or if the kernel somehow allocates a lot of memory.