Update! I believe I found the solution to this problem courtesy of @heetbeet! It seems really great and will report back my work with it. 
Here is the solution: GitHub - heetbeet/juliawin: A portable Julia for Windows, bundled with VSCode, Pluto, Conda & PyCall, and more.
Update: Updating the JULIA_DEPOT_PATH is also important to do for full portability: Change location of .julia