If you redirect stdin (that’s what the pipe does), then there isn’t much point in starting up the REPL, as it would also read from stdin, which is closed by this point.
However, you can run a script and then keep julia running, as follows:
$ julia --banner=no -ie 'println("my initial command which I would like to show up")'
my initial command which I would like to show up
julia>
Note the -i switch, which forces Julia REPL into interactive mode.
HTH.