Hi, did you read
https://fluxml.ai/Flux.jl/stable/models/advanced/#Custom-multiple-input-or-output-layer,
especially the latter part about a custom split function?
You should be able to split the shared body into task-specific paths pretty easily this way. And if you modify the last listing on that page a bit, I guss you can use a different loss for each comppnent of the resulting tuple. You just need to combine it somehow to get a single number out.