After some generous help on this forum we have solved a very chunky multicommodity network flow problem. The question we have now is that there is some discrepancy with our “backcast” using optimal flow and what is recorded for particular commodity flows. One feature of the CGE literature in economics is estimating Armington factors to show a fixed multiplier for preferences of demand notes for output from supply nodes. The classic example here is “people from country X like wheat from country X” or the like.
The issue is: if you have some target end flows (nodes in country X → country Y) you can verify with trade data, what is the best way to backsolve for fixed factors or multipliers that will lead the optimal mincost flow to match the trade data? Is there a good way to wrap a gridsearchCV or the like around a JuMP problem? Has anyone done this before?
One way is to minimize some error function by gradient descent, for which you have to backpropagate gradients through the nework flow solver. If your flow is convex, DiffOpt.jl will get the job done. If your flow is linear, the issue is that those gradients are ill-defined, which requires some smoothing tricks. The best solution is to use our InferOpt.jl package to minimize the Fenchel-Young loss between your solutions and the targets. If you need help getting this up and running, you can ask me or @BatyLeo!
What trade data are you using? There are quite a lot of discrepancies in some of the trade data, which may prevent your output from being verified based on this data. You can refer to this work for further information:
Chen, C., Jiang, Z., Li, N., Wang, H., Wang, P., Zhang, Z., … & Chen, W. Q. (2022). Advancing UN Comtrade for physical trade flow analysis: review of data quality issues and solutions. Resources, Conservation and Recycling , 186 , 106526.
Very expensive AIS ship tracking and commercial data. I don’t think the data is the problem as such, model already very close to historical the adjustments are small (<$4 per tonne) but the parameter space is large (50 country to country relationships x however many increments).