I want to convert my project code (structured like package, i.e. has /src, /test, Project.toml, Manifest.toml…) to real package. The issue is that in /src it has also ampl code so it is not full Julia. The Julia code has functions that call ampl code using run() command and ampl saves results in workdir/dump folder. All of this works fine as project. It can be run using setup.jl that has instantiate(".") command and test runs by loading the function definitions and then calling them with fixed arguments. Overall I have no errors with this. However, the code execution (cold start) is slow due to missing precompile that can (to my knowledge) only be done if the code is loaded as package that has PrecompileTools precompile workload command. Warm start execution time is more than acceptable, but cold start is not very good. My question is how can I make a package with external, non-Julia code? I have read that artifacts could be used in this way to load non-Julia code, however, I don’t want ampl code to be public. I am okay that the package is installed using dev or by using private repository.
Well, you can always create a private package and keep it in a private repository.
With which step do you have a problem?
If you want to share it with colleagues that have access to your private repo they can always do:
using Pkg
pkg"add https://github.com/ufechner7/KiteUtils.jl"
where you replace “ufechner7/KiteUtils.jl” with your user name and the name of the package repository, which should have the .jl suffix.
No need to register your package.
You can have non-julia source code in your package, but then your colleagues would need to have the correct compiler installed locally.
You can also have non-julia binaries in your package, but then it will work only for the operating system that these binaries have been compiled for.
Thank you for the reply. Having all required compilers is not an issue. Also operating system is fixed so that too is not an issue.
I don’t have full understanding how Julia installs packages. Does it copy all files or only evaluated definitions? With definitions approach I think it would not work as run command then would not find ampl code as it would not be copied or installed as part of the Pkg installation procedure. I can try it and report to you back, but my assumption is that it does not just simply copy files to different location.
Packages are installed under
.julia/packages/<MyPackage>
Example:
└── zZm2V
├── bin
│ └── run_julia
├── CHANGELOG.md
├── data
│ ├── kite.obj
│ ├── settings.yaml
│ ├── system.yaml
│ └── Test_flight.arrow
├── docs
│ ├── make.jl
│ ├── Project.toml
│ └── src
│ ├── examples.md
│ ├── functions.md
│ ├── index.md
│ ├── kite_power_tools.png
│ ├── reference_frames.md
│ ├── small_earth.png
│ └── types.md
├── LICENSE
├── Project.toml
├── README.md
├── src
│ ├── KiteUtils.jl
│ ├── logger.jl
│ ├── settings.jl
│ ├── trafo.jl
│ ├── transformations.jl
│ └── yaml_utils.jl
└── test
├── bench.jl
├── cov.jl
└── runtests.jl
There will be one subfolder per installed version of your package.
But, as you can see, all files are installed, not just the Julia files. I have binary files in the bin subfolder of my package.
The only thing to take into account is that you should treat files in the package directory as read-only.
I also have a Julia functioninstall_examples() that copies the examples into a local example folder so that they can be edited. So you are free to copy files to a different location if you prefer a different location, but then you have to provide the install() function yourself.
And it is easy to find out the installation directory of a package:
julia> using KiteUtils
julia> pkgdir(KiteUtils)
"/home/ufechner/.julia/packages/KiteUtils/uESkC"
Example of a copy_examples function:
"""
copy_examples()
Copy all example scripts to the folder "examples"
(it will be created if it doesn't exist).
"""
function copy_examples()
PATH = "examples"
if ! isdir(PATH)
mkdir(PATH)
end
src_path = joinpath(dirname(pathof(@__MODULE__)), "..", PATH)
copy_files("examples", readdir(src_path))
end
And the copy_files function:
function copy_files(relpath, files)
if ! isdir(relpath)
mkdir(relpath)
end
src_path = joinpath(dirname(pathof(@__MODULE__)), "..", relpath)
for file in files
cp(joinpath(src_path, file), joinpath(relpath, file), force=true)
chmod(joinpath(relpath, file), 0o774)
end
files
end
I explicitly make the files writable.
Thank you very much! This is of great help! I will mark it as solved, but I may return with few additional small questions.
I think I will probably need is to implement that run command calls file external path file relative to calling file the same way include command does. I assume this should be correct way to obtain path.
joinpath(dirname(@__FILE__),relpath)
If the files you need to write are temporary files, you can also use tempname to generate paths to temporary files. Then you don’t have to mess with the package directory.
This is correct and can be shortened to joinpath(@__DIR__, relpath).
You can use private artifacts with your private package, but it’s more work to set up than to have them inside the repository. (There are of course upsides as well; git repositories are not that great for storing large binaries.)
If you go the route of only having the source code in the repository and building it locally, it’s a better idea to do that in a Scratch space than in the package directory.
If you only have one private package to deal with it’s easy enough to just use the repository URL. Should it grow to multiple packages with dependencies between them, you may want to look at setting up your own registry for your packages. The LocalRegistry package can help with that.
Yes. Thank you. @__DIR__ works better.
Created files are needed for GUI so I modified package so the workdir can be any (does not need to be root of the package) and it creates simple “dump” folder within workdir so it is easily available for the frontend. That is basically the output of the Julia function.
Ampl files are simple text so from what you say it seems artifacts are not needed.
I am still working on this. I have significant progress, that is that the code does not have to have workdir at the root of the project/package thanks to @__DIR__ macro and related functions. From the high level perspective, the code is a function that should make folder in the workdir with optimization results from AMPL. The input of the function is the address of user supplied JSON.
The goal is to make a package that exports the function so that precompilation could be achieved. Evey package needs proper Test so I study this at the moment. I looked into how other packages did this, namely PowerModels. I am confused by paths in test dir used there. Functions load data using “…/test/path”. This would imply that src is the workdir in that case even though test command can be called from anywhere. I don’t find this documented anywhere. Why would src dir be workdir for pkg test command? @ufechner7
It isn’t. The workdir for the test comand is the test folder, the folder that contains the runtests.jl script.
It works! Tests pass.
Do you know what would be workdir during running @compile_workload from main module file? My guess would be either root or root/src?
Slightly off topic: but why use AMPL instead of JuMP?
@compile_workload uses current workdir, not the package root or root/src. I made it work. It reduced Julia time segment for more than factor 10 (14s->1s). Thank you all for this!! Next step is exploring options from PackageCompiler: Julia system image and Julia “app”.
@odow I tested thoroughly both JuMP and Examodels for NLP optimization (both CPU and GPU compute). I will emphasise CPU computation here as GPU version is known to have impaired convergence. Despite the CPU optimization, difficult OPF instances (the ones that need soft constraints to have a solution), have much worse convergence properties in Julia. I explored why could that be. I found that the model in Julia has about x5 times as many non zeroes in Jacobian and Hessian. I tried to isolate the behaviour and found out that Julia creates full triangle Hessian matrix for a constraint despite some of the second derivates being fixed zero, e.g. there is no V_[m]^2 in the polar AC power flow equation, but Julia generates non zero for it. Similar for shunts where one can model conductance which is frequently zero: g*V^2 which in case where g is 0 should have fixed zero in Jacobian and Hessian, but in Julia it is not. On the contrary AMPL handles this nicely. My uttermost importance is having reliable solution. For difficult cases AMPL less frequently converged to local optimal solution (more frequently to global) and needed less iterations to reach the solution leading to sometimes faster solution too (even compared to direct coding to Examodels api). That is why I use Julia packages to obtain data and AMPL for optimization where license restrictions allow. Please note that I did the comparison with AMPL presolve set to off as I felt that could have been the advantage. AMPL presolve improved further a bit the case for AMPL. Also with AMPL I don’t have significant overheads as with Julia. I sure would like if Julia team could fix the described issue with NLP.
Overall I took great care to set up initial point to the same values (warmstart) in both AMPL and Julia, but I quit when I figured that non zeroes can not be the same.
Can you post a new thread with the opf issue? We can’t fix things we dont know about 