[Ann][Tutorial] How-to develop a Julia package from scratch

Community
#1

As part of my efforts to update the Julia Concise Tutorial, I completely rewritten the chapter dedicated to “How to develop a Julia Package”.

Yes, sure we have the “Creating Packages” section in the documentation… but that’s only about the package generation (and testing).

I have long had the feeling that something was missing about the complete typical workflow required to develop a Julia package… from choosing the name, setting up a GitHub repository, generate the package, put everything together, add automatic tests, add the configurations for the documentation, deal with github actions as CompactHelper and TagBot and finally register the package.

For each step I propose in the tutorial the steps required for a “typical” package installation, but I link to the relevant documentation for users that need more options.

Following the tutorial a user with little experience should be able to develop a simple package on the lines of MyAwesomePackage.

(PS… I am particularly proud of my plusTwo() function… too often the examples we make to explain some topic are unnecessarily complex)

49 Likes
Package development - guide required for good resources
#2

I think this is extremely useful. A guide that shows a way from a to z in setting up and developing a package is really needed. I think all the bits and pieces are already documented all around but this consolidated view is really great. I’ll be sending this along to budding julia package developers in my network. :pray:t2:

1 Like
#3

I think that it is best to use a package template generator, eg

Various aspects of the tooling (CI, Tagbot, …) change from time to time, detailed guides tend to become outdated quickly.

8 Likes
#4

There is also DrWatson.jl

2 Likes
#5

…ny…

For the same reasons package generation tools get outdated too, in my case when I needed it a few months ago for example they didn’t work because of some changes… and if that is the case you are “dead”… With a good tutorial (that by no chance I claim mine to be) instead, you should get at least a feeling of how things work and you may able to adapt to the changed situation… And then, contribute back to a markdown page is easier than contribute to correct the tool…

Still, package generation tools are great if you need to create many packages …

3 Likes
#6

This tutorial is very helpful, thank you! I definitely fit the bill of “a user with little experience”

It was not always clear to me which directories I should be in. Here are my thoughts/questions as I was going through

We cd to a directory where we want the new package to appear as a subfolder, enter the Julia prompt and type (@v1.X) pkg> generate MyAwesomePackage

Ok, so … I start making the package on my desktop for now. Looks like Pkg.generate also put a file in the .julia/dev folder though

We first “activate” the new MyAwesomePackage folder with (@v1.X) pkg> activate(FULL_PATH/MyAwesomePackage)

I guess FULL_PATH refers to my desktop, not the .julia/dev copy

Let leave Julia for now and go to the local repository with a terminal

Which one? On my desktop or in .julia/dev?

Once we have the package in GitHub, we can add it to our Julia local installation with (@v1.X) pkg> add https://github.com/sylvaticus/MyAwesomePackage.jl.git and (@v1.X) pkg> dev MyAwesomePackage. The package should now reside to [USER_HOME_FOLDER]/.julia/dev/MyAwesomePackage

What does it mean, reside? Do I edit this copy, or the one that I have git linked to on my Desktop? What’s the relationship between the two?

I personally would also have benefited from another section, maybe “Developing” somewhere between “Creating” and “Documenting”, which showed the typical workflow of changing the package. Starting from a blank slate, I guess I pull it from github, then make changes, commit and push back to github. (Maybe more of a github tutorial point?)

2 Likes
#7

Hello, if you are on your Desktop, enter Julia and type ] generate Foo, than Foo shoudn’t have any entry in .julia/dev, just one folder in Desktop/Foo.

I added a few lines on continuous development of the package:

With out package in [USER_HOME_FOLDER]/.julia/dev/MyAwesomePackage/ we can continue to develop it, commit the changes and push them to GitHub (behind the scenes, the add and dev Julia package commands did actually cloned the GitHub repository to .julia/dev/MyAwesomePackage/, so we can use directly git commands on that directory, like git commit -a -m "Commit message" or git push).

Further development of the package

In the most basic situation, we would keep the main branch locally, continue our development, and when we consider it is “time” to release a new version we just change the version information in the package Project.toml, we commit/push the new “version” and we raise the @JuliaRegistrator register “command” again in GitHub.
This will have the effect to register the new version in the Julia official Register, with TagBot creating an equivalent version on the GitHub repository.

1 Like
#8

I followed the guide and created manually the directory “.github/workflow” and the files “ci.yml” and “CompatHelper.yml”. However, after I pushed it to github, none of the badges I added to README.md is working.

Was I supposed to add the actions in Github and not locally?

#9

Ok. I deleted the “.github/workflow” directory that I manually created and set up the workflows directly from Github. Now it works.

If the actions are really supposed to be added from Github, may be you should emphasize it in your guide.
Also, you might want to add how to setup Documenter SSH KEY.

#10

Hello, no, you can but you don’t need to set up the actions in GitHub, adding the workflow files in the repository should suffix.
Did you git add [file] before making the commit?
After you set the workflow file locally, commit it and push it to the repository, you should be able to check for any error into the “Actions” tab of your repository in GitHub.

Concerning the setup of the Documenter SSH key, this is explained in the tutorial, but I guess in a too much concise way.
I will add to that, thank you.

#11

Maybe the CI contained branches: - master, while new GitHub repositories now default to a brach called main instead of master.

#12

No, this change was accounted for in the guide. Moreover, I just copied to github the content of the same files I tried to push from my local machine. May there are some special git read-write rights that need to be set up correctly?

#13

I did. I saw the workflow folder with files on Github after the push. Nevertheless, thank you for your response.

I have another question related to the topic of package development. Is it possible to update the docs of tagged version without changing the tag?

#14

I’ve never had that problem before because normal rights are fine. Other things I can think of are that you forked a repository and didn’t click on “I understand the consequences, enable Actions” or didn’t put the file at the exact correct location.

When the yml file is located correctly, GitHub will let you know if the syntax is false, except when the job just doesn’t run as I suggested in my previous comment. If you really want to debug this, please send a link to your GitHub project and the failing commit.

#15

I think I do not really want to do it. In the end, I made it working, which is fine with me.
I appreciate your responses, hope you are not irritated that I decided not to push this forward.

#16

Oh, I assumed you did. Okay, no worries. :+1:

#17

Encountered the same problem; the correct directory is .github/workflows/..., not .github/workflow/.... Once I changed this, it started working.

1 Like
#18

Corrected, thanks. In these cases the best would be to make a pull request directly in the tutorial repository.

#19

I would have gotten to it eventually; I realized what the fix is mid-work on a project, lieterally minutes before I wrote that comment.

1 Like