Am I too stupid to use Pkg3?

I have been maintainig some packages since Julia 0.6. Now I am trying to develop under Julia 1.0.
Pkg3 requires me to learn how dev and activate work, how to generate dependecy files. They are a bit obscure to me. I don’t have any clue what kind of problem they have solved.

Sometimes, I can remain two versions of a same package at the same.It will cause dispatch failure. In appearance, the function exactly match the type, but after long time debuging, I finally realized they are from two different pathes.

I am pretty sure this is also introduced by some new machenism of Pkg3.

Recently, I frequently get freek out when I try to develop two packages at one time, one depends on another.
Whenever I develop one of them, the other will fall back to some non-local version.
After reading posts, I still can not get any clue.

It has wasted me a lot of time!

Could you please expain why Pkg3 need to be so complicated?

2 Likes

Am I too stupid to use Pkg3?

It’s possible, but I really doubt it! It’s different than before, but I actually think it’s pretty straightforward.

Well, dev just gets you the full git repo. If you’re planning on modifying code in a package, use dev. If you’re not, use add.

You actually don’t need to deal with activate if you don’t want to, and you’re ok with everything being in the same environment. If you were happy with pkg2, this should work fine for you. Just stay in the (v1.0) environment and don’t bother to activate.

No, don’t worry about this - it’s done automatically, and for now you should just use REQUIRE as you did before.

I think if you have them both dev’d in the same environment, tests will work locally, right? They might not when you push, but you’d expect to need the dependency to be updated before the depender anyway, right?

That sucks! Sorry to hear it. For me the new system has saved a huge amount of time. Be sure to watch the juliacon video, that completely changed the way I see it and helped clarify a bunch of stuff.

I actually think Stefan and Kristoffer have taken what could have been incredibly complicated and made it quite straightforward. It’s an adjustment from Pkg2 to be sure, but it enables a ton of really great stuff.

5 Likes

I share @1115 feelings with respect to Pkg3. I’m sure things are quite straight forward, but only if you understand the overall framework already. The Julia documentation makes a valiant effort to explain it all, but I’m still struggling with it. For example, what is the relationship between Pk3 and git? How do I push my project to github and keep it in sync? How/where do I publish my package so that others may find it?

I for one would find it helpful if there were detailed descriptions of very specific workflows, each starting with “Open the Julia REPL”.

What makes the current documentation less clear for me is that it describes multiple workflows in a single section. It is very tempting to combine the different workflows in a single description because there is so much overlap, but I’d prefer descriptions that take me by the hand step by step thru a particular task.

Here are some workflows for which I would find detailed stand-alone descriptions helpful.

  • create a new project, don’t care about packaging it.
  • create a new package, but keep it local.
  • create a new package, write it to github
  • create a new package, write it to github and publish it.
  • edit a local package.
  • update dependencies in a local package.
  • clone/checkout a package from github, edit it, push it to github.
  • publish package stored at github.
  • develop two packages at one time, one depending on the other.
15 Likes

Thanks for your very appropriate summary, workflow is super important. Would you please also post it as an issue in Julialang repo, so that this very useful suggestion can be seen by more developers?

Many of these have solutions scattered in various topics in this forum. Please try searching first, then ask questions (possibly opening another topic). You will almost surely get help.

I expect that Pkg3 will be documented thoroughly, but it takes time. I found that its maintainers are open to seeing issues that ask for clarification of documentation/workflows, so if you are confused, please study the existing issues and perhaps open a new one.

I did have struggles with Pkg3 myself, and I am still having open issues with it, but it is an amazing improvement over Pkg2, and should simplify (or even enable) a lot of workflows in the long run.

I think a second video tutorial covering all common workflows would be greatly appreciated.

2 Likes

We need a really, really, really, really good tutorial for Pkg3. Although I like the documentation, it might a bit too sparse for newcomers. If one is used to package environments, it’s fairly easy to grasp the new system, otherwise I can imagine that it’s a bit tough.

And yes, Pkg3 is one of the best package development systems I’ve ever worked with (from Python, Ruby, JavaScript to Haskell etc.).

2 Likes

Traditional wisdom in the world of technical documentation has it that you think of three main categories of information for a product:

  • Training: hand-held, guided, on-rails and training-wheels exploration
  • User: task/topic-based instructions
  • Reference: concepts and components organized for lookup

For example, if you want to speak a new language (Chinese or Italian, say), you can imagine interactive oral and written practice (Training), phrasebooks and discussions (User), and dictionaries and grammars (Reference).

“T” and “R” material is usually in plentiful supply, “U” less so, particularly at an early stage of a project. The development process often produces “R” material as part of the design and specification process, and “T” material is relatively easy to produce by early adopters, and by developers if they have time. It can be problematic if documentation blurs the boundaries of these categories too much, jumping from conceptual overviews to lists of available functions, depending on the author.

Problems arise when, in the heat of battle, in the face of rapidly approaching deadlines, users turn to the documentation, and find either simplistic “T” material which doesn’t describe their problem, or “R” material which doesn’t quickly provide the answers they seek. (“Congratulations on purchasing this fire extinguisher. Please read the following chapters carefully before attempting to put out a real fire…” is not what you want to read when you’re on fire.)

And of course, if someone provides task-based instructions for every single scenario, you could have problems finding the one you want.

Descriptions of common workflows is always a good thing to have, though, not just for package management, but for package development, and debugging. Currently, the Julia manual has plenty of room for them, but I’d hope to see better search tools if the documentation gets very much larger.

9 Likes

Maybe I’ve been doing things wrong but I sometimes had to activate my packages and add dependencies (as the docs tell you), otherwise tests would fail.

I am using this topic because it seems like I am too stupid also: I just created this minimal package for which I am trying to setup everything correctly. I enabled travis but it says No builds for this repository even after I do new commits… Could it be something wrong with the travis file? Sorry and thanks for your help

Its easy to use it to do what you want to do when you know how to use it to do what you want it to do. It should be easier to use without knowing how to use it. The peoples ^ they are too un-stupid wanting Pkg3 to be of use with easier ease.

Use this thread to provide three or four multitransition state/event/action sequences, each a canonical utilization of package management and the time of GitHub and local gitting and gitgoinging. Give this back to the Community and the Community will give its attention to this [that has been my experience].

I think you need an

os:
  - linux

line in the travis file, but I am not totally sure. You may want to check travis logs, also AFAIK travis has some functionality to check its .travis.yml.

Perhaps start from an existing working package and remove things until it breaks :wink:

A “project” is different than a package. The project is the environment. If you don’t need different versions of dependencies for different things you’re developing, don’t bother with projects and activate. Just stick with the default (v1.0)

I’m talking about package yes, and packages are projects, so you can’t really ignore them.

Package: a project which provides reusable functionality that can be used by other Julia projects via import X or using X . A package should have a project file with a uuid entry giving its package UUID. This UUID is used to identify the package in projects that depend on it.

I added the os section but still no build detected. I thought it was better from the scratch with little additions to understand what does what and learn the required steps, but perhaps I should consider doing the contrary :confused:

EDIT: it works now, guess I had to wait a bit :slight_smile:

My understanding is that that’s where things are going, and eventually there will be tooling to help with it, but for now you can just develop with a REQUIRE file as before. Someone can please correct me if I’m wrong, but this is what I’ve been doing and it works fine.

1 Like

If you create a new package with generate (which is what the docs tell you to do), add a test with using Test, then tests won’t work until you add Test as a dependency (REQUIRE doesn’t help). Maybe there’s another way to create a package though. Interestingly you can’t add the package you just created too (complains about the absence of git).

A default .travis.yml would be nice too, there’s a lot of confusion about that too.

Interesting. I’ve never started with generate, I’ve just copied the structure of other packages. I’m guessing that generate makes a Project.toml and Manifest.toml, which IIRC is incompatible with REQUIRE.

I think your best bet would be copying the structure of some other packages (or the packages you had before 1.0) until the full tooling for managing projects is ready. It’s definitely irritating that the docs tell you to do something else - but my guess is that this will get itoned out very soon. And the system we’ll have is much preferable to the one we had before (even if there are some wrinkles at the moment).

It’s not that they’re incompatible, but that REQUIRE is ignored should a Project.toml exist.

The reason it’s not as smooth sailing as it could be right now is because PkgDev.jl, which provided a lot of the development side of packages pre-0.7, has not been fully adjusted to 0.7/1.0 yet. As far as I’m aware, this is the PR in question. Another reason is that this is just the first iteration of Pkg and what can be done with it - I’m sure there will be a few additions in the future to make adding test-only dependencies easier.

1 Like

On top of what JonathanBieler said, I recall reading that this was fine, because there’ll be a PR to all registered packages.

None of mine are registered, and it’ll be a while before I have the time to polish any of them enough to get them registered.