v9: automatic import of NuGet packages into the Our Umbraco package listing

[nul800sebastiaan]

With Umbraco v9 moving to NuGet packages we would love to make it a bit easier for package developers to manage package metadata on Our Umbraco and make sure their packages get listed on Our Umbraco.

Why have Our Umbraco package listings

The NuGet package format is quite restrictive and doesn't allow for adding much information. On Our Umbraco we have plenty of fields to help site builders decide whether they want to use the package. We also have a way to list who collaborated on the package and we have a compatibility reporter. The package developer gets to write rich-text release notes, they can add documentation files and screenshots and they can choose to make a package-specific forum. All features not available on NuGet.org.

How to list your v9+ package on Our Umbraco

First of all, since last week, the way to mark a package as compatible with v9+ of Umbraco is now available on Our Umbraco as a checkbox on the package:

It requires you to enter a NuGet URL as well if you enable that checkbox. Once that is done, your package will be listed in the v9 backoffice and once selected, the NuGet install instructions will be listed.

Note: if a package happens to be compatible with v9 and v8, then that will still work as well. If there's files uploaded to the package that are marked for v8, the package still shows up in the packages section in v8 backoffices.

Automatic import of NuGet packages into Our

We've had a few ideas around this and we'd like to get some feedback on how this could be done.

Currently we think the following could be very nice:

• Package developer uploads their package to NuGet.org
• We scan NuGet periodically for new packages/versions of packages
• If we see a package that is not yet published on Our Umbraco then we create it automatically
• The package author has a way of claiming the package to add more metadata

Finding Umbraco v9+ packages on NuGet and version compatibility

The way we find packages would be to look for tags added in the NuGet metadata. Currently we have a proof of concept where we look for all packages tagged with umbraco, package and plugin.

This currently results in a small list:

Clearly, there is some packages in there that we don't want to list for v9 as they are marked for v7! 🙈

We have thought about going down the dependency chains to see if there's a dependency on any of the UmbracoCms v9 packages, but that would excluded some packages that are purely going in App_Plugins (the packages that don't contain any dlls).
Adding a version tag to the package also seems a little restrictive, what if you don't have any updates, v10 comes out and your package still works perfectly. You would be forced to upload a new version just so you can add a v10 tag.

👉 We'd love to get some thoughts on tags that would be helpful!

Importing to Our Umbraco and claiming ownership

Once we've solved the challenge above we can start creating packages automatically in Our! But of course we don't really know who the owner of the package is.

The thought here is that if we know the member id then we can easily import any new package and attach it to the member who created it. They can then go in immediately and fill in the details, add a more extensive description, add a few screenshots, etc.

A few ideas we've had there was to look for an additional tag on the package that would include your Our Umbraco member Id, so for example our1234. We could also look for it in the package description, maybe something like {our:1234} or something. This all feels a bit hacky though, but might be the best we can do with the limited option we have in the NuGet package format.

👉 We'd love to get some thoughts on how we can make claiming packages easy and reliable (and secure)!

Conclusion

In conclusion, the feedback we're most looking for is:

• How do we use NuGet tags to find the correct packages to import to Our Umbraco?
• How can we make sure that ownership is easily established?

Of course we would love to hear any other thoughts and ideas as well! 👍

[claushingebjerg]

Seeing as installation from the backoffice will be removed in 9, maybe remove the browse option altogether to avoid confusion.

[nul800sebastiaan]

@claushingebjerg the browse option is important for discoverability and will list the packages and instructions on how to install them.

[leekelleher]

Initial thought about the use of tags, do we know if NuGet has any limitations in the characters used?

Anyone old enough to use Flickr's "machine tags" (aka Triple Tags) back in 2007? The structure is like this... namespace:predicate=value

So you could have parse-able tags such as umbraco:member=1234.

[mattbrailsford]

I'm wondering whether this "automatic" search may be too problematic to have and I feel like you may be spending more time removing packages that shouldn't be listed than if you did it the other way round.

I'm wondering if there could just be an endpoint where build tools can ping (like UmbPack) that says a new package has been published and so Umbraco can then update. I think this is more explicit and prevents having to constantly update " the algorithm" as it invariably lists things it shouldn't list.

Something you'll need to consider which is evident in your screenshot of a scrape is that NuGet packages are often split up into multiple packages for the same "Umbraco Package" (Vendr is split into around 5 packages) so then it's a question of how should that get automatically listed? If it was manual (via a build process) I can easily tell you "this is the entry point and these are all the related packages".

[nul800sebastiaan]

I'm not too worried about removing packages necessarily, but we can trial it with some manual approval steps at first.

I like the idea of an additional endpoint as well! But it doesn't have to be the only way 🙂 Not everyone will have automated setups.

Good point in packages that are split into multiple packages! In that case, do you want to list only the main package, which has dependencies on the other ones?
In which case, the main package should be the only one that has the tags that will make sure it's listed on Our, that could work right?

edit: ha, you just said the same while I was typing.. 😅

[mattbrailsford]

Yea, specific tags could work 👍

Couple of other things that cross my mind

1. What about package pages that list multiple related packages (for example, Vendr lists Vendr and all the core supported payment providers) would we still have this, or would each listing be for individual package files? If the later, will there need to be some updated UI to have easier discoverability of "supporting" / "related packages"? (People who downloaded this package, also downloaded ... 😂)
2. If we use custom tags, is this going to show up in other NuGet tools and look weird / cause issues? It's ok introducing tags for Umbraco but we'll have to make sure it's not at the determent of traditional NuGet tool experience.

[nul800sebastiaan]

What about package pages that list multiple related packages

Maybe the answer is the same as it is for v8.. Would you list those on Our? Right now, I believe you haven't listed them, probably because it feels like you'd be "polluting" the overview?

Tags are indeed not an ideal way to list a member Id, they show up in VS / on NuGet

I would love to have an alternative way to add something to the NuGet metadata that is not as visible, but haven't found a great place yet (adding something in the description is equally visible).

[mattbrailsford]

Yes we do list them. So we list Vendr AND the core supported payment providers.

Also, on the contributor supplied payment providers, we list all those on a single package page to prevent polluting the package repo.

[nul800sebastiaan]

Aah!! I see, that is of course super great to have them grouped together on one package on Our.

Yes, we could add fields for listing related NuGet packages under the same main package, that is a fantastic point! 👍

[KevinJump]

How things are listed on our

Wondering if this should be more than a tick-box? and maybe the URL to the NuGet package. I know that is already there, but there might be reasons why the nuget package is different for a different version of Umbraco (after-all the Umbraco nuget package is different for v8 and v9). The ability to change things like the name is one thing we are loosing by not having a package to upload anymore.

if the nuget package is what you enter to make a package listed as v9 n our then that can be used to discover / list the packages - (as you would have a list of v9 nuget pakcage names) - they would be associated with the member, and you wouldn't have to worry about tags etc ?

[nul800sebastiaan]

Could work, but it gets confusing. What's a NuGet URL and what's a package name, and why is there 2 options for it?

Maybe in the future we can rethink the creation flow completely and start with a choice of:

• Create Umbraco v9+ package
• Create Umbraco v8- package

But it will still be confusing, plus what if it just works on both v8 & v9. I expect that most App_Plugins / js-only packages will just keep on working.

The checkbox was also a bit of laziness on my end in wanting to have a boolean to look for! 😉
All of this can pretty easily be changed, so whatever makes sense for the future that's not a hack is good with me!

[mattbrailsford]

Also, what about multi-targeted packages 😉

[nul800sebastiaan]

Also, what about multi-targeted packages 😉

That's is what I meant with: "plus what if it just works on both v8 & v9." - unless you mean something else?

[mattbrailsford]

I'm just being explicit. As where you have said this twice it's immediately followed by statements about JS only packages, so I wanted to be clear, not all packages that work on v8 and v9 will be "just JS packages", they may be explicitly multi-targeted.

[KevinJump]

yeah i think - longer term it will need to be something that works for multiple releases, as everything with be nuget when you are on v9 - you need to know what version you are actually installing . (e.g I've just had someone install a v8 version of a package on v7 today)

[KevinJump]

No sure how searchable / useful it might be - but the nuspec has space for 'packageTypes' that can be custom.
https://docs.microsoft.com/en-us/nuget/create-packages/set-package-type?tabs=dotnet

this would for example, give space for nuget packages to be defined as UmbracoPackage and have a potential version number against them.

e.g

    <packageTypes>
      <packageType name="UmbracoPackage" version="9.0" />
      <packageType name="UmbracoPackage" version="10.0" />
    </packageTypes>

but its a bit niche - would have to be communicated quite well (you could always fall back to assuming the version isn't there its compatible but it gives space for people to define the version their package works with).

[nul800sebastiaan]

Ah yes, I found this one as well, it's not something we can search on, but could work for parsing! I didn't know the version property existed, that is terribly useful. And having the ability to define multiple types is nice as well.

[mattbrailsford]

Worth taking into account, a custom package type won't install in Visual Studio

NuGet/Home#10468

[nul800sebastiaan]

Sad trombone :(

[KevinJump]

:( pity - because i think this fits exactly the use case the documentation describes.

[mattbrailsford]

Interesting challenge / thought this morning. What should packages like Vendr do that are multi targeting? For v8 we currently build a zip package that can be installed via the Umbraco back office. I could probably continue to do this for our v2 update for anyone running v8 / .NET Framework, but as we are multi-targeted, the zip file won't work for v9 / .NET Core.

[nul800sebastiaan]

There will be no option for installing zip files in the v9 backoffice either, so at least people won't accidentally try.

The way it works at the moment is that the zip files still show up in v8 backoffices so any updates will keep showing up. For v9 backoffices they will see NuGet install instructions.

[mattbrailsford]

I guess it's a question of what will this look like on our.umbraco.org where a version number isn't known. Will it be confusing to show things people can't use? Or will we introduce a "version toggle" or something?

[nul800sebastiaan]

Ah I see, yeah that could definitely do with a clear toggle! Noted! 👍

[nul800sebastiaan]

It would also be good to add some content in the backoffice saying: if you are trying to install a downloaded zip file, this is no longer an option in v9, you can search for the package you wanted though for further instructions.

[nul800sebastiaan]

@DiceBet-giris - please don't. We've moved all of that to https://marketplace.umbraco.com and the packages repository on Our Umbraco will no longer be updated.

[leekelleher]

@nul800sebastiaan I could be wrong, but the above comment sounds like ChatGPT spewed spam.

I got notifications from a couple of other discussion threads that felt similar, (e.g. a regurgitation summary of the whole thread).

[nul800sebastiaan]

Oh no... 🤦

[skttl]

I've just reported some in other discussions too...

[nul800sebastiaan]

I'll close this one as resolved and note that this has led to the creation of: https://marketplace.umbraco.com/