Feedback on "Packages in Umbraco 9 via NuGet" #10411
Replies: 11 comments 47 replies
-
|
While I do have strong opinions on this, I'd like to keep my questions focused. 🙏 My initial question is: Would the proposed "Hot Reload" feature in .NET 6 have any bearing to revert on this decision? |
Beta Was this translation helpful? Give feedback.
-
|
I thought the same thing, but that's for development only and not a production environment |
Beta Was this translation helpful? Give feedback.
-
|
@stevetemple Oh, I see. Thanks for clarifying. I'm getting less excited by .NET Core the more I find out about it. 😬 |
Beta Was this translation helpful? Give feedback.
-
|
It's very cool, much prefer developing .NET Core sites |
Beta Was this translation helpful? Give feedback.
-
|
@stevetemple I'd like to know more about the cool parts. Of course, not to distract from the original topic (apologies). Catch up on Slack or beers. 🍻😀 |
Beta Was this translation helpful? Give feedback.
-
|
Will package developers need to manually create (and maintain) the project page on Our? e.g. when we release a package update (on NuGet), do we need to update on Our too? (either manually or via UmbPack) |
Beta Was this translation helpful? Give feedback.
-
|
We're going to need some way to know which packages support V9, and as the package section is driven by an API from the information in our.umbraco.com, we do somehow need that information there. So yes, I'd expect package authors would still want to create a page on Our - so their package is promoted there, and will appear in the back-office. The difference of course is that now there's no package file to upload where Umbraco version information is provided, so there will need to be something like a check-box to tick for "V9 compatibility". In time perhaps this could be automated via the NuGet API - i.e. based on the NuGet package name provided, figure out whether there's a direct or transitive dependency on Umbraco V9. But even with a check-box, it'll be a one-off task; repeated only when we get to a future Umbraco major version. |
Beta Was this translation helpful? Give feedback.
-
Not exactly, we just host the files 😊 You are more than welcome to request removing packages you own! Anyway, that was not the point, I don't think giving us a API key would be a good idea. I definitely see it as a pull system where we pull the publicly available information from NuGet to offer a view in the package section and on Our Umbraco. Indeed, it would be awesome if we could pull in info on package not yet created on Our. The additional benefit of also creating it on Our is to provide meta data, like screenshots, enable voting on package, marking it as compatible with Umbraco Cloud etc. |
Beta Was this translation helpful? Give feedback.
-
|
Also, NuGet has started making API keys with a limited lifetime only (think it's max 1 year now), which would cause rather constant confusion when they expire on each package. |
Beta Was this translation helpful? Give feedback.
-
No, I'm not suggesting that we give our own personal API key for NuGet. It'd be that OurUmbraco would be an additional owner on the NuGet pkg. So HQ would have its own API key. (Globally set/used elsewhere, like you say it expires after a year, best to regen in a single place.) ...or alternatively, the v9 package repository could be a separately managed NuGet source/feed, (ala MyGet). Easier to query/scrape meta-data from a smaller subset of pkgs. |
Beta Was this translation helpful? Give feedback.
-
|
I see what you mean now Lee! 👍 Personally, I would not want the responsibility to be able to push to people's NuGet feeds, that kinda feels sacred to me. Since NuGet.org is the default feed in all supported IDEs, I'd also be wary of adding new feeds for the masses, we already see with the betas and nightlies that it's not an easy setup. That said, maybe we'd need to head that way if it turns out that the NuGet API is not sufficient to effectively get all packages. But I would very much prefer to go the most expected route for everyone: using NuGet as it was intended. 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Can we retain the ability to install content/datatype (i.e no dll's) only packages using the packaging service? These should still be able to be installed without the need to restart the application. |
Beta Was this translation helpful? Give feedback.
-
|
That is true. I don't think we have a way of telling them apart at this time, but it could be something we could add later... |
Beta Was this translation helpful? Give feedback.
-
|
Just out of curiosity. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, I only install pkgs via the backoffice. If I need to do C# work with them, then (and only then) I'll add a NuGet reference to my VS proj. Same for Umbraco installs, I only use the ZIP releases, I don't install using NuGet. (I fully expect to be called a charlatan.) Yes, I use Website projects! 😱 |
Beta Was this translation helpful? Give feedback.
-
|
@claushingebjerg not that this fully covers NuGet vs Umbraco packages, but for instance our Skybrud.Umbraco.Redirects has 137K downloads on NuGet, and only 746 downloads on Our (to be fair V1 for Umbraco 7 wasn't on Our, but still). Our Skybrud.Umbraco.GridData package has been on Our for both Umbraco 7 and 8, and has received 116.2K downloads on NuGet, while 3.1K on Our. These are our two most downloaded packages, but our other packages follows the same trend. |
Beta Was this translation helpful? Give feedback.
-
|
That's great data. Thanks @abjerner |
Beta Was this translation helpful? Give feedback.
-
|
NuGet numbers aren't that reliable if they're not cached on build pipelines fx, it will cause a download for each build. They are probably still way more used than Umbraco packages though 😉 |
Beta Was this translation helpful? Give feedback.
-
|
Exactly @jemayn. Overall, that means I don't trust these numbers at all 🦺 |
Beta Was this translation helpful? Give feedback.
-
|
Should we start thinking about an official Umbraco CLI? Whilst there will be some non-technical people that struggle with the NuGet change, from my impressions most folks who are used to just installing packages in the BO are those more familiar with front end technologies and not so much .NET. I wonder then if an official CLI would make things a bit more palatable and more familiar to those coming from front end frameworks where CLI's are pretty much the norm. Under the hood this may just proxy to other services, but I think it could allow the hiding of multiple tools and any weird configurations of those that might be needed. You can then provide a simplified API that makes sens to even beginners.
|
Beta Was this translation helpful? Give feedback.
-
|
With our own CLI this could also fix the issue of installing content / data types etc as this could be a feature of the CLI rather than Umbraco as you really only want those to be installed at the point of instruction, not potentially on other environments (which would be what happens if you used Migrations). |
Beta Was this translation helpful? Give feedback.
-
|
That's an interesting idea... worth exploring absolutely |
Beta Was this translation helpful? Give feedback.
-
|
In netcore it's much simpler to run the app from the CLI so I think this would become much easier to implement than through framework |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry, but I don't like the idea.. This would just be yet another tool, that we should manage and document 😓. Microsoft has excellent documentation on how to use the Regarding "installing content / data types etc", we have plans for this inside Umbraco, as a new runtime state before the |
Beta Was this translation helpful? Give feedback.
-
|
@mattbrailsford do you mean like Chauffeur? https://github.com/aaronpowell/Chauffeur and it's 'deliverables' |
Beta Was this translation helpful? Give feedback.
-
|
I feel I may be making a mute point, as the NuGet decision has already been made, but if I don't mention this it will niggle at me. What about maintenance/triage packages? Throughout my Umbraco career (14 years so far) there have been times when certain maintenance packages have saved the day. What I mean by a maintenance package is a utility feature to assist with older Umbraco installations. A general example would be that I pick up a legacy/brownfield project where the client has had a falling out with their existing agency, I'm given zero access to the web-server itself, only access to the CMS backoffice. (Honestly this happens way more that you'd expect!) To this, older packages such as Config Tree (not currently available on Our, but there are modern versions of it, Config Editor Manager, Configuration Editor), User Control File Editor, F.A.L.M. Housekeeping, UnVersion, community Health Checks, et al. could help triage issues for supporting clients with their legacy Umbraco sites. In my mind I'm likening this to compilation types, Ahead of Time (AOT) (features we know we need in advance), and Just In Time (JIT) (features we don't know we need until that moment we need them). I appreciate this may not be an issue or concern for new/current Umbraco implementations, but come 5 years down the line when agencies/freelancers will pick up on old v9 projects, they'll be difficult to triage. Like I say, this may be a mute point, but worth flagging up. |
Beta Was this translation helpful? Give feedback.
-
|
Depending on the hosting environment and whether it supports restarting the app when dlls are dropped in (.NET 6 supports shadow copying in IIS but even that requires pre-configuration to enable) will be the ultimate dictator. Even if you were able to drop in the files, they wouldn’t get picked up without a restart so you’d still be stuck. Maybe if you had access to some portal that could restart the service it’d be possible, but if you have access to the portal you can probably get more access from there anyways. I guess there is also going to be potential issues with how packages install. If they all need “UseXyz” methods in startup.cs you’d be even further screwed, though I think it is potentially possible to auto install a pkg if no “UseXyz” method is called. But that in itself feels like a none standard behaviour that people might not expect (for .NET Core devs). Ultimately I think this comes down to Umbraco is a .NET CMS and these are the new limitations. Maybe this will change in future, I don’t know. |
Beta Was this translation helpful? Give feedback.
-
|
Question. In the blog post it says various routes were investigated but it couldn’t be made to work. I wonder if what was tried could be detailed so that if someone knows some other way they could suggest something without suggesting things already tried. Ie Microsoft document how to create an application with plugins here https://docs.microsoft.com/en-gb/dotnet/core/tutorials/creating-app-with-plugin-support but I’m guessing something like this was dismissed but I’m not sure why? |
Beta Was this translation helpful? Give feedback.
-
At the end of the day, no matter what trick you do with dynamically loading DLLs, the app will require a restart because Dependency Injection needs to re-boot. The only way to do that is to restart the application. Dynamically loading in DLLs is easy, but you cannot at runtime load them in and also have them play nicely with DI because DI is configured at startup and cannot be re-configured. There's various unsupported tricks to force this like restarting the Generic Host but it's unsupported and comes with all sorts of caveats - like you would absolutely never be able to use statics, even for locks, or for lazy initialization (and others) without even more tricks that can never be guaranteed. We want to be able to have a supported and predictable packaging/extensions platform with features like dependency resolution and the end result is always Nuget. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, that makes sense. Weird that MS would have a plugin mechanism that can require dependencies but not register your own, but I get what the issue is 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, For many years I've been talking about "Package Migrations" which is a way to unite Nuget with Umbraco data installation. This is what will be happening now. The great thing is that the CMS code has come a long way to be able to accomplish this now much more easily than it was all those years ago when I created an initial PR for that. I've put together a document internally that I'll share here outlining the high level way this will work. Hopefully this will answer some of your questions. Package MigrationsOriginal implementation WIP here https://github.com/umbraco/Umbraco-CMS/pull/1307/files What is it?We need to be able to install Umbraco data on site load after an Umbraco extensions Nuget package is installed. Previously this was only possible while installing a package via the back office because Umbraco was already running. How to do this?We will re-use much of the logic done for Umbraco's Core migration logic. On startup, the CMS will do some assembly scans for a package migration type, it will then check what "ID" (this will be a GUID) is the last "Step" in a packages published "Plan". The CMS will check in the database key/value table for that package's row and if the value in that row doesn't match the "ID", then it means a package migration needs to execute. This will be a first class citizen. The v8 way that devs hacked their own Migrations in place using the Umbraco migrations classes was an ugly experience because the CMS never natively supported it and Developers would have to try to track state of what was previously run, etc... This will no longer be necessary and will be consistent. The "Plan"Just like in the CMS core, Umbraco has an "UmbracoPlan". This is a class that defines all of the steps involved in migrating from one ID to the next until it reaches the final ID. It is essentially a list of "ID"s mapped to Classes (migrations). This means that a migration will never run a second time on the same database. The "ID"The Umbraco CMS "ID" is a GUID based Id but it uses the registry GUID format which we won't use for packages. For package migrations we'll just use a normal GUID. StartupOn startup we will assembly scan for a new type such as IPackageMigration (TBD). This will resolve an implementation that allows checking for the packages "Plan" to then determine if Umbraco has already tracked its latest "ID". Umbraco will collect all migrations that are needed to run for all packages and will execute these silently during startup. Upon the successful execution of each package's migrations, we will track the successful final ID in the database. We will execute each pacakge's steps within it's own transaction so that if one package's migrations fails, it doesn't rollback all others. Runtime levelIf runnable package migrations are detected the application will be put into a new runtime state: PackagesMigration FailureIf any migrations fail, we will stop Umbraco from booting and show a modified version of the installer where we will list the errors and packages and have a button to retry. This will work very similar to the installer and unattended install failures. Old Package ActionsPackage actions will be gone - we will use the normal Umbraco migration system instead and eventually have some additional handy ext methods for common tasks if necessary. Migrations are just c# code, so you can execute anything. Umbraco package data manifestThis is the xml file that is created by the packager today. This file will be exportable from the Umbraco packages section based on data selected. For the most part, the file format will not change. This file can either be an embedded DLL resource, it it can exist in the root of the packages App_Plugins folder. For packages that don't have c#, the only option is to have a file. Some packages are only c#, in which case it must be an embedded resource. This will be the primary method for installing Umbraco data via package migrations. Explicit vs Implicit migrationsAn implicit package migration is when a package doesn't contain a "Plan" at all and only contains a package data manifest. In this case Umbraco will automatically generate a plan for implicit migrations . This is done by generating a final "ID" which will be the hash of the package version. If the final ID stored in the database doesn't match the hashed ID of the version, then the migration will need to be run. This migration will be a single execution step: Installing the package data manifest (i.e. just like it is done in the back office today). An explicit package migration is when a package contains a c# "Plan". In this case, the developer is entirely in charge of the package migration plan. If the developer wants the package data manifest file to be installed, that will be a line of code in a migration step. Typically the developer will want to run this code in every step (i.e. this is the same as what would happen to day when installing a package in the back office). This could probably be part of a base class for a package migration step. Umbraco Cloud / Deploy / uSyncFor Cloud sites or other sites where there are multiple environments that use tools like Deploy or uSync to synchronize them, it is probably safest to only have Package Migrations enabled on the most downstream environment (i.e. the environment where the package was installed). The developer flow would mean that the Umbraco data installed with a Package Migration is done by the developer locally and then any schema changes that were done will be committed to git (i.e. with Deploy it would be uda files). For any content items that need to be transferred to upstream environments, the developer would transfer the items that they want via the back office. How to do this... Configuration basedIOptions can be created and used to determine if Package Migrations should be enabled/disabled . This can be configured dynamically or via config files. Umbraco Deploy could dynamically disable Package Migrations from running entirely, else since Umbraco Deploy knows the most downstream environment, it could only enable them for that environment. Deployment basedUmbraco Deploy could track package migrations just like it does for schema items. When a migration is successfully run, a uda file can be created for that package. The uda file only needs to contain the final "ID". When Deploy extracts schema (which will need to occur before package migrations are resolved and executed) it can also extract package migrations. This means that migrations for a package would only ever execute on the environment they were installed on. If executed on each environment?If the same package migration is executed on each environment it shouldn't cause issues, however in some cases it may mean extra data will exist on an upstream environment. For example, if a package is installed on the local environment, then a developer deletes some of the Umbraco items installed with the package, then deploys the new codebase to an upstream environment; the package migration will run on the new environment and add the Umbraco items that were removed in the downstream environment. This also depends on if the items are schema vs content and how they are installed from the package. In the majority of cases, Umbraco data will be installed via the package data manifest and all items will be GUID based which ensures duplicates won't occur and data remains in sync. In some cases, a developer will write a custom package migration using Umbraco c# services; in this case it would be possible that duplicate data can be created and data will be out of sync. For example: if a package is installed on the local environment, then a developer pushes the code changes to an upstream environment and the migration runs there, then the same Umbraco objects may be created in both environments but without the same GUID; this can result in duplicates since Deploy will also run and install the same data with different GUIDs. What about DB schema changes?There are some examples of Migrations that would need to execute on all environments such as DB schema migrations. For example, if your package creates a custom DB table in a migration, that migration would need to execute on all environments else the DB table won't exist on those environments. How to solve that? At this stage I don't have a perfect answer. Perhaps there are 2x types of migrations that packages can expose. A default type which should only execute on one environment and another that should explicitly execute on all environments. This could be defined by an Enum on the migration and perhaps that tool that manages environment synchronization like Deploy or uSync dynamically filters the package migrations to be executed based on this enum value and what environment is current running. |
Beta Was this translation helpful? Give feedback.
-
|
Pondering this and some other things, I know an Umbraco CLI is being shot down, but I do think this might be the better approach for triggering content deploys. We only ever want content deploys to be applied at point of install. The issue I have with suggested options is that they either rely on some setting in config, or a flag in the DB as to whether they run, which means there is always the risk of bad config causing the content to be reinstalled accidentally. If instead, Umbraco just had some API / endpoint that was capable of taking a serialized piece of content and say "create this", then a CLI could be responsible for being the trigger mechanism and everything that lives in Umbraco is just benign unless it is called from the external tooling. Just a thought 💡 |
Beta Was this translation helpful? Give feedback.
-
That would require Umbraco running though. A CLI approach would require to boot umbraco as a command line app and then process the install. But then we're back at square one with what we had before - no package dependencies, having to dynamically load in DLLs instead of having them registered as dependencies in your csproj, requiring restarts, etc... We still need to do all of the above to properly support content/schema working with a Nuget package.
This isn't entirely true, some package migrations will need to run on all environments like ones that create new DB tables. I wrote about this above and something to be figured out still.
Moving forward, we need to move away from all file based flags and persistence. These types of things are not Cloud hosting (any cloud hosting) friendly since this is not scalable and doesn't work with load balancing scenarios. We store the state in the DB just like we do with the CMS state and will have ways to control over migrations running. |
Beta Was this translation helpful? Give feedback.
-
|
Fair points. RE content deploys I specifically said "content deploys" meaning deployment of content, not DB structure. I still see the current migrations functionality being necessary for DB migrations. Maybe I'm over simplifying it but I guess it saw a CLI as effectively a .NET CLI wrapper with added Umbraco-ness, so packages would still be installed as NuGet but it might then for example be extended to force content file deploys, and where possible, trigger the content deploys (I must admit, I haven't thought fully through how this would work). |
Beta Was this translation helpful? Give feedback.
-
|
For "content deploys" are you referring to only Umbraco content or Umbraco content + schema (i.e. anything that can be included in an umbraco package from the back office)? Thanks for the clarification, the idea makes a bit more sense. If we are ever to go down a route like this, it could work in tandem with what we're proposing above. I also haven't thought this fully through, but it could be something like:
|
Beta Was this translation helpful? Give feedback.
-
|
👍 so by content I mean anything that is held within a DB table, rather than the definition for the table, so things like content, media, doc type definitions, data type configs etc would all be covered by "content deploys" or "content migrations". Things that create DB tables would continue to use the current migrations system. Sounds like it could all be workable together so maybe they can all be supported. My biggest worry is that we just do everything the "microsoft way" which is often not that friendly and I think we can do better (obviously happy for there to be a solid foundation first if this can be built on top of). |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, interesting thought this morning after this discussion on Twitter https://twitter.com/darrenferguson/status/1402876276591955970 Given the comments about NuGet being really a .NET DLL package system and not really geared towards content files, is NuGet and/or I think there is defo going to be an issue around JS only packages as these can't simply be installed. Anything that contains content files is also going to need the solution to be rebuilt in order to trigger to copying of the content files. If we had a custom Umbraco CLI however, it could be the CLI that controls this and packages could just be installed. |
Beta Was this translation helpful? Give feedback.
-
JS only packages can still use the dotnet CLI to create a csproj and having a build target to copy the content files on build. This still means you can have package dependencies or others can have dependencies on your package and things will all just work without reinventing the wheel. For things like App_Plugins, it means that when you upgrade the upgrade is processed correctly and the files will be updated. Manually editing this if required is still just editing an XML file. JS only packages generally require a restart anyways. If there's a package manifest, and property editors, dashboards, etc... declared, a restart is still required. The items within that manifest are part of DI which needs a restart to be re-initialized. Is the main concern here being able to install JS only packages at runtime without restarts? |
Beta Was this translation helpful? Give feedback.
-
|
Not necessarily that is requires a restart, but the current requirement for install is a REBUILD to trigger the copying of content files. This means way more than just restarting an app, it means having a coding environment setup to trigger a .NET build. My thought is could that just be hidden behind a CLI so no solution build is required 🤷♂️ |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
or |
Beta Was this translation helpful? Give feedback.
-
Could |
Beta Was this translation helpful? Give feedback.
-
|
Hey all, thanks for the discussions here! We are getting closer to launch and we've been thinking about making it easy to get your NuGet package listed on Our Umbraco, we'd love for your help in the follow-up discussion here: #10595 |
Beta Was this translation helpful? Give feedback.
-
An easier way to copy schema between different sites using just the package.xml file?I'm way late to this discussion, but came across it today for a specific reason. I am working on a v10 site, and I want to just "copy" some doctypes/datatypes from a different site (v9, in this case). In versions <=8, one could use the back-office Package section to select a bunch of schema items, create a "package" and download a zip with the XML file, which could be uploaded into the other site via the back-office Package section there. Easy-peasy, no IDE/Compiling/Etc needed. In v10, I see we can still create a "package.xml" file via a nice UI in the back-office, but how do we install it into a new site? Well....... according to Documentation on Creating a Package:
And now... we have a reference to a package which really isn't needed any longer (cause hopefully the DB schema has been updated, so we should be DONE), so I guess we need to remove the package reference in the new site. Anyway, for a one-and-done, local-only sort of "schema import" operation... it would be great if there were a simpler process, with less overhead, more akin to the "old" style of backoffice-to-backoffice export/import. Would it be possible to add some functionality to just import a straight package.xml file via the back-office as a single-time operation for schema updating? What do people think about this? What am I missing in my thought-process? |
Beta Was this translation helpful? Give feedback.
-
@skttl - What sort of dynamism are you looking for? |
Beta Was this translation helpful? Give feedback.
-
|
My idea was creating some kind of dashboard in the package section, where you could upload a package.xml file. That file should then be migrated in, but I couldn't find how to do it, without the file being built into a "real" package. @ronaldbarendse makes some good pointers, I might take another stab at it, one day. Unless HQ wants to include this feature themselves :) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @ronaldbarendse & @skttl (as well as @NikRimington & @KevinJump )! Today I created a "minimum viable" version - the back-end bit works just fine, I just need to figure out the angular to get a file upload control. 🙄 Feel free to take a look/contribute here: https://github.com/hfloyd/Dragonfly.Umbraco10.SchemaImporter
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, I actually have a lot of this code (file upload/download) already in a local uSync build, so I've merged that and it would probably be the best place to look if you wanted to work out how uploads and file downloads can be done. there is also stuff about fileuploads in the doStuffWithUmbraco repo, but i haven't had a chance to v10 ify the Upload stuff yet - but maybe that will answer some of the Angular questions |
Beta Was this translation helpful? Give feedback.
-
|
Thanks so much, @KevinJump - your code samples helped me get the upload file control working 😁. The only thing I am still struggling with is showing some info on the dashboard with the result of the upload... I was trying to set values based on your uSync code... and I'd like it to be visible on the screen, though perhaps I have to "refresh" the display somehow??? (https://github.com/hfloyd/Dragonfly.Umbraco10.SchemaImporter/blob/master/src/Dragonfly/App_Plugins/Dragonfly.SchemaImporter/schemaImporterController.js#L41) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Today we published a blog post detailing our decision to move v9 to using NuGet packages: https://umbraco.com/blog/packages-in-umbraco-9-via-nuget/
This discussion allows you to give us some feedback ask questions, we understand that most of all package developers might have a lot of questions on their next steps. Umbraco HQ is here to help as much as we can as we know the package eco-system is very important to the success of Umbraco 9.
Beta Was this translation helpful? Give feedback.
All reactions