Is it possible to defer loading a plugin after another plugin being loaded?

[sharpchen]

I want to conditionally load a plugin(an extension plugin for a language server) using cond = vim.lsp.is_enabled(ls_name), so I would like to load this extension plugin after lspconfig spec because I put all vim.lsp.config stuff within it.
That is, to evaluate the cond correctly, lspconfig needs to be loaded before the extension plugin.

I know I may migrate vim.lsp.enable() stuff to general lua file but I want to keep it lazy together with lspconfig if possible.