RFC 35 - Easier Schemas

[jonathanyeung]

Discussion Thread for Easier Schemas

---

10/17/2021 Update

• Added sections for proposed changes to the schema definition language
  • Support inline schemas
  • Support pattern matching of descendants
• As a consequence of the proposed changes to the schema defintiion, the non-goals section has been removed.

[kevinslin]

First of all, I like the approach this is taking in terms of making it easier to create schemas and templates from existing notes. Some high level thoughts:

• we are using the quickpick as the main interface for supplying informatoin and feedback. does it make sense to have a graphical interface like https://app.graphqleditor.com/explore-projects/feature-mole?
• nitpick: i would introduce creating schemas from hiearchies ahead of creating templates for hiearchy in the RFC since its the more general concept

Some specifics

Creating a template

We ask the user to pick a name for this schema. By default, the value will be filled in to the last non-wildcard/regex hierarchy, so here it would be prefilled as daily.journal. The user can change the name in the input box.

Currently the schema convention is one schema per hiearchy. Its not set in stone but it seems like with this convention, you would have multiple for a given hiearchy. Can we default the name to the base path (eg. daily in this case)?

Some more thoughts:

• if the user already has a schema and wants to add a template, what is the behavior? Does it make sense to prompt the Create schema from hiearchy flow first?

Creating a schema

• to make sure I understand, when we create a schema, its assuming that there is always one component that is a namespace?
  • correct in assuming that multiple namespaces are out of scope? or could we handle that by turning the namespace selection into a multi-select?

Other thoughts

• a common case people have is schema for daily journal. we should built that in to Dendron

[jonathanyeung]

we are using the quickpick as the main interface for supplying informatoin and feedback. does it make sense to have a graphical interface like https://app.graphqleditor.com/explore-projects/feature-mole?

I like the thought, and something we should try to get to - for the sake of 80/20, I think let's stick to a simple QuickPick for now since it's also quick to implement 😄 .

nitpick: i would introduce creating schemas from hiearchies ahead of creating templates for hiearchy in the RFC since its the more general concept

I don't disagree - the ordering of the commands in the spec were arbitrary, not necessarily a priority ordering.

Currently the schema convention is one schema per hiearchy. Its not set in stone but it seems like with this convention, you would have multiple for a given hiearchy. Can we default the name to the base path (eg. daily in this case)?

The only objective of that point was to create a placeholder id name, I wasn't aware that the id shouldn't have . in them. defaulting to base is fine.

if the user already has a schema and wants to add a template, what is the behavior? Does it make sense to prompt the Create schema from hiearchy flow first?

That's a good question - one way I see people using these commands is to use them to create a starting point for their schema definitions, which they can then modify as they see fit. The commands can also be used as a way to learn about how schemas work - if someone already had a schema and wanted to add a template to a component of it, then they could try modifying the schema directly. But if they can't get it working, they could try the command and see what the auto-generated schema looks like.

One potential problem here is when multiple-schemas apply to the same file. We should have better user feedback for when this is happening, or else I see this as a hard-to-debug situation.

correct in assuming that multiple namespaces are out of scope? or could we handle that by turning the namespace selection into a multi-select?

We can have multi-select. I should have reworded my question to "always at least one component that is a namespace?"

a common case people have is schema for daily journal. we should built that in to Dendron

Good idea. Actually, maybe we can fold this more tightly into the tutorial experience - i.e., as part of the first user workspace, we already have a daily journal note in their workspace, as well as a schema template for it. We can use this as an example to the user of how schemas, templates, and hierarchies work in Dendron all in one example.

[emarthinsen]

If it's useful, I'll throw a couple of thoughts on here, since I've been deep into schemas the last few days.

1. Since it's a technical audience, I'm not sure an interface for creating schemas is all that important. I found the act of creating them was fairly easy, but understanding how they got applied was not. The exception to "creating them was fairly easy" was the way that hierarchies are represented in the schemas. Having the children created at the same level as the root of the schema and then referencing those children from the root was confusing. I found myself wishing that I was just working with a nested data structure.
2. I think schemas can be enormously powerful for the creation of notes. I think the use case of ensuring note's conform to a schema is less important, but my use cases may not be the common one. I see the built-in notions of the scratch notes and daily journal as something that could be built with a slightly more powerful schema system. I think the missing delta would be the ability to specify a filename in the schema and the ability to easily map creation of a note from a schema to a keystroke. Being able to hit shift+ctrl+i to create a daily journal note is super convenient.
3. It would be nice if the template property in a schema would be inherited by the children specified in that schema. For instance, if I create a hierarchy that starts with kb and I create children like kb.math.algebra, if there was a template specified at the kb level, then the kb.math.algebra notes would inherit it as well.
4. Purely speculative, but I wonder if hooks can be incorporated into schemas as well. I can imagine someone wanting a hook on one note type and not another. I haven't used hooks yet, so take this with a huge grain of salt.

Those are the main thoughts. Overall, I think schemas could be very powerful and could be a construct that unlocks more power for users to customize how Dendron works to their specific workflows.

[jonathanyeung]

Thanks @emarthinsen for the feedback! Some responses:

1. Where we're coming from with this is that we get a lot of questions from folks like "I can't get my template to get applied to my not here!" (myself included sometimes 😄 ). The idea is to have a quick and reliable way to create a basic schema to get the user started, which they can then enhance and modify as they see fit. It's something that will likely be better suited for users who are just getting started with Dendron. Re: nested data structure - @nickolay-kondratyev has proposed something similar, and we're probably going to include that into this spec too.
2. Interesting idea - can you elaborate a little bit on "the ability to easily map creation of a note from a schema to a keystroke"? Are you saying that as a user, I want to be able to construct a command (with a registered shortcut) that can create a note matching some schema and possibly some generated pattern for the note name? I have another RFC in this domain, which is around creating a type system for notes that would integrate with the schema system: RFC 34 - Discussion Link here. We're thinking to encapsulate more advanced programmatic functionality into a concept of types, which can then be applied to notes via schema in the same way you can currently apply a template.
3. Yes - totally agree on this point. We've seen other folks ask for this as well; we need to add this functionality to the schema definition language.
4. Related to point 2, but take a look at RFC 34 - Type System for Notes

[emarthinsen]

Hi @jonathanyeung. Happy to expand.

For point #2, I see the current ability to create a daily journal with ctrl+shift+I and the ability to create a scratch note as a specialization of a larger feature. I think with a slightly more powerful schema implementation, both of those notes could be implemented as a schema. Once that happens, anyone can create a specialized note type that is as powerful as a built-in note. For the keystrokes, it's super helpful having the keystroke that will create a daily journal. I'd love to see the same idea be applied to an arbitrary note type. Perhaps I want to re-map crtl+shift+I to create a note from an "ideas" schema that defaults to a title similar to a scratch note, but in the ideas hierarchy. That's just an example. I scanned RFC 34 and it seems to be doing a lot of what I'm suggesting.

One thing I'd watch out for is the overlap between templates, schemas, and types. Having to create all three for a new type of note seems cumbersome. I think the idea of schemas and types can be combined. I can see templates being useful on their own.

[kevinslin]

Creating custom schemas via keyboard shortcut makes sense and is probably something we'll want to support. For existing note types (journals and scratch), you can create custom keyboard shortcuts to create them, docs here.

[jonathanyeung]

One thing I'd watch out for is the overlap between templates, schemas, and types.

Yes, definitely agree. Dendron has enough usability quirks as is - in order for these concepts to gel smoothly there's still some reconciliation that's lacking from the specs as they stand right now.