Allow filtering by arbitrary values in note frontmatter

[claremacrae]

Motivation

From #716, provide a way to either omit tasks from kanban boards or only contain tasks from kanban boards, and then apply other Task plugin query criteria.

---

kanban-plugin: basic

---

But instead of making a query specifically for kanban-plugin values, let's make something that is more general, to filter in or out any frontmatter values.

Ideas

Perhaps something simple like this, which would match any key or value in the front matter:

frontmatter (includes|does not include) <some text in a key>

frontmatter (includes|does not include) <some text in a value>

Or should it also searches to distinguish between keys and values?

frontmatter <key> (includes|does not include) <some text in a value for that key>

Related

We have a couple of requests to be able to search the tags from YAML frontmatter:

• filter by YAML tag #232
• Access YAML tags #318

[ezeeman]

The obsidian code must contain a YAML parser that is uses to process the front matter. If this is accessible to plugin developers, you could make the parse tree representation of the front matter for available for filter queries. This would be sort of like how the dataviewjs plugin allows queries using the hierarchical parts of the page, file, and task collections.

Here are some examples of how Tasks querues might look like if frontmatter represents the parsed front matter from a kanban file.

Overdue tasks from kanban notes (boards):

 ```tasks
 not done
 due before today
 frontmatter frontmatter.kanban-plugin exists
 ```

Exclude tasks from basic kanban boards:

 ```tasks
 frontmatter 'frontmatter.kanban-plugin.value != "basic"'
 ```

This idea is similar to how the program yq accesses and manipulates YAML and jq does JSON. This isn't restricted to kanban notes, and would operate on any type of YAML data that any plugin developer might support, including deeply nested lists and maps, and so on, as long as obsidian's parser supports the language feature.

[claremacrae]

Hi @ezeeman,

Wow, thank very much indeed for this really helpful post. I really like that structure for queries.

frontmatter frontmatter.kanban-plugin exists

OK, so having exists as a word with special meaning is a lovely idea.

One question though, what if someone wanted to search for the actual word exists in their front matter?

It seems to me that requiring presence or absence of some frontmatter needs to somehow be distinguished from words that might be present in the frontmatter.

Do you have any suggestions?

frontmatter 'frontmatter.kanban-plugin.value != "basic"'

I feel that we will likely get feedback that the duplication of frontmatter there is unnecessary.

I would have thought that this would work the same, and be more readable.

frontmatter 'kanban-plugin.value != "basic"'

I note that you have taken care with the quotes there, mixing single and double quotes.

I would welcome feedback from others on that. It's a step away from the existing includes and does not include of existing queries in Tasks. It will probably be easier to program, and allow for more flexibility.

[ezeeman]

I was thinking that there might be more than one YAML document in a note. (Each section in a YAML file, delimited by "---", is called a document.) The first document would be obsidian's "official" front matter, and others might be used for other purposes by plugins. Using frontmatter.key.... instead of key.... would always select the content of the official front matter. But I just tested this, and obsidian treats extra documents as part of the note's text instead of parsing them as YAML. So it's not worth worrying about.

The front matter document is structured as a tree of maps and lists and atomic types, so specific elements can be tested instead of doing simple text searches on the whole thing.

Obsidian itself defines the first four key/value pairs in the example below, and I've added a bit for discussion.

---
tags: [tag1, tag2]
aliases: [alias1, alias2]
cssclass: my_cssclass
publish: true
my_stuff:
  data: "exists"
  data_array: [ "one" 2 "three"]
---

You could select tasks using elements of this YAML with this syntax in a task block

frontmatter aliases contains "alias1"
-> true
frontmatter publish
-> true
frontmatter kanban-plugin exists
-> false
frontmatter my_stuff exists 
-> true
frontmatter my_stuff.data = "exists"
-> true
frontmatter my_stuff.data_array[0] = "exists"
-> false
frontmatter my_stuff.data_array contains 2
-> true
frontmatter my_stuff.data contains "x"
-> true

I'm defining a DSL here with the operators contains, exists, and =, and YAML data accessors using dot and [] notation. The contains operator is overloaded.

Instead of defining a new DSL, one might simply use javascript to write expressions that return booleans for task selection. In that case, you'd supply the js interpreter with a top level object, such as "fm", populated with data from the YAML parse, and the user would write queries against fm using the full capabilities of js. It would be like writing expressions with data pulled out of the DOM.

For example,

```tasks
done
frontmatter fm.my_stuff.data.includes("x")
# -> true
```

I don't know what the path of least resistance is on the obsidian platform.

[claremacrae]

Thank you. This is super helpful.

[montemartin]

Above @claremacrae said:

I note that you have taken care with the quotes there, mixing single and double quotes.

I would welcome feedback from others on that. It's a step away from the existing includes and does not include of existing queries in Tasks. It will probably be easier to program, and allow for more flexibility.

Most queries can technically be accomplished with Dataview, or even built-in Obsidian queries. The value in Tasks is how much less resistance I have to creating a new query when I need to find something specific. My first thought reading these comments was how far it strays from that simplicity.

Based on the current syntax I would expect a query of frontmatter key to include only notes that have that key with any value. Querying frontmatter myKey includes myValue would only include notes with the myKey: myValue pair and frontmatter includes myValue would search all values across keys. Hopefully my 2-cents helps!

[donthorp]

I just spent a good part of the day reworking how I organize tasks in my system. This feature would definitely help. I think a general syntax based on root type of the yaml entity would be the most flexible. My current use case involves selecting files to include or exclude based on frontmatter.

  ```tasks
  path does not include Templates
  path does not include Dashboards
  path does not include Tasks
  path does not include One on Ones
  path does not include Company
  (heading does not include Personal) AND (heading does not include Company) AND (heading does not include Entity2) AND (heading does not include Entity3) AND (heading does not include Entity4)
  
  not done
  is not recurring
  
  sort by urgency, happens
  
  group by backlink
  
  hide urgency
  hide scheduled date```

Basically I'm trying to identify tasks that might be orphaned in other views. I spent several minutes trying to filter by adding

  (heading does not include Personal) AND (heading does not include Company) AND (heading does not include Entity2) AND (heading does not include Entity3) AND (heading does not include Entity4) AND (tag does not include entity)

where entity was defined in the front matter

---
tags: ["entity/entity2"]
---

before I realized tasks can't reduce the set of files as one of the filters. I also have another tag I like to use flag/ignore-tasks that I put on checklists for trips, etc.

Having access for comparisons to the frontmatter would really help. The data type based queries that handle simple data types, objects, lists, would be extremely useful.

[claremacrae]

I appreciate the value of this feature. However, the Tasks code will need quite a lot of refactoring before this feature could be added in a sustainable and maintainable way.

In the meantime, I wrote up a workaround that can be used right now, to find notes tasks in notes with particular tags:
https://github.com/obsidian-tasks-group/obsidian-tasks/blob/main/resources/sample_vaults/Tasks-Demo/How%20To/Find%20tasks%20in%20notes%20with%20particular%20tag.md

[claremacrae]

Hi @kenlefeb, thanks for the suggestion.

That's an interesting use of alias - changing its meaning, rather than creating a dedicated field in frontmatter - but it obviously works for you, so why not...

You are right that what you request is different from the topic of this request, and so it would definitely be out of the scope for this ticket.

Please feel free to go to Issues and create a Feature Request so that your request is not lost.

[kenlefeb]

I used "alias" just because that's what we call the employee's network usernames here.

I'll create a new thread, next week when I'm back in the office. Thanks for such a prompt response!

[claremacrae]

Haha - that makes perfect sense.

[axelson]

@kenlefeb Did you ever start a separate discussion for this? I'm trying to figure out if there's an existing feature request for this.

[claremacrae]

@axelson - I'm not sure about that specific question, but if you search all the Issues and all the Discussions for YAML and Frontmatter (so, 4 searches) you will see a lot of requests. I'm not sure whether having any more will add any value...

[claremacrae]

Another request - in Reddit

Let's assume i have a specific project note called 'learn scraping' which has the frontmatter data

---
'area: coding'
---

Then in the 'coding' note i want to queue all tasks from notes with the frontmatter data area = 'coding'

[claremacrae]

This would of course also include making the frontmatter accessible to custom filters and groups.

Requested in #2145

[pianoman723]

To add another use case to this issue: I'm using the Projects plugin for project/task management. I want to build a query in Tasks that pulls in the open actions from Projects with a particular status. The Projects plugin uses frontmatter for all properties, which are completely customizable. So I want to be able to plan tasks into a project that is not Active yet (and thus the tasks do not appear in my query), but then 'activate' the project in Projects (via changing a frontmatter property), so that all its tasks will appear in my query. I can do this in dataview via the code below, but I'd rather utilize Tasks' ability to keep the link to the project note inline with the task description.

TASK FROM "Projects" WHERE Status = "0-Active" GROUP by file.name

[claremacrae]

Thanks. Much appreciated.

[claremacrae]

This has just been released in Tasks 7.7.0.

For details and examples, see the Obsidian Properties documentation page.