docs: Finish docs and tests for regex searches (#1051)

* docs,test: Add examples of searching for time stamps

* test: Add tests to show the text that is searched by description

* docs: Document how description's search text is constructed.

* test: Search description for short tags, excluding sub-tags

It's more complicated than I expected!
Roll on regex searching of tags

* test: Search description for sub-tags

* test: Simplify regexes by using String.raw

* test: Add missing 'it' block, for consistency

* test: More useful example for sub-tag search

And remove irrelevant comment

* docs: Document searching short tags and sub-tags

Roll on regex searching in tag and tags instructions.

* docs: Be consistent with case-insensitive flag

* docs: Heading regex searches are also case-sensitive

by default

* docs: Add a regex demo to examples.md

* test: Use String.raw for all regex searches

This makes it more likely that anyone added a future test
that requires an escape character will use this form,
instead of using the less readable '\\'.

* docs: Explain what [...] means

* docs: Add 'Escaping special characters' section

* docs,vault: Link between regex docs and sample page

The links won't work until both this PR is merged,
and the next release is done.

* docs: Improve the 'Important links' section

* docs: Link to tickets in 'known limitations' section

* test: Add regex demo of an example used for Boolean searches

* docs: Document pipe (|) in regex - and add example

* docs: Remove stray URL

Author: claremacrae

docs/queries/examples.md

@@ -95,3 +95,12 @@ All open tasks that are due today or earlier, sorted by due date, then grouped t
     sort by due
     group by folder
     ```
+
+---
+
+All open tasks that begin with a time stamp in `HH:mm` format, followed by any white space character:
+
+    ```tasks
+    not done
+    description regex matches /^[012][0-9]:[0-5][0-9]\s/
+    ```

docs/queries/filters.md

@@ -140,6 +140,21 @@ As well as the date-related searches above, these filters search other propertie
 
 > `regex matches` and `regex does not match` were introduced in Tasks 1.12.0.
 
+For precise searches, it may help to know that `description`:
+
+- first removes all each task's signifier emojis and their values,
+- then removes any global filter,
+- then removes an trailing spaces
+- and then searches the remaining text
+
+For example:
+
+| Global Filter    | Task line                                                                | Text searched by `description`   |
+| ---------------- | ------------------------------------------------------------------------ | -------------------------------- |
+| No global filter | `'- [ ] Do stuff  ⏫  #tag1 ✅ 2022-08-12 #tag2/sub-tag '`               | `'Do stuff #tag1 #tag2/sub-tag'` |
+| `#task`          | `'- [ ] #task Do stuff  ⏫  #tag1 ✅ 2022-08-12 #tag2/sub-tag '`         | `'Do stuff #tag1 #tag2/sub-tag'` |
+| `global-filter`  | `'- [ ] global-filter Do stuff  ⏫  #tag1 ✅ 2022-08-12 #tag2/sub-tag '` | `'Do stuff #tag1 #tag2/sub-tag'` |
+
 ### Priority
 
 - `priority is (above|below)? (low|none|medium|high)`
@@ -209,7 +224,7 @@ These filters allow searching for tasks in particular files and sections of file
   - `does not include` will match a task that does not have a preceding heading in its file.
   - Matches case-insensitive (disregards capitalization).
 - `heading (regex matches|regex does not match) /<JavaScript-style Regex>/`
-  - Whether or not the heading preceding the task includes the given regular expression.
+  - Whether or not the heading preceding the task includes the given regular expression (case-sensitive by default).
   - Always tries to match the closest heading above the task, regardless of heading level.
   - `regex does not match` will match a task that does not have a preceding heading in its file.
   - Essential reading: [Regular Expression Searches]({{ site.baseurl }}{% link queries/regular-expressions.md %}).

docs/queries/regular-expressions.md

@@ -78,27 +78,54 @@ description regex matches /pc_abigail|pc_edwina|at_work/i
 - Note that `tags` searches do not yet support regex searches.
   - As a workaround, search `description` instead.
 
+## Escaping special characters
+
+To search for any of the characters `[ \ ^ $ . | ? * + ( ) /` literally in Tasks, you should put a `\` character before each of them.
+
+This is called 'escaping'. See [Escaping, special characters](https://javascript.info/regexp-escaping).
+
+See the next section for the meaning of some of these characters.
+
 ## Special characters
 
 If using regex searches, it is important to be aware of the available special characters for several reasons:
 
 1. They enable complex queries to written in simple ways
 2. They can cause confusing results or broken searches, if not "escaped" in the search.
 
-Here are a few examples of the [many special characters](https://www.rexegg.com/regex-quickstart.html):
+Here are a few examples of the [many special characters](https://javascript.info/regexp-escaping):
 
 - `.` matches any character
-- `^` matches the start of the string (but when `[^inside brackets]`, it means "not")
-- `$` matches the end of the string
+- `[...]` means search for any of the characters in the square brackets.
+  - For example, `[aeiou]` will match any of an `a`, `e`, `i`, `o` or `u`.
+  - See [Sets and ranges \[...\]](https://javascript.info/regexp-character-sets-and-ranges)
+- Start and end
+  - `^` matches the start of the string (but when `[^inside brackets]`, it means "not")
+  - `$` matches the end of the string
+  - See [Anchors: string start ^ and end $](https://javascript.info/regexp-anchors)
+- `|` is an `OR` in regular expressions
+  - See [Alternation (OR) |](https://javascript.info/regexp-alternation)
 - `\` adds special meaning to some characters. For example:
   - `\d` matches one digit, from 0 to 9
   - `\D` matches character that is not a digit
+  - See [Character classes](https://javascript.info/regexp-character-classes)
+
+For a thorough, clear introduction to all the options, see [Regular expressions](https://javascript.info/regular-expressions) at JavaScript.info.
 
 ## Important links
 
+Learning resources:
+
+- [Regular expressions](https://javascript.info/regular-expressions) at JavaScript.info
 - [Regex Tutorial](https://regexone.com/)
 - [Regex Cheat Sheet](https://www.rexegg.com/regex-quickstart.html)
+
+Online tools for experimenting with - and testing - regular expressions:
+
 - [Regex Testing Tool: regex101](https://regex101.com/): Select the flavor 'ECMAScript (JavaScript)'
+
+Implementation details:
+
 - Implemented using [JavaScript's RegExp implementation](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions)
 - Supports [JavaScript RegExp Flags](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions#advanced_searching_with_flags), although not all of them are relevant in this context.
 
@@ -109,22 +136,103 @@ Please be aware of the following limitations in Tasks' implementation of regular
 - The single error message `Tasks query: cannot parse regex (description); check your leading and trailing slashes for your query` may mean any of:
   - The opening or closing `/` is missing from the query.
   - The regular expression is not valid, for example `description regex matches /[123/`.
+  - Logged in [#1038](https://github.com/obsidian-tasks-group/obsidian-tasks/issues/1038) and [#1039](https://github.com/obsidian-tasks-group/obsidian-tasks/issues/1039)
 - No error when part of the pattern is lost, for example because unescaped slashes are used inside the pattern.
   - For example, `path regex matches /a/b/c/d/` actually searches for `path regex matches /a/`.
   - In this case, the query should be `path regex matches /a\/b\/c\/d/`.
+  - Logged in [#1037](https://github.com/obsidian-tasks-group/obsidian-tasks/issues/1037)
 - Illegal flags are ignored.
   - For example, the query `description regex matches /CASE/&` should give an error that `&` (and similar) are unrecognised flags.
 - The `tag` or `tags` instruction does not yet support regular expression searches.
+  - Logged in [#1040](https://github.com/obsidian-tasks-group/obsidian-tasks/discussions/1040)
 - [Lookahead and Lookbehind](https://www.regular-expressions.info/lookaround.html) searches are untested, and are presumed not to work on Apple mobile devices, or to cause serious performance problems with slow searches.
 
 ## Regular expression examples
 
-Example searches:
+Below are some example regex searches, to give some ideas of what can be done.
+
+There are some more examples in the [Tasks-Demo sample vault](https://github.com/obsidian-tasks-group/obsidian-tasks/tree/main/resources/sample_vaults/Tasks-Demo), in the file [Regular Expression Searches](https://github.com/obsidian-tasks-group/obsidian-tasks/blob/main/resources/sample_vaults/Tasks-Demo/Filters/Regular%20Expression%20Searches.md).
+
+### Searching the start of a field
+
+Find tasks whose description begins with Log, exact capitalisation:
 
 ```text
-# Find tasks whose description begins with Log, exact capitalisation
 description regex matches /^Log/
+```
+
+---
 
-# Find tasks whose description begins with Log, ignoring capitalisation
+Find tasks whose description begins with Log, ignoring capitalisation
+
+```text
 description regex matches /^Log/i
 ```
+
+### Finding tasks that are waiting
+
+I want to find tasks that are waiting for something else. But 'waiting' can be spelled in several different ways:
+
+```text
+description regex matches /waiting|waits|wartet/i
+```
+
+### Finding times
+
+Find tasks containing a time in the description - simple version. This matches invalid times, such as `99:99`, as `\d` means 'any digit'.
+
+```text
+description regex matches /\d\d:\d\d/
+```
+
+---
+
+Find tasks containing a time in the description. This is more precise than the previous example, thanks to specifying which digits are allowed in each position.
+
+```text
+description regex matches /[012][0-9]:[0-5][0-9]/
+```
+
+### Finding sub-tags
+
+Currently `tag` and `tags` searches do not yet support regular expressions. Therefore, for precise searching of tags, use  `description` instead.
+
+Suppose you wanted to search for tags of this form: `#tag/subtag3/subsubtag5`, where the `3` and the `5` are allowed to be any single digit.
+
+- We can use either `[0-9]` or `\d` to match a single digit.
+- To find a sub-tag, any `/` characters must be 'escaped' to prevent them truncating the rest of the search pattern.
+
+Escaping the `/` leads us to this instruction, which we have made case-insensitive to find capitalised tags too:
+
+```text
+description regex matches /#tag\/subtag[0-9]\/subsubtag[0-9]/i
+```
+
+### Finding short tags
+
+Currently `tag` and `tags` searches do not yet support regular expressions. Therefore, for precise searching of tags, use  `description` instead.
+
+Suppose you wanted to search for tasks with a very short tag in: `#t`, and to not match tags line `#task` and `#t/subtag`.
+
+The most general query is:
+
+```text
+(description regex matches /#t\s/i) OR (description regex matches /#t$/i)
+```
+
+We have made it case-insensitive to find capitalised tags too.
+
+The Boolean `OR` allows us to search for two different patterns, for a thorough search:
+
+- `description regex matches /#t\s/i`
+  - Matches `#t` or `#T`, followed by any white-space character. This might be a literal space, or it might be a tab, for example.
+  - It won't be a newline character though, as task descriptions are always exactly one line, with no newline character at the end.
+  - For example, this will match:
+    - `- [ ] #t Do stuff`
+    - `- [ ] Do #t stuff`
+  - But it will not match:
+    - `- [ ] Do stuff #t`
+- `description regex matches /#t$/i`
+  - Matches `#t` or `#T` at the very end of the task line (after all signifiers and trailing white space has been removed)
+  - For example, this will match:
+    - `- [ ] Do stuff #t`

resources/sample_vaults/Tasks-Demo/Filters/Regular Expression Searches.md

@@ -1,5 +1,10 @@
 # Regular Expression Searches
 
+This file contains a few examples that were useful when testing
+the feature to search for text using regular expressions.
+
+Full documentation is available: see [Regular Expressions](https://obsidian-tasks-group.github.io/obsidian-tasks/queries/regular-expressions/).
+
 ## Sample Tasks
 
 - [ ] #task Do thing 1 #context/pc_abigail

src/Query/Filter/DescriptionField.ts

@@ -15,10 +15,12 @@ export class DescriptionField extends TextField {
 
     /**
      * Return the task's description, with any global tag removed
+     *
+     * Promoted to public, to enable testing.
      * @param task
-     * @protected
+     * @public
      */
-    protected value(task: Task): string {
+    public value(task: Task): string {
         // Remove global filter from description match if present.
         // This is necessary to match only on the content of the task, not
         // the global filter.