Merge pull request #2599 from obsidian-tasks-group/fix-default-sort-order

fix!!: Make default sort order sort first by status type

Author: claremacrae

docs/Getting Started/Priority.md

@@ -43,8 +43,8 @@ The following instructions use the priority signifiers in tasks.
 - `priority is (above, below)? (lowest, low, none, medium, high, highest)`
   - [[Filters#Priority|Documentation]]
 - `sort by priority`
-  - [[Sorting#Basics|Documentation]]
+  - [[Sorting#Priority|Documentation]]
 - `group by priority`
-  - [[Grouping#Basics|Documentation]]
+  - [[Grouping#Priority|Documentation]]
 - `hide priority`
   - [[Layout|Documentation]]

docs/Introduction.md

@@ -13,10 +13,17 @@ _In recent [releases](https://github.com/obsidian-tasks-group/obsidian-tasks/rel
     Move the older ones down to the top of the comment block below...
 -->
 
-- X.Y.Z: 🔥 Add [[Custom Sorting|custom sorting]].
-- 5.6.0: 🔥 The [[Postponing|postpone]] menu now offers `today` and `tomorrow`.
-- 5.5.0: 🔥 The [[Create or edit Task]] modal can now edit Created, Done and Cancelled dates
-- 5.5.0: 🔥 Add support for [[Dates#Cancelled date|cancelled dates]].
+- X.Y.Z:
+  - Add [[Custom Sorting|custom sorting]].
+  - Document the [[Sorting#Default sort order|default sort order]].
+  - **Warning**: This release contains some **bug-fixes** to **sorting** and to treatment of **invalid dates**.
+    - The changes are detailed in [[breaking changes#Tasks [X.Y.Z](https //github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/X.Y.Z) (21 January 2024)|breaking changes]], even though they are all improvements to the previous behaviour.
+    - You may need to update any CSS snippets for the Edit or Postpone buttons: see [[How to style buttons]].
+- 5.6.0:
+  - The [[Postponing|postpone]] menu now offers `today` and `tomorrow`.
+- 5.5.0:
+  - The [[Create or edit Task]] modal can now edit Created, Done and Cancelled dates
+  - Add support for [[Dates#Cancelled date|cancelled dates]].
 
 > [!Released]- Earlier Releases
 >

docs/Queries/Sorting.md

@@ -12,7 +12,7 @@ publish: true
 
 This page is long. Here are some links to the main sections:
 
-- [[#Basics]]
+- [[#Default sort order]]
 - [[#Sort by Task Statuses]]
 - [[#Sort by Dates in Tasks]]
 - [[#Sort by Other Task Properties]]
@@ -22,11 +22,32 @@ This page is long. Here are some links to the main sections:
 - [[#Reverse sorting]]
 - [[#Examples]]
 
-## Basics
+## Default sort order
 
-By default Tasks sorts tasks by [[Urgency|a calculated score we call "urgency"]].
+The following instructions are the default sort order, and they are **automatically appended to the end of *every* Tasks search**:
 
-To sort the results of a query different from the default, you must add at least one `sort by` line to the query.
+<!-- snippet: Sort.test.Sort_save_default_sort_order.approved.text -->
+```text
+sort by status.type
+sort by urgency
+sort by due
+sort by priority
+sort by path
+```
+<!-- endSnippet -->
+
+It first sorts tasks in the order `IN_PROGRESS`, `TODO`, `DONE`, `CANCELLED` then `NON_TASK` to ensure that actionable tasks appear first, which is important in searches without a filter like `not done`.
+
+Then it sorts by [[Urgency]], which is a calculated score derived from several Task properties.
+
+The above lines are *always* appended to the end of any `sort by` instructions supplied by the user. There is no way to disable this.
+
+However, any `sort by` instructions in queries take precedence over these default ones.
+
+> [!tip]
+> To sort the results of a query differently from the default, you must add at least one `sort by` line to the query. The sort instructions you supply will take priority over the appended defaults.
+>
+> Adding `sort by` lines to the [[Global Query]] provides a way override to the default sort order for **all** searches (except those that [[Global Query#Ignoring the global query|ignore the global query]]).
 
 ## Custom Sorting
 

docs/Support and Help/Breaking Changes.md

@@ -16,15 +16,32 @@ In this case, as we use [semantic versioning](https://semver.org), we will alway
 
 To help users updating across multiple Tasks releases, we collect here links to the few Tasks breaking changes.
 
-- Tasks [2.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/2.0.0) (22 March 2023):
-  - The introduction of filtering for date ranges [[Filters#Appendix Tasks 2.0.0 improvements to date filters|improved the results of some date searches]].
-- Tasks [3.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/3.0.0) (2 April 2023):
-  - Some CSS snippets that use `>` in their selectors [[Styling#Appendix Fixing CSS pre-existing snippets for Tasks 3.0.0|may need updating]].
-  - For example, see comment mentioning `3.0.0` in the CSS sample in [[How to style backlinks#Using CSS to replace the backlinks with icons|Using CSS to replace the backlinks with icons]].
-- Tasks [4.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/4.0.0) (15 June 2023):
-  - The order of `group by urgency` [[Grouping#Urgency|was reversed]].
-- Tasks [5.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/5.0.0) (21 October 2023):
-  - The meaning of final backslash (`\`) characters on query lines [[Line Continuations#Appendix Updating pre-5.0.0 searches with trailing backslashes|has changed]].
-- Tasks [X.Y.Z](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/X.Y.Z) (21 January 2024):
-  - Code snippets used to change the look of the edit and postpone buttons must be changed, as explained in [[How to style buttons]], which gives several examples.
-  - When sorting tasks, invalid dates are now sorted before valid dates. See [[Sorting#How dates are sorted]].
+## Tasks [2.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/2.0.0) (22 March 2023)
+
+- The introduction of filtering for date ranges [[Filters#Appendix Tasks 2.0.0 improvements to date filters|improved the results of some date searches]].
+
+## Tasks [3.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/3.0.0) (2 April 2023)
+
+- Some CSS snippets that use `>` in their selectors [[Styling#Appendix Fixing CSS pre-existing snippets for Tasks 3.0.0|may need updating]].
+- For example, see comment mentioning `3.0.0` in the CSS sample in [[How to style backlinks#Using CSS to replace the backlinks with icons|Using CSS to replace the backlinks with icons]].
+
+## Tasks [4.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/4.0.0) (15 June 2023)
+
+- The order of `group by urgency` [[Grouping#Urgency|was reversed]].
+
+## Tasks [5.0.0](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/5.0.0) (21 October 2023)
+
+- The meaning of final backslash (`\`) characters on query lines [[Line Continuations#Appendix Updating pre-5.0.0 searches with trailing backslashes|has changed]].
+
+## Tasks [X.Y.Z](https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/X.Y.Z) (21 January 2024)
+
+These are all bug-fixes, improving the default behaviour, and recorded here for transparency.
+
+- Sorting
+  - The [[Sorting#Default sort order|default sort order]] now sorts first by status type, to greatly improve search results that include completed tasks.
+- Invalid dates
+  - [[Filters#Happens|happens]] date now ignores any invalid dates.
+  - `sort by [date]` now puts invalid dates before valid dates, as action is required. See [[Sorting#How dates are sorted]].
+  - `group by [date]` now puts `Invalid [date] date` as the first heading.
+  - `task.due.category` and `task.due.fromNow` now handle invalid dates as different from future dates. See [[Task Properties#Values in TasksDate Properties|Values in TasksDate Properties]].
+- Code snippets used to change the look of the Edit and Postpone buttons must be changed, as explained in [[How to style buttons]], which gives several examples.

src/Query/Sort.ts

@@ -1,7 +1,6 @@
 import type { Task } from '../Task';
-import type { Comparator } from './Sorter';
-import type { Sorter } from './Sorter';
-import { StatusField } from './Filter/StatusField';
+import type { Comparator, Sorter } from './Sorter';
+import { StatusTypeField } from './Filter/StatusTypeField';
 import { DueDateField } from './Filter/DueDateField';
 import { PriorityField } from './Filter/PriorityField';
 import { PathField } from './Filter/PathField';
@@ -12,13 +11,7 @@ type PlainComparator = (a: Task, b: Task) => number;
 
 export class Sort {
     public static by(sorters: Sorter[], tasks: Task[], searchInfo: SearchInfo) {
-        const defaultComparators: Comparator[] = [
-            new UrgencyField().comparator(),
-            new StatusField().comparator(),
-            new DueDateField().comparator(),
-            new PriorityField().comparator(),
-            new PathField().comparator(),
-        ];
+        const defaultComparators: Comparator[] = this.defaultSorters().map((sorter) => sorter.comparator);
 
         const userComparators: Comparator[] = [];
 
@@ -29,6 +22,16 @@ export class Sort {
         return tasks.sort(Sort.makeCompositeComparator([...userComparators, ...defaultComparators], searchInfo));
     }
 
+    public static defaultSorters() {
+        return [
+            new StatusTypeField().createNormalSorter(),
+            new UrgencyField().createNormalSorter(),
+            new DueDateField().createNormalSorter(),
+            new PriorityField().createNormalSorter(),
+            new PathField().createNormalSorter(),
+        ];
+    }
+
     private static makeCompositeComparator(comparators: Comparator[], searchInfo: SearchInfo): PlainComparator {
         return (a, b) => {
             for (const comparator of comparators) {

tests/Scripting/ScriptingReference/CustomSorting/CustomSortingExamples.test.other_properties_task.isDone_results.approved.txt

@@ -6,11 +6,11 @@ sort by function !task.isDone
 `sort by function` sorts `true` before `false`
 Hence, we use `!` to negate `task.isDone`, so tasks with [[Status Types|Status Type]] `TODO` and `IN_PROGRESS` tasks are sorted **before** `DONE`, `CANCELLED` and `NON_TASK`.
 =>
+- [/] Status In Progress
 - [ ] Status Todo
 - [] Status EMPTY
-- [/] Status In Progress
-- [-] Status Cancelled
 - [x] Status Done
+- [-] Status Cancelled
 - [Q] Status Non-Task
 ====================================================================================
 

tests/Scripting/ScriptingReference/CustomSorting/CustomSortingExamples.test.statuses_task.status.nextSymbol_results.approved.txt

@@ -6,10 +6,10 @@ sort by function task.status.nextSymbol
 Sort by the next status symbol.
 =>
 - [] Status EMPTY
-- [-] Status Cancelled
 - [x] Status Done
+- [-] Status Cancelled
 - [Q] Status Non-Task
-- [ ] Status Todo
 - [/] Status In Progress
+- [ ] Status Todo
 ====================================================================================
 

tests/Sort.test.Sort_save_default_sort_order.approved.text

@@ -0,0 +1,5 @@
+sort by status.type
+sort by urgency
+sort by due
+sort by priority
+sort by path

tests/Sort.test.Sort_visualise_default_sort_order.approved.txt

@@ -0,0 +1,40 @@
+- [/] #task In progress 🔺 📅 1970-01-02
+- [/] In progress
+- [ ] #task Highest priority 🔺
+- [ ] #task High priority ⏫
+- [ ] my description 📅 Invalid date
+- [ ] my description 📅 2023-05-30
+- [ ] my description 📅 2023-05-31
+- [ ] my description 📅 2023-06-01
+- [ ] #task Medium priority 🔼
+- [ ] my description
+- [ ] my description 🛫 Invalid date
+- [ ] my description
+- [ ] my description ⏳ 2023-05-30
+- [ ] my description ⏳ 2023-05-31
+- [ ] my description ⏳ 2023-06-01
+- [ ] my description ⏳ Invalid date
+- [ ] my description
+- [ ] #task Normal priority
+- [ ] Todo
+- [%] Unknown
+- [ ] xyz in '' in heading 'heading'
+- [ ] xyz in 'a_b_c.md' in heading 'a_b_c'
+- [ ] xyz in 'a/b.md' in heading 'null'
+- [ ] xyz in 'a/b/_c_.md' in heading 'null'
+- [ ] xyz in 'a/b/_c_.md' in heading 'heading _italic text_'
+- [ ] xyz in 'a/b/c.md' in heading 'null'
+- [ ] xyz in 'a/b/c.md' in heading 'c'
+- [ ] xyz in 'a/d/c.md' in heading 'heading'
+- [ ] xyz in 'e/d/c.md' in heading 'heading'
+- [ ] #task Low priority 🔽
+- [ ] my description 🛫 2023-05-30
+- [ ] my description 🛫 2023-05-31
+- [ ] my description 🛫 2023-06-01
+- [ ] #task Lowest priority ⏬
+- [x] #task Done 🔺 📅 1970-01-02
+- [x] Done
+- [-] #task Cancelled 🔺 📅 1970-01-02
+- [-] Cancelled
+- [^] Non-task
+- [] Empty task

tests/Sort.test.ts

@@ -14,9 +14,12 @@ import { DueDateField } from '../src/Query/Filter/DueDateField';
 import { PathField } from '../src/Query/Filter/PathField';
 import { SearchInfo } from '../src/Query/SearchInfo';
 import { Sort } from '../src/Query/Sort';
-import { fromLine, toLines } from './TestHelpers';
+import { StatusRegistry } from '../src/StatusRegistry';
+import { StatusConfiguration, StatusType } from '../src/StatusConfiguration';
+import { SampleTasks, fromLine, fromLines, toLines } from './TestHelpers';
 import { TaskBuilder } from './TestingTools/TaskBuilder';
 import { sortBy } from './TestingTools/SortingTestHelpers';
+import { verifyWithFileExtension } from './TestingTools/ApprovalTestHelpers';
 
 const longAgo = '2022-01-01';
 const yesterday = '2022-01-14';
@@ -34,6 +37,15 @@ afterAll(() => {
     jest.useRealTimers();
 });
 
+afterEach(() => {
+    StatusRegistry.getInstance().resetToDefaultStatuses();
+});
+
+function verifySortedTasks(tasks: Task[]) {
+    const sortedTasks = Sort.by([], tasks, SearchInfo.fromAllTasks(tasks));
+    verify(toLines(sortedTasks).join('\n'));
+}
+
 describe('Sort', () => {
     it('constructs Sorting both ways from Comparator function', () => {
         const comparator: Comparator = (a: Task, b: Task) => {
@@ -126,6 +138,29 @@ describe('Sort', () => {
         ).toEqual(expectedOrder);
     });
 
+    it('save default sort order', () => {
+        const sorters = Sort.defaultSorters();
+        const defaultSortInstructions = sorters.map((sorter) => sorter.instruction).join('\n');
+        verifyWithFileExtension(defaultSortInstructions, 'text');
+    });
+
+    it('visualise default sort order', () => {
+        StatusRegistry.getInstance().add(new StatusConfiguration('^', 'Non-task', ' ', false, StatusType.NON_TASK));
+
+        const extraTaskLines = `- [x] #task Done        🔺 📅 1970-01-02
+- [/] #task In progress 🔺 📅 1970-01-02
+- [-] #task Cancelled   🔺 📅 1970-01-02`;
+        const extraTasks = fromLines({ lines: extraTaskLines.split('\n') });
+        const tasks = SampleTasks.withAllRepresentativeDueDates()
+            .concat(SampleTasks.withAllRepresentativeStartDates())
+            .concat(SampleTasks.withAllRepresentativeScheduledDates())
+            .concat(SampleTasks.withAllPriorities())
+            .concat(SampleTasks.withAllStatusTypes())
+            .concat(SampleTasks.withAllRootsPathsHeadings())
+            .concat(extraTasks);
+        verifySortedTasks(tasks);
+    });
+
     it('visualise date impact on default sort order', () => {
         const dates = [
             ['long ago', longAgo],
@@ -166,7 +201,6 @@ describe('Sort', () => {
                 }
             }
         }
-        const sortedTasks = Sort.by([], tasks, SearchInfo.fromAllTasks(tasks));
-        verify(toLines(sortedTasks).join('\n'));
+        verifySortedTasks(tasks);
     });
 });