Repeatable to-dos: include next in `things.upcoming()`.

[mikez]

To Reproduce
Steps to reproduce the behavior:

1. Create a new to-do which repeats monthly (e.g. repeat every month on the 1st day).
2. Run things.upcoming().
3. The next upcoming event is not included; however, it is visible if you go to "Upcoming" in Things.app.

Expected behavior
Include the next upcoming event in things.upcoming().

[bkleinen]

I'll give it a shot.

[bkleinen]

while adapting the test cases I noticed that not all three upcoming tasks are retrieved by "upcoming". Both cancelled and completed tasks are shown in my upcoming view:

[mikez]

while adapting the test cases I noticed that not all three upcoming tasks are retrieved by "upcoming". Both cancelled and completed tasks are shown in my upcoming view

I can't reproduce that on my end. Is it because of this default setting?

[bkleinen]

I can't reproduce that on my end. Is it because of this default setting?

yes. as soon as the manualLogDate is after the completion, they are no longer shown.
This is not the case in the test database - but the start=Someday, and the current implementation of upcoming sets requires start to be 2/Someday.

[bkleinen]

So, I've started working on this (see closed PR 145) but before I go on, I need some clarification:

0. Naming:

in the db, there are templates for recurring tasks:

• rt1_recurrenceRule contains just that
  those dates are set:
• rt1_nextInstanceStartDate
• rt1_instanceCreationStartDate
  I do not know what the difference is between them, but I'll explore more.
• startDate is not set

For now, I assume that rt1_nextInstanceStartDate is the equivalent to startDate for sorting tasks by start_date.

presumably at rt1_instanceCreationStartDate things.app creates a new instance of the repeating task, which seems
to be just like all other tasks except that rt1_repeatingTemplate contains the uuid of the template.

currently the templates are named like so:

# Repeats
IS_NOT_RECURRING = "rt1_recurrenceRule IS NULL"

How should we name the repeats? repeats, repeating, recurring, repeatingTemplate (as in the reference field above)?
options for the field name:

• is_recurring matching the current constant "IS_NOT_RECURRING" (taking out the negation)
• is_repeating_task_template is used in Include Repeat Templates in upcoming(), fixes #122 #145 (seems quite long now)
• is_repeating_template

2. how should the "repeats" be requested?

As I understand the request logic, I think implementing it just like trashed

    trashed : bool or None, optional, default False
        - `trashed == False` (default), only include non-trashed tasks.
        - `trashed == True`, only include trashed tasks.
        - `trashed == None`, include both kind of tasks.

tasks = things.tasks(is_repeating_task_template=True)
tasks = things.tasks(is_repeating_task_template=False) #default, as this is set in the current version
tasks = things.tasks(is_repeating_task_template=None)

3. How to handle the start_date?

the repeatTemplates do not have a start date (see above). I would populate the start_date field in task
with the appropriate rt1-field, to ease handling them in python.

4. Cancelled and Completed Repeats

They are shown in "Upcoming" as long as they are not logged (either automatically or manually).
Now I understand that is_recurring should just be another parameter, the question wether they should
be included is obsolete.

5. start=Someday? or: how to request upcoming

def upcoming(**kwargs):
    """
    Read Upcoming tasks into dicts.

    Note: unscheduled tasks with a deadline are not included here.
    See the `things.api.deadline` function instead.

    For details on parameters, see `things.api.tasks`.
    """
    return tasks(start_date="future", start="Someday", **kwargs)

things.app sets start=Someday (2) as soon as a startDate is set in any task.
I assume they do that to avoid the task beeing shown in the Anytime list.

All repeat templates also have start=Someday (2) in my real db.
But I don't see why it needs to be part of the request. (The test database contains data - maybe old? where
start is not set to 2).

Things.app shows tasks with deadline in upcoming. So upcoming could also be implemented like this, closer
to the behaviour of upcoming in things.app:

return tasks(start_date="future",  **kwargs) + tasks(deadline="future",  **kwargs) + tasks(rt1_nextInstanceStartDate="future",  **kwargs)

the latter would return the repeat templates.

[bkleinen]

After some more digging, I think only the _nextInstanceStartDate needs to be considered:

repeat templates

title	start	startDate	rt1_repeatingTemplate	rt1_instanceCreationStartDate	rt1_instanceCreationPaused	rt1_instanceCreationCount	rt1_afterCompletionReferenceDate	rt1_nextInstanceStartDate
Repating Todo: Repeat daily	2	null	null	132747392	0	1	null	null
Created as non-repeating, repeat added later	2	null	null	132747520	0	0	null	132748160
daily repeating, ends 31.10.2025 PAUSED	2	null	null	132747392	1	1	null	null
repeating 2 days after completion WAITING	2	null	null	132747520	0	1	null	null

dates = [(132747392,'2025-09-01 00:00:00','Repating Todo: Repeat daily | 2 | null | null | 132747392 | 0 | 1 | null | null |'),
         (132747520,'2025-09-02 00:00:00',"Created as non-repeating, repeat added later | 2 | null | null | 132747520 | 0 | 0 | null | 132748160 |"),
         (132748160,'2025-09-07 00:00:00',"_nextInstanceStartDate: Created as non-repeating, repeat added later | 2 | null | null | 132747520 | 0 | 0 | null | 132748160 |"),
         (132747392,'2025-09-01 00:00:00',"daily repeating, ends 31.10.2025 PAUSED | 2 | null | null | 132747392 | 1 | 1 | null | null |"),
         (132747520,'2025-09-02 00:00:00',"repeating 2 days after completion WAITING | 2 | null | null | 132747520 | 0 | 1 | null | null |"),
 ]

this task is marked as create next instance on 2.9., startdate 7.9.:

And shown in upcoming() like this:

[bkleinen]

Note: the field rt1_recurrenceRule is readable in DataGrip, this one is from Created as non-repeating, repeat added later:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>ed</key>
	<real>64092211200</real>
	<key>fa</key>
	<integer>1</integer>
	<key>fu</key>
	<integer>256</integer>
	<key>ia</key>
	<real>1757203200</real>
	<key>of</key>
	<array>
		<dict>
			<key>wd</key>
			<integer>0</integer>
		</dict>
	</array>
	<key>rc</key>
	<integer>0</integer>
	<key>rrv</key>
	<integer>4</integer>
	<key>sr</key>
	<real>1756684800</real>
	<key>tp</key>
	<integer>0</integer>
	<key>ts</key>
	<integer>0</integer>
</dict>
</plist>

[mikez]

Thank you for looking into this!

It is unclear to me what to show in things.upcoming() for TMTask repeat-templates. Since the instance hasn't been created yet, we would mimic what Things.app does in Upcoming, namely to show the repeat-template TMTask (of which an instance is created). However, its start_date is always set to NULL.

I propose the following plan. Feel free to veto or comment (@AlexanderWillner):

1. In database.py, track rt1_nextInstanceStartDate as "next_instance_start_date";
2. only include it in the output when it's actually set (i.e., in an active TMTask repeat-template) – in particular, filter out 01-01-01;
3. things.upcoming() sorts the output by start_date or next_instance_start_date, whichever is set.

That's all. No new parameters in things.tasks or the like.

[bkleinen]

For getting the desired behaviour in upcoming(), your suggestion would work.

No new parameters in things.tasks or the like.

I first thought that this was not possible without changing the default behaviour of
excluding all recurrence templates.

I digged deeper: recurrence templates have always been excluded:

IS_NOT_RECURRING = "rt1_recurrenceRule IS NULL" - it is included in all task selects, and seems to have bin there ever since the first commit, see things3.py in. 1115ef8#diff-0b1eeae268fdd8253ef021720be55a4eb45c7de90b714e486ce710e0615f0c4b

BUT all inactive recurring tasks have NULL for rt1_nextInstanceStartDate,
and all 'normal' tasks the ominous 69769.

next_date ->		NULL	NOT NULL	EITHER
template V
NULL		0	7339*	7339
NOT NULL		13	6	19
EITHER		13	7345	7348

There are 6 active and 13 inactive templates, and 7339 'normal' tasks.
*these have the ominous value 69769 rt1_nextInstanceStartDate

so the condition applied to all requests could be changed to 'rt1_nextInstanceStartDate IS NOT NULL'

• this would still change the default behaviour, though, but maybe in an acceptable way?
  the tests still pass with `IS_NOT_RECURRING = "rt1_nextInstanceStartDate IS NOT NULL"``
  (is now rather EXCLUDE_INACTIVE_RECURRING_TEMPLATES)

If the reliance on rt1_nextInstanceStartDate = 69769 seems brittle, TASK.rt1_recurrenceRule IS NULL OR TASK.rt1_nextInstanceStartDate IS NOT NULL would be more explicit. Both replacements for the condition leave the tests green: (the first one does not rely on the ominous default value).

IS_NOT_RECURRING = "( TASK.rt1_recurrenceRule IS NULL OR TASK.rt1_nextInstanceStartDate IS NOT NULL )"
#IS_NOT_RECURRING = "TASK.rt1_nextInstanceStartDate IS NOT NULL"

So, either this default behaviour needs to be changed to always include the active templates (but not the
waiting/pausing ones), or a new parameter needs to be introduced.

For the active templates (both NOT NULL), a marker like 'is_repeating_template' should be added and start_date set to rt1_nextInstanceStartDate.

(and for beeing thourough, rt1_repeatingTemplate could be set in the instances created from the templates?)
it's a key with the template uuid.

[mikez]

I'm glad to see we never included the recurring tasks. :)

I see your point on IS_NOT_RECURRING.
Let's add a parameter to things.tasks and Database.get_tasks:

is_repeating_template : bool or None, optional, **default False**

That will let you include those repeating templates as well... and only they would have the next_instance_start_date set.

P.S. 69769 is 01-01-01 (like stated earlier)