star globs do not work in generates attribute

[superg]

Description

I am working on setting up a task that generates sqlc source code.
According to documentation in https://taskfile.dev/reference/schema/#task, generates attribute supports star globs when using timestamp method, however this is not what I observe.
First generation with task sqlc creates output files in output. After deleting any file that matches output/*.sql.go, for example output/asdf.sql.go and rerunning task sqlc, the file is not getting regenerated. When instead of star glob I enumerate all output files manually, task sqlc works as expected but it's impractical for my case where a have a lot of files.

Expected:

task sqlc
task: [sqlc] sqlc generate -f sqlc.yaml

Happened:

task sqlc
task: Task "sqlc" is up to date

Version

v3.39.2 (h1:Zt7KXHmMNq5xWZ1ihphDb+n2zYLCo4BdRe09AnMMIgA=)

Operating system

Ubuntu 24.04

Experiments Enabled

No response

Example Taskfile

version: '3'

tasks:
  sqlc:
    method: timestamp
    sources:
      - sqlc.yaml
      - sqlc/*.sql
    generates:
      - output/*.sql.go
    cmds:
      - sqlc generate -f sqlc.yaml

[superg]

Just to clarify, it also happens for the latest version:

task --version
3.44.0

[deicon]

Looks like it considers only the presence of the timestamp file but does not compare it with any other timestamp.

So after deleting every file within output folder of your example, it still thinks its up to date.

`go
timestampFile := checker.timestampFilePath(t)

// If the file exists, add the file path to the generates.
// If the generate file is old, the task will be executed.
_, err = os.Stat(timestampFile)
if err == nil {
	generates = append(generates, timestampFile)
} else {
	// Create the timestamp file for the next execution when the file does not exist.
	if !checker.dry {
		if err := os.MkdirAll(filepath.Dir(timestampFile), 0o755); err != nil {
			return false, err
		}
		f, err := os.Create(timestampFile)
		if err != nil {
			return false, err
		}
		f.Close()
	}
}

`

[deicon]

To Reproduce

version: '3'

tasks:
  default:
    method: timestamp
    sources:
      - sqlc.yaml
      - sqlc/*.sql
    generates:
      - output/sqlc/*.sql
    cmds:
      - for: sources
        cmd: mkdir -p "$(dirname output/{{ .ITEM }})" && touch output/{{ .ITEM }};

❯ task
task: [default] mkdir -p output/sqlc/demo.sql && touch output/sqlc/demo.sql
task: [default] mkdir -p output/sqlc/demow.sql && touch output/sqlc/demow.sql
❯ task
task: Task "default" is up to date
❯ rm -rf output
❯ task
task: Task "default" is up to date
❯ tree .task
.task
└── timestamp
└── default
❯ rm .task/timestamp/default
❯ task
task: [default] mkdir -p output/sqlc/demo.sql && touch output/sqlc/demo.sql
task: [default] mkdir -p output/sqlc/demow.sql && touch output/sqlc/demow.sql

[deicon]

I would also argue, that generates with checksum does not behave as expected.
The check tries to find the files in the generates list. So, if any file given in the generates list is not present, it will run the task. But in case of Glob Pattern, the list only contains files, which are present. Meaning, it does not work as expected, if, as in this case the directory was cleared

Not sure what to expect, but if generates is defined as a star glob, i would assume, that this needs to keep more info about the last run, in order to determine the upToDate state

[superg]

I specifically used timestamp method because documentation mentioned that it supports glob patterns. I tried with the default method (checksum) before and it didn't work so I assumed it's implemented only for the timestamp method.
For my use case I would not care which method to use as long as I get the behavior where I can use glob patterns in generates as it'll greatly simplify a lot of my workflows.

[deicon]

#2181 Already went into details about why the current behaviour seems to confuse a lot of users.

[deicon]

Looking at the Sources and digging through discord, one could argue, that the behaviour we are seeing might be confusing but works as designed.

Method timestamp compares the the max timestamp of all currently available sources (using glob) against the max timestamp of the available generates (using glob).
So one can simply not trigger re-execution by just deleting either sources nor generates as the max timestamp will stay the same. Which kind of makes sense and is confusing at the same time.

I believe the reasoning behind this might be, that the task itself is the only component changing the generates based on the sources. If an out of band change on the output happens, is this nothing the task cares about. The sources are still unchanged.

If one wants to react on changes in the generates one might use status: and come up with a shell script detecting the changes.
The following example detects newer sources as well as missing outputs.

version: '3'

tasks:
  default:
    method: timestamp
    sources:
      - sqlc/*.sql
    generates:
      - output/sqlc/*.sql
    cmds:
      - for: sources
        cmd: mkdir -p "$(dirname output/{{ .ITEM }})" && touch output/{{ .ITEM }} ;
    status:
      - |
        find sqlc -name '*.sql' | while read f; do [ -f "output/$f" ] || { echo "Missing: output/${f}"; exit 1; }; done

[optimuspaul]

I am using the timestamp method and it never executes the task after the first time, even if I delete the generated files and specify the each by name. This feature seems completely broken.

[superg]

@deicon thanks for the details on timestamp method. While it works as designed, it doesn't really resolve the problem of setting up direct source->destination generated files relation so task executes only when needed.
I spent some time reading the links you shared and implemented generates support in the default checksum method: #2313

[lucas-labs]

It might be a different issue (I'm sorry if that's the case), but I can't get the sources/generates functionality to work in any way.

e.g.

version: '3'

tasks:
  test:
    dir: .
    label: Test
    cmds:
      - cp foo.txt bar.txt
      - echo "copied foo.txt to bar.txt"
    sources:
      - foo.txt
    generates:
      - bar.txt

I expected the first run would copy foo.txt to bar.txt, and then while foo.txt remains the same and bar.txt exists, it will not run the cmds again.

But:

$ task --version       
3.44.0

$ task test
task: [Test] cp foo.txt bar.txt
task: [Test] echo "copied foo.txt to bar.txt"
copied foo.txt to bar.txt

$ ls
bar.txt   foo.txt   Taskfile.yml

$ task test
task: [Test] cp foo.txt bar.txt
task: [Test] echo "copied foo.txt to bar.txt"
copied foo.txt to bar.txt

$ task test
task: [Test] cp foo.txt bar.txt
task: [Test] echo "copied foo.txt to bar.txt"
copied foo.txt to bar.txt

$ task test
task: [Test] cp foo.txt bar.txt
task: [Test] echo "copied foo.txt to bar.txt"
copied foo.txt to bar.txt

The task is being run every single time it's called.