Pulumi preview and up incorrectly says that it will change / is changing a service

[JiriKovar]

Describe what happened

Hello,

In case of some of our service, the pulumi preview and pulumi up commands shows that it will do "something" with a service (the service root is flagged with ~, prints details of the service, but in the details, nothing is flagged to be changed (no line starts with ~/-/+).

In case of pulumi up, it says that its updating the services, the service is listed in the number of the updated services, but at the end of the day, it does nothing at all = not even a new version of the fastly service is created, which shows that at the runtime the Pulumi detects that there is no change - version management requires extra requests AFAIK and its not running even these requests. Even so, the next round of pulumi preview and pulumi up shows the same results.

Have you ever observed the same behaviour? Do you have any idea what could be the cause? In case of some of the types of our Fastly services, it happens only in certain configuration (for QA configurations some objects like ACLs or WAF are missing), in some of them it doesn't happen (even though they have ACLs and WAF), in some of them it happens consistently no matter the configuration.

This has been bothering us for quite a while, we had a couple of theories, but none of them proved to be right - it could be some handling of whitespaces, or whatnot. I'm reporting it as a bug, since the logs and the actual behaviour does not match, but at the same time I'm asking for assistance, because I cannot pinpoint the cause and give you a reasonable repro.

Sample program

import { ServiceVcl } from '@pulumi/fastly';
import type {
  ServiceVclBackend,
  ServiceVclCacheSetting,
  ServiceVclDictionary,
  ServiceVclDomain,
  ServiceVclHeader,
} from '@pulumi/fastly/types/input';
import * as pulumi from '@pulumi/pulumi';
import { WebTest, type WebTestArgs } from '../../../@components/azure/webTest';
import type { FastlyMonitoringConfig } from '../../../@components/fastly/fastlyMonitoringConfig';
import type { Domains, ServiceBackend } from '../../../@components/fastly/types';
import { Waf, type WafConfig } from '../../../@components/fastly/waf/waf';
import type { ResourceContext } from '../../../@components/utils/resourceContext';

export interface FastlyServiceWhereItAlwaysHappensArgs {
  context: ResourceContext;
  domains: Domains;
  monitoring: FastlyMonitoringConfig | undefined;
  serviceBackend: ServiceBackend;
  waf: WafConfig | undefined;
}

export class FastlyServiceWhereItAlwaysHappens {
  public readonly id: pulumi.Output<string>;
  public readonly name: pulumi.Output<string>;
  public readonly primaryDomain: pulumi.Output<string>;

  constructor(name: string, args: FastlyServiceWhereItAlwaysHappensArgs) {
    const domains: ServiceVclDomain[] = [args.domains.primary, ...args.domains.other].map((domain) => ({
      name: domain,
    }));
    const primaryDomain = args.domains.primary;

    const cacheSettings: ServiceVclCacheSetting[] = [
      {
        name: 'Do not cache',
        action: 'pass',
        ttl: 0,
        staleTtl: 0,
      },
    ];

    const backends: ServiceVclBackend[] = [
      {
        address: args.serviceBackend.address,
        name: `Backend_${args.serviceBackend.name.replace(/[^a-zA-Z0-9]+/, '_')}`,
        overrideHost: args.serviceBackend.address,
        port: 443,
        useSsl: true,
        sslCertHostname: '*.azurewebsites.net',
        autoLoadbalance: false,
        betweenBytesTimeout: 20000,
        connectTimeout: 20000,
        firstByteTimeout: 300000,
      },
    ];

    const headers: ServiceVclHeader[] = [
      {
        name: 'X-Azure-FDID',
        action: 'set',
        type: 'request',
        destination: 'http.X-Azure-FDID',
        source: pulumi.interpolate`\"${args.serviceBackend.azureFrontDoorId}\"`,
        priority: 10,
      },
    ];

    const dictionaries: ServiceVclDictionary[] = [
      {
        name: 'Edge_Security',
      },
    ];

    const serviceVcl = new ServiceVcl(
      name,
      {
        backends,
        cacheSettings,
        defaultTtl: 3600,
        dictionaries,
        domains,
        headers,
        dynamicsnippets: [],
      },
      {
        ignoreChanges: ['versionComment', 'dynamicsnippets'],
      },
    );

    if (args.waf) {
      new Waf(
        `${name}-waf`,
        {
          ...args.waf,
          origins: backends.map((b) => b.address),
          serviceId: serviceVcl.id,
        },
        {
          parent: serviceVcl,
        },
      );
    }

    if (args.monitoring) {
      for (const domain of [primaryDomain, ...args.domains.other]) {
        new WebTest(
          `subscription-cdn-webtest-${domain.replace(/[^a-zA-Z0-9]/g, '-').toLowerCase()}`,
          {
            resourceGroup: args.monitoring.resourceGroup,
            webTestProperties: {
              name: `subscription-${domain.replace(/[^a-zA-Z0-9]/g, '-').toLowerCase()}`,
              url: `https://${domain}/`,
              expectedHttpStatusCode: 200,
            },
            azureSubscriptionId: args.monitoring.azureSubscriptionId,
            applicationInsights: args.monitoring.applicationInsights,
            alert: {
              description: '      Website is DOWN',
              monitoringEmails: [args.monitoring.emails.service],
              actionGroups: [args.monitoring.actionGroupId.azureAlertMessageFormatter],
              severityLevel: 3,
            },
          } as WebTestArgs,
          {
            parent: serviceVcl,
          },
        );
      }
    }

    this.id = serviceVcl.id;
    this.name = serviceVcl.name;
    this.primaryDomain = pulumi.output(primaryDomain);
  }
}

Log output

@ updating....
    ~ fastly:index/serviceVcl:ServiceVcl: (update)
        [id=<REDACTED_SERVICE_ID>]
        [urn=urn:pulumi:qa::persistent-fastly::fastly:index/serviceVcl:ServiceVcl::qa-serviceWhereItAlwaysHappens]
        [provider=urn:pulumi:qa::persistent-fastly::pulumi:providers:fastly::default_8_11_0::021b2e85-0f62-46e3-9bea-a3381e86b531]
        activate       : true
        backends       : [secret]
        cacheSettings  : [
            [0]: {
                action        : "pass"
                cacheCondition: ""
                name          : "Do not cache"
                staleTtl      : 0
                ttl           : 0
            }
        ]
        comment        : "Managed by Terraform"
        defaultTtl     : 3600
        dictionaries   : [
            [0]: {
                forceDestroy: false
                name        : "Edge_Security"
                writeOnly   : false
            }
        ]
        domains        : [
            [0]: {
                name      : "serviceWhereItAlwaysHappens.ourinternaldomain.com"
            }
            [1]: {
                name      : "qa-serviceWhereItAlwaysHappens.global.ssl.fastly.net"
            }
        ]
        dynamicsnippets: []
        headers        : [
            [0]: {
                action           : "set"
                cacheCondition   : ""
                destination      : "http.X-Azure-FDID"
                ignoreIfSet      : false
                name             : "X-Azure-FDID"
                priority         : 10
                requestCondition : ""
                responseCondition: ""
                source           : "\"<REDACTED_GUID>\""
                type             : "request"
            }
        ]
        http3          : false
        name           : "qa-serviceWhereItAlwaysHappens"
        staleIfError   : false
        staleIfErrorTtl: 43200
Resources:
    ~ 1 updated
    116 unchanged

Affected Resource(s)

No response

Output of pulumi about

CLI          
Version      3.136.1
Go Version   go1.23.2
Go Compiler  gc

Plugins
KIND      NAME    VERSION
language  nodejs  unknown

Host     
OS       Microsoft Windows Server 2022 Standard
Version  10.0.20348 Build 20348
Arch     x86_64

This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v22.9.0'

Backend        
Name           Agent-06
URL            azblob://qastacks?storage_account=***
User           Agent-06\Administrator
Organizations  
Token type     personal

Pulumi locates its logs in C:\Users\ADMINI~1\AppData\Local\Temp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[JiriKovar]

FYI I have also tried to run pulumi refresh --diff --skip-preview --non-interactive -j and analyzed the JSON output, where there are no differences between the "new" and the "old" objects in the results are the same and the "detailedDiff" is null.

[guineveresaenger]

Hi @JiriKovar, thank you for reporting this. We do have a category of bugs involving spurious diffs, and we understand they're confusing an unwieldy for our users.

Some of these bugs stem from the pulumi-terraform bridge rather than an individual provider. In fact, we're currently working on improved diffing logic in the bridge.
Other bugs of this type are rooted in the upstream provider implementation, such as this one on ServiceVcl bigquery logging.

Even though they may have different causes, we'd love it if you could send us a sample program with such an unexplained diff.

[JiriKovar]

Hi,

interresting is that we used to manage these Fastly services via Terraform until recently and with Terraform, we werent having these issues. The service in mention does not use any logging and the service itself is as simple as mentioned in the aforementioned code. I will try to come up with a minimal repro when I have a few moments to spare, but I have a couple of questions that could help me with that:

• The objects dependant on the very VLC service can cause this. Is this correct? (In this case most probably the WAF configuration that adds a dynamic snippet, even though its specified to ignore them).
• It might also be caused by some kind of incorrect interpolation in some of the attributes. Is this correct?
• Do you have any other ideas for source of this issue?

Thank you very much!

[guineveresaenger]

Hi @JiriKovar -

The objects dependant on the very VLC service can cause this. Is this correct? (In this case most probably the WAF configuration that adds a dynamic snippet, even though its specified to ignore them).

It is possible; it is also possible that ignoreChanges isn't working as we think it should. The fact that you see no diff between the olds and the news makes me think this is unlikely.

It might also be caused by some kind of incorrect interpolation in some of the attributes. Is this correct?
I don't think that should cause a spurious diff, but again, a repro would hlep here.

it is helpful to know that you do not see this issue in Terraform. There are some significant differences between the way Pulumi creates a diff preview and the way Terraform does, and you may be tripping over some of them.

This leads me to believe this is almost certainly a bug on our end with Diff. Please update here for a repro once you have it.

In the meantime... I don't think we've addressed anything quite like this, but the team has put in a lot of work on Diff in the bridge in the last several weeks. I've triggered a patch release of this provider for you to try out. Once https://github.com/pulumi/pulumi-fastly/actions/runs/11674513405 is done building, try v8.12.2 and let us know if your troubles persist.

[JiriKovar]

Hello @guineveresaenger,

Unfortunatelly the patch v8.12.2 produces the same results.

And even worse, I was unable to come up with a reliable minimal repro. Parallel to that I was tryting to remove some parts of the affected service and I have noticed that whenever I remove WAF from an affected service, it fixed the issue, until I put the WAF back. But at the same time, not all services that use WAF are affected (which is why I dont have a reliable minimal repro yet).

Another little piece of information is that when I create the Fastly Service with WAF, everything is OK (pulumi preview and pulumi up shows no changes) until the first refresh is run, then it starts behaving according to the issue. This should mean that it is not something between the pulumi code and stack, but rather something that changes with the first refresh. I have tried to compare the stacks before and after the refresh and it could be summarized as:

• enabling WAF publishes a new version that the pulumi state does not know about until the refresh (this is presumably due to the dynamic snippets that needs to be added)
• backend inputs get encripted as secrets (they are written in the stack in the plain text after the apply which is probably OK and to be expected)
• the version and modified date gets updated to match the aforementioned new version created by WAF
• the empty propertyDependencies object is removed (for some reason pulumi adds it with each apply and removes it with each refresh)

This list that it is indeed caused by something around either WAF, or dynamic snippets.

Based on that I'm thinking it might be the best option to move this conversation to the slack / Fastly support ticket, where we could safely share customer IDs and Fastly service IDs so that you can possibly look at the differences between the affected and unaffected services. Is that OK with you?