[WIP] Incremental Watched Queries

[stevensJourney]

Overview

Our current Watched query implementations emit results whenever a change to a dependant SQLite table occurs. The table changes might not affect the query result set, but we still query and emit a new result set for each table change. The result sets typically contain the same data, but these results are new Array/object references which will cause re-renders in certain frameworks like React.

This PR overhauls, improves and extends upon the existing watched query implementations by introducing incremental watched queries.

The first version of incremental watched queries operates on the principle of comparing result sets. Internally we still query on each dependant table change, but we compare the previous and new result sets - only emitting values if there is a change.

Implementation

The logic required for incrementally watched queries requires additional computation and introduces additional complexity to the implementation. For these reasons a new concept of a WatchedQuery class is introduced, along with a new incrementalWatch method allows building an instance of WatchedQuery.

Note: APIs here are still a work in progress.

// Create an instance of a WatchedQuery.
// This instance can be long-lived and shared across multiple components.
const assetsQuery = powersync
  .incrementalWatch({ mode: IncrementalWatchMode.COMPARISON })
  .build({
    watch: {
      // The query executor is customizable. The watched query can return any data structure. A helper is provided for simple queries
      query: new GetAllQuery({
        sql: 'SELECT * FROM assets',
        parameters: []
      }),
      // The initial state of the data. This should be of the form returned by the executor.
      placeholderData: []
    }
});

The assetsQuery is smart, it:

• Automatically reprocesses itself if the PowerSync schema has been updated with updateSchema
• Automatically closes itself when the PowerSync client has been closed.
• Allows for the query parameters to be updated after instantiation.
• Allows shared subscribing to state changes.

// The subscribe method can be used multiple times to listen for updates.
// The returned dispose function can be used to unsubscribe from the updates.
const disposeSubscriber = assetsQuery.subscribe({
  onData: (data) => {
    // This callback will be called whenever the data changes.
    // The data is the result of the executor.
    console.log('Data updated:', data);
  },
  onStateChange: (state) => {
    // This callback will be called whenever the state changes.
    // The state contains metadata about the query, such as isFetching, isLoading, etc.
    console.log(
      'State changed:', 
      state.error, 
      state.isFetching, 
      state.isLoading, 
      state.data
    );
  },
  onError: (error) => {
    // This callback will be called if the query fails.
    console.error('Query error:', error);
  }
});

WatchedQuery instances retain the latest state in memory. Sharing WatchedQuery instances can be used to introduce caching and reduce the number of duplicate DB queries between components.

The incremental logic is customisable. Comparison based queries can specify custom logic for performing comparisons on the relevant data set. By default a JSON.stringify approach is used. Different data sets might have more optimal implementations.

const assetsQuery = powersync
  .incrementalWatch({ mode: IncrementalWatchMode.COMPARISON })
  .build({
    watch: {
      /// ....
    },
    comparator: new ArrayComparator({
      compareBy: (item) => JSON.stringify(item)
    })
});

Updates to query parameters can be performed in a single place, affecting all subscribers.

assetsQuery.updateSettings({
  query: {
    compile: () => ({
        sql: `SELECT * FROM assets OFFSET ? LIMIT 100`,
        parameters: [
          newOffset
        ]
      }),
      execute: ({ sql, parameters }) => db.getAll(sql, parameters)
    },
    placeholderData: [],
  });

Reactivity

The existing watch method and Reactivity packages have been updated to use incremental queries with comparison defined as an opt-in feature (defaults to no changes).

powersync.watch(
  'select * from assets',
  [],
  {
    onResult: () => {
      // This callback will be called whenever the data changes.
      console.log('Data updated');
    }
  },
  {
    comparator: {
      checkEquality: (current, previous) => {
        // This comparator will only report updates if the data changes.
        return JSON.stringify(current) === JSON.stringify(previous);
      }
    }
  }
);

/// React hooks
useQuery('SELECT * FROM lists WHERE name = ?', ['aname'], {
  comparator: new ArrayComparator({
    compareBy: (item) => JSON.stringify(item)
})

New hooks have also been added to use shared WatchedQuery instances.

const assetsQuery = powersync.incrementalWatch({
/// ....
});

/// In the component
export const MyComponet = () => {
   // In React one could import the `assetsQuery` or create a context provider for various queries
   const { data } = useWatchedQuerySubscription(assetsQuery)
}

Still to Come

The comparison processor is just one method for performing incremental queries. SQLite trigger based methods will also be introduced shortly.

[changeset-bot]

🦋 Changeset detected

Latest commit: 8a42cda

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 7 packages

Name	Type
@powersync/common	Minor
@powersync/web	Minor
@powersync/node	Patch
@powersync/op-sqlite	Patch
@powersync/react-native	Patch
@powersync/tanstack-react-query	Patch
@powersync/diagnostics-app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[stevensJourney]

This fixes an issue where the docs state the useStatus is deprecated due to usePowerSyncStatus being deprecated.