Skip to content

[BUG] create_backup_dag produces IndexError if xcom and task_instance are not included in tables #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
1 task done
holly-evans opened this issue Feb 14, 2025 · 0 comments · May be fixed by #41
Open
1 task done
Labels
bug Something isn't working

Comments

@holly-evans
Copy link
Contributor

holly-evans commented Feb 14, 2025

Is there an existing issue for this?

  • I have searched the existing issues

What MWAA versions are you seeing the problem on?

2.5.1

Current Behavior

Custom DAG factories cannot create backup dags unless they include the xcom and task_instance tables.

I am following the instructions to create a custom backup dag factory, with just the active dag table. But create_backup_dag creates an import error. The import error is an IndexError or KeyError, because separate step of the backup DAG definition looks for the xcom table task and task_instance table task, which I am not using here.

Expected Behavior

The DAG imports without an error.

Steps To Reproduce

Custom factory:

class PtcDRFactory_2_4_Pause(DRFactory_2_4):
    def setup_tables(self, model):
        active_dag = self.active_dag(model)
        return [active_dag]

Using that factory to create a DAG:

from airflow import DAG
from ... import PtcDRFactory_2_4_Pause

the_pause_dag = PtcDRFactory_2_4_Pause(
    dag_id="utility_pause_active_dags",
    path_prefix="backup_metadata",
    storage_type='local_fs,'
).create_backup_dag()

DAG import errors:

  File "/usr/local/airflow/.local/lib/python3.10/site-packages/mwaa_dr/framework/factory/base_dr_factory.py", line 349, in create_backup_dag
    python_callable=[
IndexError: list index out of range

  File "/usr/local/airflow/.local/lib/python3.10/site-packages/mwaa_dr/framework/factory/base_dr_factory.py", line 354, in create_backup_dag
    xcom_task >> export_tasks["task_instance"]
KeyError: 'task_instance'

Successful custom DAG factory:

class PtcDRFactory_2_4_Pause(DRFactory_2_4):
    def setup_tables(self, model):
        active_dag = self.active_dag(model)
        xcom = self.xcom(model)
        task_instance = self.task_instance(model)
        return [active_dag, xcom, task_instance]

Anything else?

I am actually using 2.4.3 MWAA but that is not an option.

Relevant sections:

    def create_backup_dag(self) -> DAG:
        ...

        with TaskGroup(group_id="export_tables", dag=dag) as export_tables_t:
            export_tasks = {}
            for table in self.tables():
                if table.name != "xcom":
                    export_task = PythonOperator(
                        task_id=f"export_{table.name}s",
                        python_callable=table.backup,
                        dag=dag,
                    )
                    export_tasks[table.name] = export_task

            xcom_task = PythonOperator(
                task_id="export_xcoms",
                python_callable=[
                    table.backup for table in self.tables() if table.name == "xcom" <<<<<
                ][0],
                dag=dag,
            )
            xcom_task >> export_tasks["task_instance"] <<<<<
@holly-evans holly-evans added the bug Something isn't working label Feb 14, 2025
@holly-evans holly-evans changed the title [BUG] create_backup_dag produces IndexError if xcom table is not included in tables [BUG] create_backup_dag produces IndexError if xcom and task_instance are not included in tables Feb 14, 2025
@holly-evans holly-evans linked a pull request Feb 15, 2025 that will close this issue
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant