Skip to content

Improve database handling #24

@Trippy3

Description

@Trippy3
  1. If you are creating a new connection, create the table structure in advance.
    if self.mode == DBMode.OVERWRITE:
            # Delete files first because duckdb.connect() does not have an overwrite option.
            self.path.unlink(missing_ok=True)
            return create_table()
    return ddb.connect(str(self.path))

add create_table().
2. Fix write_data (Assumption that 1. is correctly implemented)

def write_data(self, src: pl.LazyFrame) -> ddb.DuckDBPyConnection:
        def get_only_additional(new: pl.LazyFrame) -> pl.LazyFrame:
            original = self.to_ldf()
            return new.join(original, on="number", how="anti")

        if self.mode == DBMode.DELTA:
            src = get_only_additional(src)
        self.conn.sql(f"INSERT INTO {self.top_table} SELECT * FROM src")
        return self.conn

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions