dbnd.log_dataset_op

dbnd.log_dataset_op(op_path: Union[targets.base_target.Target, str], op_type: Union[dbnd._core.constants.DbndDatasetOperationType, str], *, success: bool = True, error: Optional[str] = None, data: Optional[Any] = None, with_histograms: Optional[bool] = None, with_partition: Optional[bool] = None, with_stats: Optional[bool] = True, with_preview: bool = False, with_schema: bool = True, send_metrics: bool = True, row_count: Optional[int] = None, column_count: Optional[int] = None)

Logs dataset operation and meta data to dbnd.

Parameters

  • op_path – Target object to log or a unique path representing the operation location.

  • op_type – Type of operation that been done with the target - read, write, delete.

  • success – True if the operation succeeded, False otherwise.

  • error – Optional error message.

  • data – Optional value of data to use build meta-data on the target.

  • with_histograms – Should calculate histogram of the given data - relevant only with data param. - Boolean to calculate or not on all the data columns.

  • with_stats – Should extract schema of the data as meta-data of the target - relevant only with data param. - Boolean to calculate or not on all the data columns.

  • with_partition – If True, the webserver tries to detect partitions of our datasets and extract them from the path, otherwise not manipulating the dataset path at all.

  • with_preview – Should extract preview of the data as meta-data of the target - relevant only with data param.

  • with_schema – Should extract schema of the data as meta-data of the target - relevant only with data param.

  • send_metrics – Should report preview, schemas and histograms as metrics.

  • row_count – should report row count no matter what is the data

  • column_count – should report column count no matter what is the data

Example:

@task
def prepare_data():
    log_dataset_op(
        "/path/to/value.csv",
        DbndDatasetOperationType.read,
        data=pandas_data_frame,
        with_preview=True,
        with_schema=True,
    )