dbnd.dataset_op_logger

dbnd.dataset_op_logger(op_path: Union[targets.base_target.Target, str], op_type: Union[dbnd._core.constants.DbndDatasetOperationType, str], *, data: Optional[Any] = None, with_histograms: Optional[bool] = None, with_partition: Optional[bool] = None, with_stats: Optional[bool] = True, with_preview: bool = False, with_schema: bool = True, send_metrics: bool = True, row_count: Optional[int] = None, column_count: Optional[int] = None)

Wrapper to Log dataset operation and meta data to dbnd.

Make sure to only wrap operation related code!

Parameters

  • op_path – Target object to log or a unique path representing the target logic location.

  • op_type – Type of operation that been done with the dataset - read, write, delete.

  • data – Optional value of data to use build meta-data on the dataset.

  • with_histograms – Should calculate histogram of the given data - relevant only with data param. - Boolean to calculate or not on all the data columns.

  • with_stats – Should extract schema of the data as meta-data of the target - relevant only with data param. - Boolean to calculate or not on all the data columns.

  • with_partition – If True, the webserver tries to detect partitions of our datasets and extract them from the path, otherwise not manipulating the dataset path at all.

  • with_preview – Should extract preview of the data as meta-data of the dataset - relevant only with data param.

  • with_schema – Should extract schema of the data as meta-data of the dataset - relevant only with data param.

  • send_metrics – Should report preview, schemas and histograms as metrics.

  • row_count – should report row count no matter what is the data

  • column_count – should report column count no matter what is the data

Good Example:

with dataset_op_logger("location://path/to/value.csv", "read"):
    value = read_from()
    # Read is successful

unrelated_func()

Bad Example:

with dataset_op_logger("location://path/to/value.csv", "read"):
    value = read_from()
    # Read is successful
    unrelated_func()
    # If unrelated_func raises an exception, failed read operation is reported to databand.