Asset Observations#

An asset observation is an event that records metadata about a given asset. Unlike asset materializations, asset observations do not signify that an asset has been mutated.

Relevant APIs#

Name	Description
AssetObservation	Dagster event indicating that an asset's metadata has been recorded.
AssetKey	A unique identifier for a particular external asset.

Overview#

AssetObservation events are used record metadata in Dagster about a given asset. Asset observation events can be logged at runtime within ops and assets. An asset must be defined using the @asset decorator or have existing materializations in order for its observations to be displayed.

Logging an AssetObservation from an Op#

To make Dagster aware that we have recorded metadata about an asset, we can log an AssetObservation event from within an op. To do this, we use the method OpExecutionContext.log_event on the context:

from dagster import AssetObservation, op


@op
def observation_op(context):
    df = read_df()
    context.log_event(
        AssetObservation(asset_key="observation_asset", metadata={"num_rows": len(df)})
    )
    return 5

We should now see an observation event in the event log when we execute this asset.

Attaching Metadata to an AssetObservation#

There are a variety of types of metadata that can be associated with an observation event, all through the MetadataEntry class. Each observation event optionally takes a dictionary of metadata entries that are then displayed in the event log and the Asset Details page. Check our API docs for MetadataEntry for more details on the types of event metadata available.

from dagster import op, AssetObservation, MetadataValue


@op
def observes_dataset_op(context):
    df = read_df()
    remote_storage_path = persist_to_storage(df)
    context.log_event(
        AssetObservation(
            asset_key="my_dataset",
            metadata={
                "text_metadata": "Text-based metadata for this event",
                "path": MetadataValue.path(remote_storage_path),
                "dashboard_url": MetadataValue.url(
                    "http://mycoolsite.com/url_for_my_data"
                ),
                "size (bytes)": calculate_bytes(df),
            },
        )
    )
    context.log_event(AssetMaterialization(asset_key="my_dataset"))
    return remote_storage_path

In the Asset Details page, we can see observations in the Asset Activity table.

Specifying a partition for an AssetObservation#

If you are observing a single slice of an asset (e.g. a single day's worth of data on a larger table), rather than mutating or creating it entirely, you can indicate this to Dagster by including the partition argument on the object.

from dagster import op, AssetMaterialization


@op(config_schema={"date": str})
def partitioned_dataset_op(context):
    partition_date = context.op_config["date"]
    df = read_df_for_date(partition_date)
    context.log_event(
        AssetObservation(asset_key="my_partitioned_dataset", partition=partition_date)
    )
    return df

Observable source assets#

SourceAsset objects may have a user-defined _observation function_ that returns a LogicalVersion. Whenever the observation function is run, an `AssetObservation` will be generated for the source asset and tagged with the returned logical version. The logical version is used in staleness calculations for downstream assets.

The observable_source_asset decorator provides a convenient way to define source assets with observation functions. The below observable source asset takes a file hash and returns it as the logical version. Every time you run the observation function, a new observation will be generated with this hash set as logical version.

from hashlib import sha256
from dagster import LogicalVersion, observable_source_asset


@observable_source_asset
def foo_source_asset(_context):
    content = read_some_file()
    hash_sig = sha256()
    hash_sig.update(bytearray(content, "utf8"))
    return LogicalVersion(hash_sig.hexdigest())

When the file content changes, the hash and therefore the logical version will change-- this will notify Dagster that downstream assets derived from an older value (i.e. a different logical version) of this source asset are stale.

Source asset observations can be triggered via the "Observe sources" button in the Dagit graph explorer view. Note that this button will only be visible if at least one source asset in the current graph defines an observation function.