The performance of loading the Definitions snapshot from a code server when large @multi_asset s are in use has been drastically improved.
The snowflake quickstart example project now renames the “by” column to avoid reserved snowflake names. Thanks @jcampbell!
The existing group name (if any) for an asset is now retained if the_asset.with_attributes is called without providing a group name. Previously, the existing group name was erroneously dropped. Thanks @ion-elgreco!
[dagster-dbt] Fixed an issue where Dagster events could not be streamed from dbt source freshness.
[dagster university] Removed redundant use of MetadataValue in Essentials course. Thanks @stianthaulow!
[ui] Increased the max number of plots on the asset plots page to 100.
The tag_keys argument on DagsterInstance.get_run_tagsis no longer optional. This has been done to remove an easy way of accidentally executing an extremely expensive database operation.
The maximum number of concurrent runs across all branch deployments is now configurable. This setting can now be set via GraphQL or the CLI.
[ui] In Insights, fixed display of table rows with zero change in value from the previous time period.
[ui] Added deployment-level Insights.
[ui] Fixed an issue causing void invoices to show up as “overdue” on the billing page.
[experimental] Branch deployments can now indicate the new and modified assets in the branch deployment as compared to the main deployment. To enable this feature, turn on the “Enable experimental branch deployment asset graph diffing” user setting.
[ui] When viewing logs for a run, the date for a single log row is now shown in the tooltip on the timestamp. This helps when viewing a run that takes place over more than one date.
Added suggestions to the error message when selecting asset keys that do not exist as an upstream asset or in an AssetSelection.
Improved error messages when trying to materialize a subset of a multi-asset which cannot be subset.
[dagster-snowflake] dagster-snowflake now requires snowflake-connector-python>=3.4.0
[embedded-elt] @sling_assets accepts an optional name parameter for the underlying op
[dagster-openai] dagster-openai library is now available.
[dagster-dbt] Added a new setting on DagsterDbtTranslatorSettings called enable_duplicate_source_asset_keys that allows users to set duplicate asset keys for their dbt sources. Thanks @hello-world-bfree!
Log messages in the Dagster daemon for unloadable sensors and schedules have been removed.
[ui] Search now uses a cache that persists across pageloads which should greatly improve search performance for very large orgs.
[ui] groups/code locations in the asset graph’s sidebar are now sorted alphabetically.
Fixed issue where the input/output schemas of configurable IOManagers could be ignored when providing explicit input / output run config.
Fixed an issue where enum values could not properly have a default value set in a ConfigurableResource.
Fixed an issue where graph-backed assets would sometimes lose user-provided descriptions due to a bug in internal copying.
[auto-materialize] Fixed an issue introduced in 1.6.7 where updates to ExternalAssets would be ignored when using AutoMaterializePolicies which depended on parent updates.
[asset checks] Fixed a bug with asset checks in step launchers.
[embedded-elt] Fix a bug when creating a SlingConnectionResource where a blank keyword argument would be emitted as an environment variable
[dagster-dbt] Fixed a bug where emitting events from dbt source freshness would cause an error.
[ui] Fixed a bug where using the “Terminate all runs” button with filters selected would not apply the filters to the action.
[ui] Fixed an issue where typing a search query into the search box before the search data was fetched would yield “No results” even after the data was fetched.
[asset checks] UI performance of asset checks related pages has been improved.
[dagster-dbt] The class DbtArtifacts has been added for managing the behavior of rebuilding the manifest during development but expecting a pre-built one in production.
Microsoft Teams is now supported for alerts. Documentation
A send sample alert button now exists on both the alert policies page and in the alert policies editor to make it easier to debug and configure alerts without having to wait for an event to kick them off.
Added a new run_retries.retry_on_op_or_asset_failures setting that can be set to false to make run retries only occur when there is an unexpected failure that crashes the run, allowing run-level retries to co-exist more naturally with op or asset retries. See the docs for more information.
dagster dev now sets the environment variable DAGSTER_IS_DEV_CLI allowing subprocesses to know that they were launched in a development context.
[ui] The Asset Checks page has been updated to show more information on the page itself rather than in a dialog.
[ui] Fixed an issue where the UI disallowed creating a dynamic partition if its name contained the “|” pipe character.
AssetSpec previously dropped the metadata and code_version fields, resulting in them not being attached to the corresponding asset. This has been fixed.
The new @multi_observable_source_asset decorator enables defining a set of assets that can be observed together with the same function.
[dagster-embedded-elt] New Asset Decorator @sling_assets and Resource SlingConnectionResource have been added for the [dagster-embedded-elt.sling](http://dagster-embedded-elt.sling) package. Deprecated build_sling_asset, SlingSourceConnection and SlingTargetConnection.
Added support for op-concurrency aware run dequeuing for the QueuedRunCoordinator.
dagster-polars has been added as an integration. Thanks @danielgafni!
[dagster-dbt] @dbt_assets now supports loading projects with semantic models.
[dagster-dbt] @dbt_assets now supports loading projects with model versions.
[dagster-dbt] get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!
[dagster-duckdb] The Dagster DuckDB integration supports DuckDB version 0.10.0.
[UPath I/O manager] If a non-partitioned asset is updated to have partitions, the file containing the non-partitioned asset data will be deleted when the partitioned asset is materialized, rather than raising an error.
Fixed an issue where creating a backfill of assets with dynamic partitions and a backfill policy would sometimes fail with an exception.
Fixed an issue with the type annotations on the @asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!
[ui] On the asset graph, nodes are slightly wider allowing more text to be displayed, and group names are no longer truncated.
[ui] Fixed an issue where the groups in the asset graph would not update after an asset was switched between groups.
[dagster-k8s] Fixed an issue where setting the security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!
Observable source assets can now yield ObserveResults with no data_version.
You can now include FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.
[ui] In Dagster Cloud, a new feature flag allows you to enable an overhauled asset overview page with a high-level stakeholder view of the asset’s health, properties, and column schema.
[kubernetes] Fixed an issue where the Kubernetes agent would sometimes leave dangling kubernetes services if the agent was interrupted during the middle of being terminated.
Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order of partition key, instead of in an unpredictable order.
[dagster-k8s] Include k8s pod debug info in run worker failure messages.
[dagster-dbt] Events emitted by DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.
A previous change prevented asset backfills from grouping multiple assets into the same run when using BackfillPolicies under certain conditions. While the backfills would still execute in the proper order, this could lead to more individual runs than necessary. This has been fixed.
[dagster-k8s] Fixed an issue introduced in the 1.6.4 release where upgrading the Helm chart without upgrading the Dagster version used by user code caused failures in jobs using the k8s_job_executor.
[instigator-tick-logs] Fixed an issue where invoking context.log.exception in a sensor or schedule did not properly capture exception information.
[asset-checks] Fixed an issue where additional dependencies for dbt tests modeled as Dagster asset checks were not properly being deduplicated.
[dagster-dbt] Fixed an issue where dbt model, seed, or snapshot names with periods were not supported.
@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.
[auto-materialize] A new AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.
[Global op/asset concurrency] Ops and assets now take run priority into account when claiming global op/asset concurrency slots.
Auto-materialize history – We’ve added a UI that tracks why assets were or were not materialized according to theirAutoMaterializePolicy. It’s located under Assets → Select an asset with an AutoMaterializePolicy → Auto-materialize history tab.
Auto-materialize performance – We’ve made significant performance improvements to the Asset Daemon, allowing it to keep up with asset graphs containing thousands of assets and assets with a large history of previously-materialized partitions.
Asset backfill cancellation — Asset backfills can now be canceled, bring them to parity with job backfills. When an asset backfill is requested for cancellation, the daemon cancels runs until all runs are terminated, then marks the backfill as “canceled”.
non_argument_deps → deps – We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.
Group-level asset status UI – the new Assets Overview dashboard, located underneath the Activity tab of the Overview page, shows the status all the assets in your deployment, rolled up by group.
Op concurrency (experimental) — We’ve added a feature that allows limiting the number of concurrently executing ops across runs. [docs]
DynamicPartitionsDefinition and SensorResult are no longer marked experimental.
Automatically observe source assets, without defining jobs (experimental – The @observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval. Downstream assets with eager auto-materialize policies will automatically run if the observation function indicates that the source asset has changed. [docs]
Dagit → Dagster UI – To reduce the number of Dagster-specific terms that new users need to learn when learning Dagster, “Dagit” has been renamed to the “The Dagster UI”. The dagit package is deprecated in favor of the dagster-webserver package.
Default config in the Launchpad - When you open the launchpad to kick off a job or asset materialization, Dagster will now automatically populate the default values for each field.
The new @dbt_assets decorator allows much more control over how Dagster runs your dbt project. [docs]
The new dagster-dbt project scaffold command line interface makes it easy to create files and directories for a Dagster project that wraps an existing dbt project.
Improved APIs for defining asset dependencies – The new get_asset_key_for_model and get_asset_key_for_source utilities make it easy to specify dependencies between upstream dbt assets and downstream non-dbt assets. And you can now more easily specify dependencies between dbt models and upstream non-dbt assets by specifying Dagster asset keys in the dbt metadata for dbt sources.
The published Dagster Docker images now use Python 3.10, instead of 3.7.
We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.
The UPathIOManager can now be extended to load multiple partitions asynchronously (Thanks Daniel Gafni!).
By default, Dagster will now automatically load default config values into the launchpad. This behavior can be disabled in the user settings page.
[dagster-k8s] The Helm chart now sets readiness probes on user code deployment servers by default. These can be disabled with dagster-user-deployments.deployments.[...].readinessProbe.enabled=false.
[dagster-airbyte] In line with the deprecation of non_argument_deps in favor of deps, build_airbyte_assets now accepts a deps parameter.
[dagstermill] In line with the deprecation of non_argument_deps in favor of deps, define_dagstermill_asset now accepts a deps parameter.
[dagster-dbt] A new CLI utility dagster-dbt project scaffold has been provided to scaffold a Dagster code location for an existing dbt project.
Duplicate partition keys passed to StaticPartitionsDefinition will now raise an error.
Fixed a bug that caused lazy AutoMaterializePolicy's to not materialize missing assets.
[ui] Fixed an issue where global search and large DAGs were broken when using --path-prefix.
Schedule and sensor run submissions are now kept up to date with the current workspace, fixing an issue where a stale reference to a server would be used in some conditions.
[dagster-dbt] Fixed an issue where materializing dbt models with the same name as a dbt source would cause an error.
build_asset_reconciliation_sensor (Experimental) has been removed. It was deprecated in 1.3 in favor of AutoMaterializePolicy.
asset_key(s) properties on AssetIn and AssetDefinition have been removed in favor of key(s). These APIs were deprecated in 1.0.
root_input_manager and RootInputManagerDefinition have been removed in favor of input_manager and InputManagerDefinition. These APIs were deprecated in 1.0.
[dagster-pandas] The event_metadata_fn parameter on create_dagster_pandas_dataframe_type has been removed in favor of metadata_fn.
[dagster-dbt] The library has been substantially revamped to support the new @dbt_assets and DbtCliResource. See the migration guide for details.
Group names for dbt assets are now taken from a dbt model's group. Before, group names were determined using the model's subdirectory path.
Support for dbt-rpc has been removed.
The class alias DbtCloudResourceV2 has been removed.
DbtCli has been renamed to DbtCliResource. Previously, DbtCliResource was a class alias for DbtCliClientResource.
load_assets_from_dbt_project and load_assets_from_dbt_manifest now default to use_build=True.
The default assignment of groups to dbt models loaded from load_assets_from_dbt_project and load_assets_from_dbt_manifest has changed. Rather than assigning a group name using the model’s subdirectory, a group name will be assigned using the dbt model’s dbt group.
The argument node_info_to_definition_metadata_fn for load_assets_from_dbt_project and load_assets_from_dbt_manifest now overrides metadata instead of adding to it.
The arguments for load_assets_from_dbt_project and load_assets_from_dbt_manifest now must be specified using keyword arguments.
When using the new DbtCliResource with load_assets_from_dbt_project and load_assets_from_dbt_manifest, stdout logs from the dbt process will now appear in the compute logs instead of the event logs.
The dagit python package is deprecated and will be removed in 2.0 in favor of dagster-webserver. See the migration guide for details.
The following fields containing “dagit” in the Dagster helm chart schema have been deprecated in favor of “dagsterWebserver” equivalents (see migration guide for details):
[Dagster Cloud ECS Agent] We've introduced performance improvements that rely on the AWS Resource Groups Tagging API. To enable, grant your agent's IAM policy permission to tag:GetResources. Without this policy, the ECS Agent will log a deprecation warning and fall back to its old behavior (listing all ECS services in the cluster and then listing each service's tags).
DbtCliClientResource, dbt_cli_resource and DbtCliOutput are now being deprecated in favor of DbtCliResource.
A number of arguments on load_assets_from_dbt_project and load_assets_from_dbt_manifest are now deprecated in favor of other options. See the migration for details.
All public methods in the Dagster API now have docstrings.
The entirety of the documentation has been updated to now refer to the “Dagster webserver” or “Dagster UI” where “Dagit” was previously used for both entities.
DynamicPartitionsDefinition and SensorResult are no longer marked experimental
DagsterInstance now has a get_status_by_partition method, which returns the status of each partition for a given asset. Thanks renzhe-brian!
DagsterInstance now has a get_latest_materialization_code_versions method, which returns the code version of the latest materialization for each of the provided (non-partitioned) assets.
The error message for when an asset illegally depends on itself is now more informative.
Further performance improvements for the Asset Daemon.
Performance improvements in the asset graph view for large asset graphs.
Pandas 2.x is now supported in all dagster packages.
build_asset_context has been added as an asset focused replacement for build_op_context.
build_op_context now accepts a partition_key_range parameter.
New AssetSelection.upstream_source_assets method allows selecting source assets upstream of the current selection.
AssetSelection.key_prefixes and AssetSelection.groups now accept an optional include_sources parameter.
The AutoMaterialize evaluations UI now provides more details about partitions and waiting on upstream assets.
[dbt] The DbtCli resource is no longer marked experimental.
[dbt] The global_config parameter of the DbtCli resource has been renamed to global_config_flags
[dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now work with the DbtCli resource.
[dbt] The manifest argument of the @dbt_assets decorator now additionally can accept a Path argument representing a path to the manifest file or dictionary argument representing the raw manifest blob.
[dbt] When invoking DbtCli.cli from inside a @dbt_assets-decorated function, you no longer need to supply the manifest argument as long as you provide the context argument.
[dbt] The DbtManifest object can now generate schedules using dbt selection syntax.
[dbt] When invoking DbtCli.cli and the underlying command fails, an exception will now be raised. To suppress the exception, run the DbtCli.cli(..., raise_on_error=False).
[ui] You can now alphabetically sort your partitions on the asset partitions page
[ui] A button in the “Run is materializing this asset” and “Run failed to materialize this asset” banners provides direct access to the relevant run logs
Fixed a bug that caused asset metadata to not be available available on the OutputContext when using with_attributes or AssetsDefinition.from_graph.
Previously, if a partitioned asset at the root of the graph had more missing partitions than its AutoMaterializePolicy’s max_materializations_per_minute parameter, those older partitions would not be properly discarded from consideration on subsequent ticks. This has been fixed.
Fixed a bug that caused AutoMaterializePolicy.lazy() to not materialize missing assets that were downstream of assets without an AutoMaterializePolicy.
In rare cases, the AssetDaemon could hit an exception when using a combination of freshness policies and observable source assets. This has been fixed.
Previously, string type annotations (most commonly via modules containing from __future__ import annotations) would cause errors in most cases when used with Dagster definitions. This has been fixed for the vast majority of cases.
AssetExecutionContext has returned to being a type alias for OpExecutionContext.
[ui] Date filtering on the runs page now takes your timezone into consideration
[ui] Fixed a bug where selecting partitions in the launchpad dialog cleared out your configuration
[ui] In the run Gantt chart, executed steps that follow skipped steps no longer render off the far right of the visualization.
[ui] Cancelling a running backfill no longer makes canceled partitions un-selectable on the job partitions page and backfill modal, and cancellation is shown in gray instead of red.
[experimental] The internal time_window_partition_scope_minutes parameter of the AutoMaterializePolicy class has been removed. Instead, max_materializations_per_minute should be used to limit the number of runs that may be kicked off for a partitioned asset.
The databricks_pyspark_step_launcher will no longer error when executing steps that target a single partition of a DynamicPartitionsDefinition (thanks @weberdavid!).
Increased timeout on readinessProbe for example user code images, which prevents breakages in certain scenarios (thanks @leehuwuj)!
Avoid creation of erroneous local directories by GCS IO manager (thanks @peterjclaw)!
The @observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval.
[dbt] DbtCliTask has been renamed to DbtCliInvocation
[dbt] The get_asset_key_by_output_name and get_node_info_by_output_name methods of DbtManifest have been renamed toget_asset_key_for_output_name and get_node_info_for_output_name, respectively.
[ui] A new feature flag allows you to switch Asset DAG rendering to a tighter horizontal layout, which may be preferable in some scenarios
Many public methods that were missing in the API docs are now documented. Updated classes include DagsterInstance, *MetadataValue, DagsterType, and others.
dagster-pandera now has an API docs page.
Deprecated methods in the API docs now are marked with a special badge.