OpenLineage (Roadmap)

OpenLineage Integration Guidelines

Concur's Data Mapping solution now supports integration with OpenLineage, providing a comprehensive understanding of data pipelines by ingesting and displaying lineage information from various data processing frameworks.

Features

1. REST Endpoint Support:

   • Concur Data Mapping includes a REST endpoint capable of understanding OpenLineage events. This allows users to send lineage information directly to Concur Data Mapping, facilitating easy integration with various data processing frameworks.

2. Spark Event Listener Plugin:

   • Concur Data Mapping offers a Spark Event Listener plugin that seamlessly integrates with OpenLineage's Spark plugin. This plugin enhances Concur's OpenLineage support by providing additional features such as PathSpec support, column-level lineage, patch support, and more.

OpenLineage Support with Concur Data Mapping

1. REST Endpoint Support:

   • Concur Data Mapping's REST endpoint allows users to send OpenLineage events directly to Concur. This integration provides users with a centralized location for viewing and managing data lineage information, ensuring a cohesive and transparent view of their data pipelines.

2. Spark and Airflow Integration:

   • For optimal integration with Concur Data Mapping, it is recommended to use the Spark Lineage or Concur's Airflow plugin. These plugins offer tighter integration, enhancing the visibility and traceability of data workflows within Spark and Airflow environments.

REST Endpoint Support​

Concur's REST endpoint allows users to send OpenLineage events directly to Concur Data Mapping. This enables easy integration with various data processing frameworks, providing users with a centralized location for viewing and managing data lineage information.

With Spark and Airflow we recommend using the Spark Lineage or Concur's Airflow plugin for tighter integration with Concur Mapper.

Known Limitations​

With Spark and Airflow we recommend using the Spark Lineage or DataHub's Airflow plugin for tighter integration with DataHub.

• PathSpec Support: While the REST endpoint supports OpenLineage messages, full PathSpec) support is not yet available.

• Column-level Lineage: Concur's current OpenLineage support does not provide full column-level lineage tracking.

• Tesing & Discovering more...

2. Spark Event Listener Plugin​

Concur's Spark Event Listener plugin enhances OpenLineage support by providing additional features such as PathSpec support, column-level lineage, and more.

How to Use​

Follow the guides of the Spark Lineage plugin page for more information on how to set up the Spark Lineage plugin. The guide can be found here.