Commit
Β·
66a4b03
1
Parent(s):
d71b95a
Revert "update README and app.py for MCP 1st Birthday submission details and features"
Browse filesThis reverts commit cd3cd19c6a7ed0314863f29ef802eaa51398ef21.
README.md
CHANGED
|
@@ -9,16 +9,12 @@ app_file: app.py
|
|
| 9 |
pinned: false
|
| 10 |
license: mit
|
| 11 |
short_description: An agent that extracts data lineage, pipeline dependencies
|
| 12 |
-
tags:
|
| 13 |
-
- MCP-1st-Birthday
|
| 14 |
---
|
| 15 |
|
| 16 |
-
# Lineage Graph Accelerator π₯
|
| 17 |
|
| 18 |
A Gradio-based agent that extracts, summarizes, and visualizes data lineage from multiple metadata sources (BigQuery, dbt, Airflow, APIs, and more). Designed as a small, extendable framework of sub-agents that parse metadata, infer relationships, and render clear graph visualizations for exploration and debugging.
|
| 19 |
|
| 20 |
-
**Prepared as a submission for the MCP 1st Birthday celebration** β see [Hugging Face MCP-1st-Birthday activity](https://huggingface.co/organizations/MCP-1st-Birthday/activity/all).
|
| 21 |
-
|
| 22 |
## Architecture
|
| 23 |
|
| 24 |
This project is organized as a collection of lightweight sub-agents (workers): a metadata parser, a graph visualizer, and optional integration adapters (BigQuery, URL fetcher, dbt, Airflow). The UI (Gradio) orchestrates these components and displays results as Mermaid diagrams.
|
|
@@ -57,8 +53,7 @@ flowchart TD
|
|
| 57 |
- Mermaid for graph visualizations (client-side)
|
| 58 |
- Langsmith's Agent Builder (used to design and orchestrate the agent/sub-agent structure)
|
| 59 |
|
| 60 |
-
This project was prepared as a submission for the
|
| 61 |
-
|
| 62 |
|
| 63 |
## Quickstart (local)
|
| 64 |
|
|
|
|
| 9 |
pinned: false
|
| 10 |
license: mit
|
| 11 |
short_description: An agent that extracts data lineage, pipeline dependencies
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
|
| 14 |
+
# Lineage Graph Accelerator π₯
|
| 15 |
|
| 16 |
A Gradio-based agent that extracts, summarizes, and visualizes data lineage from multiple metadata sources (BigQuery, dbt, Airflow, APIs, and more). Designed as a small, extendable framework of sub-agents that parse metadata, infer relationships, and render clear graph visualizations for exploration and debugging.
|
| 17 |
|
|
|
|
|
|
|
| 18 |
## Architecture
|
| 19 |
|
| 20 |
This project is organized as a collection of lightweight sub-agents (workers): a metadata parser, a graph visualizer, and optional integration adapters (BigQuery, URL fetcher, dbt, Airflow). The UI (Gradio) orchestrates these components and displays results as Mermaid diagrams.
|
|
|
|
| 53 |
- Mermaid for graph visualizations (client-side)
|
| 54 |
- Langsmith's Agent Builder (used to design and orchestrate the agent/sub-agent structure)
|
| 55 |
|
| 56 |
+
This project was prepared as a submission for the MCP 1st Birthday celebration. See the Hugging Face MCP-1st-Birthday activity for context: https://huggingface.co/organizations/MCP-1st-Birthday/activity/all
|
|
|
|
| 57 |
|
| 58 |
## Quickstart (local)
|
| 59 |
|
app.py
CHANGED
|
@@ -182,36 +182,16 @@ def extract_lineage_from_url(
|
|
| 182 |
# Create Gradio interface
|
| 183 |
with gr.Blocks(title="Lineage Graph Extractor", theme=gr.themes.Soft()) as demo:
|
| 184 |
gr.Markdown("""
|
| 185 |
-
# π Lineage Graph Extractor
|
| 186 |
|
| 187 |
Extract and visualize data lineage from various metadata sources including BigQuery, dbt, Airflow,
|
| 188 |
APIs, and more. This tool helps you understand complex data relationships through clear graph visualizations.
|
| 189 |
|
| 190 |
-
## About
|
| 191 |
-
|
| 192 |
-
This project was prepared as a submission for the **MCP 1st Birthday celebration**. It demonstrates
|
| 193 |
-
how modular agent-based architecture (built with **Langsmith's Agent Builder**) can be used to extract
|
| 194 |
-
and visualize complex data relationships.
|
| 195 |
-
|
| 196 |
-
### Features
|
| 197 |
-
- π Multi-source metadata ingestion (Text, BigQuery, URLs/APIs)
|
| 198 |
-
- π€ AI-assisted metadata parsing and relationship extraction (pluggable agent backend)
|
| 199 |
-
- π Mermaid and DOT visualization support (rendered dynamically in the UI)
|
| 200 |
-
- π§© Lightweight, modular code designed for easy extension and testing
|
| 201 |
-
|
| 202 |
-
### Built With
|
| 203 |
-
- **Gradio** β Interactive web UI
|
| 204 |
-
- **Mermaid** β Graph visualization (client-side rendering)
|
| 205 |
-
- **Langsmith's Agent Builder** β Agent orchestration and sub-agent design
|
| 206 |
-
- **Python** β Core application logic
|
| 207 |
-
|
| 208 |
## Supported Sources
|
| 209 |
-
- **Text/File Metadata**: Paste metadata directly
|
| 210 |
- **BigQuery**: Query table metadata and relationships
|
| 211 |
- **URLs/APIs**: Fetch metadata from web endpoints
|
| 212 |
- **dbt, Airflow, Snowflake**: Through MCP integration (when configured)
|
| 213 |
-
|
| 214 |
-
Learn more: [MCP 1st Birthday](https://huggingface.co/organizations/MCP-1st-Birthday/activity/all)
|
| 215 |
""")
|
| 216 |
# Load Mermaid.js once (hidden). We set startOnLoad=false and will initialize
|
| 217 |
# individual diagrams after inserting them into the DOM.
|
|
|
|
| 182 |
# Create Gradio interface
|
| 183 |
with gr.Blocks(title="Lineage Graph Extractor", theme=gr.themes.Soft()) as demo:
|
| 184 |
gr.Markdown("""
|
| 185 |
+
# π Lineage Graph Extractor
|
| 186 |
|
| 187 |
Extract and visualize data lineage from various metadata sources including BigQuery, dbt, Airflow,
|
| 188 |
APIs, and more. This tool helps you understand complex data relationships through clear graph visualizations.
|
| 189 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 190 |
## Supported Sources
|
| 191 |
+
- **Text/File Metadata**: Paste metadata directly
|
| 192 |
- **BigQuery**: Query table metadata and relationships
|
| 193 |
- **URLs/APIs**: Fetch metadata from web endpoints
|
| 194 |
- **dbt, Airflow, Snowflake**: Through MCP integration (when configured)
|
|
|
|
|
|
|
| 195 |
""")
|
| 196 |
# Load Mermaid.js once (hidden). We set startOnLoad=false and will initialize
|
| 197 |
# individual diagrams after inserting them into the DOM.
|