Commit
Β·
cd3cd19
1
Parent(s):
f12921d
update README and app.py for MCP 1st Birthday submission details and features
Browse files
README.md
CHANGED
|
@@ -9,12 +9,16 @@ app_file: app.py
|
|
| 9 |
pinned: false
|
| 10 |
license: mit
|
| 11 |
short_description: An agent that extracts data lineage, pipeline dependencies
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
|
| 14 |
-
# Lineage Graph Accelerator π₯
|
| 15 |
|
| 16 |
A Gradio-based agent that extracts, summarizes, and visualizes data lineage from multiple metadata sources (BigQuery, dbt, Airflow, APIs, and more). Designed as a small, extendable framework of sub-agents that parse metadata, infer relationships, and render clear graph visualizations for exploration and debugging.
|
| 17 |
|
|
|
|
|
|
|
| 18 |
## Architecture
|
| 19 |
|
| 20 |
This project is organized as a collection of lightweight sub-agents (workers): a metadata parser, a graph visualizer, and optional integration adapters (BigQuery, URL fetcher, dbt, Airflow). The UI (Gradio) orchestrates these components and displays results as Mermaid diagrams.
|
|
@@ -53,7 +57,8 @@ flowchart TD
|
|
| 53 |
- Mermaid for graph visualizations (client-side)
|
| 54 |
- Langsmith's Agent Builder (used to design and orchestrate the agent/sub-agent structure)
|
| 55 |
|
| 56 |
-
This project was prepared as a submission for the MCP 1st Birthday celebration
|
|
|
|
| 57 |
|
| 58 |
## Quickstart (local)
|
| 59 |
|
|
|
|
| 9 |
pinned: false
|
| 10 |
license: mit
|
| 11 |
short_description: An agent that extracts data lineage, pipeline dependencies
|
| 12 |
+
tags:
|
| 13 |
+
- MCP-1st-Birthday
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# Lineage Graph Accelerator π₯ #MCP-1st-Birthday
|
| 17 |
|
| 18 |
A Gradio-based agent that extracts, summarizes, and visualizes data lineage from multiple metadata sources (BigQuery, dbt, Airflow, APIs, and more). Designed as a small, extendable framework of sub-agents that parse metadata, infer relationships, and render clear graph visualizations for exploration and debugging.
|
| 19 |
|
| 20 |
+
**Prepared as a submission for the MCP 1st Birthday celebration** β see [Hugging Face MCP-1st-Birthday activity](https://huggingface.co/organizations/MCP-1st-Birthday/activity/all).
|
| 21 |
+
|
| 22 |
## Architecture
|
| 23 |
|
| 24 |
This project is organized as a collection of lightweight sub-agents (workers): a metadata parser, a graph visualizer, and optional integration adapters (BigQuery, URL fetcher, dbt, Airflow). The UI (Gradio) orchestrates these components and displays results as Mermaid diagrams.
|
|
|
|
| 57 |
- Mermaid for graph visualizations (client-side)
|
| 58 |
- Langsmith's Agent Builder (used to design and orchestrate the agent/sub-agent structure)
|
| 59 |
|
| 60 |
+
This project was prepared as a submission for the [MCP 1st Birthday celebration](https://huggingface.co/organizations/MCP-1st-Birthday/activity/all).
|
| 61 |
+
|
| 62 |
|
| 63 |
## Quickstart (local)
|
| 64 |
|
app.py
CHANGED
|
@@ -108,16 +108,36 @@ def extract_lineage_from_url(
|
|
| 108 |
# Create Gradio interface
|
| 109 |
with gr.Blocks(title="Lineage Graph Extractor", theme=gr.themes.Soft()) as demo:
|
| 110 |
gr.Markdown("""
|
| 111 |
-
# π Lineage Graph Extractor
|
| 112 |
|
| 113 |
Extract and visualize data lineage from various metadata sources including BigQuery, dbt, Airflow,
|
| 114 |
APIs, and more. This tool helps you understand complex data relationships through clear graph visualizations.
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
## Supported Sources
|
| 117 |
-
- **Text/File Metadata**: Paste metadata directly
|
| 118 |
- **BigQuery**: Query table metadata and relationships
|
| 119 |
- **URLs/APIs**: Fetch metadata from web endpoints
|
| 120 |
- **dbt, Airflow, Snowflake**: Through MCP integration (when configured)
|
|
|
|
|
|
|
| 121 |
""")
|
| 122 |
# Load Mermaid.js once (hidden). We set startOnLoad=false and will initialize
|
| 123 |
# individual diagrams after inserting them into the DOM.
|
|
|
|
| 108 |
# Create Gradio interface
|
| 109 |
with gr.Blocks(title="Lineage Graph Extractor", theme=gr.themes.Soft()) as demo:
|
| 110 |
gr.Markdown("""
|
| 111 |
+
# π Lineage Graph Extractor #MCP-1st-Birthday
|
| 112 |
|
| 113 |
Extract and visualize data lineage from various metadata sources including BigQuery, dbt, Airflow,
|
| 114 |
APIs, and more. This tool helps you understand complex data relationships through clear graph visualizations.
|
| 115 |
|
| 116 |
+
## About
|
| 117 |
+
|
| 118 |
+
This project was prepared as a submission for the **MCP 1st Birthday celebration**. It demonstrates
|
| 119 |
+
how modular agent-based architecture (built with **Langsmith's Agent Builder**) can be used to extract
|
| 120 |
+
and visualize complex data relationships.
|
| 121 |
+
|
| 122 |
+
### Features
|
| 123 |
+
- π Multi-source metadata ingestion (Text, BigQuery, URLs/APIs)
|
| 124 |
+
- π€ AI-assisted metadata parsing and relationship extraction (pluggable agent backend)
|
| 125 |
+
- π Mermaid and DOT visualization support (rendered dynamically in the UI)
|
| 126 |
+
- π§© Lightweight, modular code designed for easy extension and testing
|
| 127 |
+
|
| 128 |
+
### Built With
|
| 129 |
+
- **Gradio** β Interactive web UI
|
| 130 |
+
- **Mermaid** β Graph visualization (client-side rendering)
|
| 131 |
+
- **Langsmith's Agent Builder** β Agent orchestration and sub-agent design
|
| 132 |
+
- **Python** β Core application logic
|
| 133 |
+
|
| 134 |
## Supported Sources
|
| 135 |
+
- **Text/File Metadata**: Paste metadata directly (JSON, YAML, SQL, etc.)
|
| 136 |
- **BigQuery**: Query table metadata and relationships
|
| 137 |
- **URLs/APIs**: Fetch metadata from web endpoints
|
| 138 |
- **dbt, Airflow, Snowflake**: Through MCP integration (when configured)
|
| 139 |
+
|
| 140 |
+
Learn more: [MCP 1st Birthday](https://huggingface.co/organizations/MCP-1st-Birthday/activity/all)
|
| 141 |
""")
|
| 142 |
# Load Mermaid.js once (hidden). We set startOnLoad=false and will initialize
|
| 143 |
# individual diagrams after inserting them into the DOM.
|