data-catalog

Purpose

This skill manages metadata for data assets, enabling discovery, governance, and lineage tracking in data engineering workflows. It catalogs datasets, schemas, and dependencies to support data-driven projects.

When to Use

Use this skill when you need to track data assets in a project, such as during ETL processes, data governance audits, or when building data pipelines. Apply it in scenarios involving large-scale data repositories, compliance requirements, or collaborative data teams.

Key Capabilities

• Register and update metadata for datasets using JSON structures, e.g., {"name": "sales_data", "schema": {"columns": ["id", "date"]}} .

• Search and query assets via full-text or tag-based filters, supporting lineage queries like tracing data origins.

• Enforce governance policies, such as access controls, by associating tags like "sensitive" to assets.

• Generate lineage graphs in JSON format, e.g., {"source": "raw_logs", "target": "processed_reports"} .

• Integrate with storage systems like S3 or databases, using connectors that require API keys via $DATA_CATALOG_API_KEY .

Usage Patterns

To use this skill, first authenticate with an environment variable like export DATA_CATALOG_API_KEY=your_key . Then, follow a pattern: initialize the catalog, register assets, query as needed, and handle updates. For pipelines, embed it in scripts to auto-register outputs. Always validate metadata before operations to avoid conflicts.

Common Commands/API

Use the dcatalog CLI or REST API for interactions. Authentication requires $DATA_CATALOG_API_KEY in requests.

CLI Commands:

• Register an asset: dcatalog register --asset-name sales_data --type dataset --metadata '{"schema": ["id", "amount"]}' --api-key $DATA_CATALOG_API_KEY

• Query assets: dcatalog search --query "sales" --tags metadata --limit 10

• Update lineage: dcatalog update-lineage --source raw_data --target processed_data --relation depends_on

API Endpoints:

• POST /api/v1/assets: Create a new asset. Example curl: curl -H "Authorization: Bearer $DATA_CATALOG_API_KEY" -d '{"name": "sales_data", "tags": ["metadata"]}' -X POST https://api.opencclaw.com/api/v1/assets

• GET /api/v1/assets/search?query=sales: Search assets. Example: curl -H "Authorization: Bearer $DATA_CATALOG_API_KEY" https://api.opencclaw.com/api/v1/assets/search?query=sales

• PUT /api/v1/lineage: Update lineage. Code snippet: import requests headers = {"Authorization": f"Bearer {os.environ['DATA_CATALOG_API_KEY']}"} response = requests.put('https://api.opencclaw.com/api/v1/lineage', headers=headers, json={"source": "raw_data", "target": "report"})

Config formats are JSON-based, e.g., for CLI config file (~/.dcatalog/config.json ):

{"default_tags": ["data-governance"], "api_endpoint": "https://api.opencclaw.com"}

Integration Notes

Integrate this skill with data tools like Apache Airflow or AWS Glue by wrapping API calls in custom operators. For example, in a Python script, import the API client and pass $DATA_CATALOG_API_KEY . Ensure compatibility by matching schema versions; use JSON configs for mappings, e.g., link to S3 buckets via {"bucket": "my-bucket", "prefix": "data/"} . Test integrations in a sandbox environment before production.

Error Handling

Handle errors by checking HTTP status codes in API responses; for example, if status is 401, prompt for $DATA_CATALOG_API_KEY revalidation. For CLI, use try-catch in scripts:

try: subprocess.run(["dcatalog", "register", "--asset-name", "test"], check=True) except subprocess.CalledProcessError as e: print(f"Error: {e.returncode} - {e.output}")

Common issues include invalid JSON metadata (fix by validating with json.loads() before sending) or authentication failures (retry with refreshed keys). Log errors with timestamps for debugging.

Graph Relationships

• Related Cluster: data-engineering

• Connected Tags: metadata, data-governance, data-discovery

• Dependencies: Often links to skills in storage or processing clusters, e.g., for data ingestion or transformation.