Enhanced treemap with new features, lineage graph integration and UI updates.

What's Changed

Frontend Service

New Features and Enhancements:

  • Treemap Visualization Updates:

    • Improved treemap with the display of types, children count on tiles, and modification to the copied node's width.

    • Added base dataset name and similar dataset details at the top of the treemap.

    • Introduced legend colors at the top of the treemap for better data interpretation.

    • Updated treemap to version 0.2.5 for enhanced features and functionality.

  • Lineage Graph Integration:

    • Implemented setup for showing the lineage graph on the side panel, enhancing data navigation and visualization.

    • Updated functionality to show only the highlighted path and nodes when clicking on a particular node in the lineage graph.

UI/UX Improvements:

  • Inventory and Capacity Table Enhancements:

    • Made minor updates to capacity table columns on the impact page and inventory card tooltip, improving data display and user interaction.

    • Updated the tooltip to be visible only on mouse hover over the icon.

  • Color Scheme and Accessibility:

    • Revised treemap color scheme to be colorblind safe, ensuring accessibility for all users.

Gateway Service

New Features and Enhancements:

  • Treemap Schema Update: Added "count_children" integer value to treemap schemas, enhancing the frontend's capability to display hierarchical data more effectively.

  • Snowflake Proxy Enhancements:

    • Introduced new methods in the Snowflake proxy specifically for duplicate datasets treemap, improving data handling and visualization capabilities.

    • Fixed handling of null children in the Snowflake BI proxy, ensuring better data integrity and error handling.

UI/UX Improvements:

  • Capacity Table Update on Impact Page: Made a minor update to the capacity table columns on the impact page, likely for better data representation and user understanding.

Data Model and API Updates:

  • RecommendationPreview Model Enhancements:

    • Added workspace_id and workspace_name to the RecommendationPreview Pydantic model, enriching the data model with more contextually relevant information.

    • Implemented workspace_id and workspace_name in the /preview endpoint, aligning the API's output with the updated data model.

Diagnostics Service

New Features and Enhancements:

  • Treemap Visualization Updates:

    • Added "removed" objects to the treemap, enhancing data representation.

    • Updated the Impact Page to show all months of capacity impact in a single table.

    • Simplified calculation for the count of duplicate datasets for PowerBI.

  • Anonymization and Data Privacy:

    • Implemented anonymization of metric descriptions and other data for demo connectors.

    • Added more anonymizer functions for creating fake data, supporting nested JSON structures.

UI/UX Improvements:

  • Tableau API Hardening: Strengthened the Tableau API extraction process for better data integrity and reliability.

  • Minor Updates to Metric Descriptions: Adjusted metric descriptions for PowerBI datasource and Tableau dataset to remove PII and ensure more controlled anonymization.

Operational and Infrastructure Changes:

  • Dockerfile and Deployment Updates:

    • Multiple updates to Dockerfile for Airflow images to ensure the vault folder is writable by the Airflow user.

    • Adjusted demo data to meet gateway Pydantic model requirements for issue preview data.

  • Development Integrations and Merge Resolutions: Multiple merges from the development branch, integrating features and resolving conflicts.

  • SQL and Model Refinements:

    • Various SQL adjustments (referred to as 'fluffing') and model updates, including refresh history model remake.

    • Fixed a bug related to unreasonably high refresh counts.

Last updated