Enhanced UI/UX in inventory and impact pages, lineage graph improvements, and new impact chart features.

What's Changed

Frontend Service

New Features and Enhancements:

  • Inventory Table Update: Minor update to the inventory table columns for improved data representation and user experience.

  • Grouped Bar Chart on Impact Page: Implemented a new grouped bar chart feature on the impact page, replacing the previous stacked bar chart for a clearer visualization of data.

  • Impact Page Developments: Integration of the impact API and inclusion of impact chart data from Redux for enhanced data handling and visualization.

  • Lineage Graph Enhancements: Significant improvements to the lineage graph, including:

    • Highlighting the full path on node click for better focus and understanding.

    • Custom styling for children and root nodes.

    • Addition of child nodes inside the parent node using ReactFlow subflow and dagre layout.

    • Workspace filter addition and default selection of the first workspace on the lineage graph page.

    • Customization of lineage node handles to match dagre layout.

UI/UX Improvements:

  • Column Width Adjustment: Fixed the column width on the issueTracking table for better readability.

  • Clock Icon Addition: Added a clock icon to the impact card for a more intuitive representation of time-related metrics.

  • Potential Dev Hours Savings: Included potential development hours savings on the impact tab, providing a clearer indication of efficiency gains.

  • Licensing Table Modifications: Made specific changes to the licensing table, including separate columns lists for PowerBI and Tableau.

Operational and Infrastructure Changes:

  • Enhanced Hover Functionality: Updated the hover-over mechanism on metric cards to reduce accidental triggers.

  • Workspace Filter and Input Field Enhancements: Improved workspace filter functionality and focus management on the lineage graph window.

Gateway Service

New Features and Enhancements:

  • Inventory Page Update: Minor update to inventory page columns for better data representation and user experience.

  • Recommendation Configuration Enhancement: Added asset_id, asset_name, and asset_type in Recommendation Config to provide more detailed recommendations.

  • Snowflake Proxy Improvement: Updated the Snowflake proxy to deserialize JSON properly from the VARIANT column, enhancing data handling and accuracy.

  • Usage Impact Chart Development: Introduced a method for get_usage_impact_chart in the snowflake_bi_proxy, along with a refactor of the query to match the latest design. This feature aims to provide better insights into usage impact through comprehensive charting.

Bug Fixes and Refinements:

  • Bot User Creation Bug Fix: Resolved an issue causing new bot users not to be created on fresh builds due to a missing account_id foreign key in the UserOrmModel.

  • SQL Query Refinement for Impact Charts: Updated and cleansed the SQL query used to retrieve data for impact charts, ensuring more accurate and relevant information.

Operational and Infrastructure Changes:

  • Data Model Naming Correction: Fixed the naming of the database model used to gather data for usage impact charts.

  • API Addition for Impact Charts: Added a new API to return impact charts data, enhancing the ability to visualize usage impact effectively.

  • Column Additions for Inventory Page: Added dev_usage_ratio and link columns to the inventory page, providing more detailed and relevant information for inventory analysis.

  • Engineering vs. Usage Hours Analysis: Introduced analysis for Engineering versus Usage Hours, enabling deeper insights into engineering efforts versus actual usage.

Diagnostics Service

New Features and Enhancements:

  • Enhanced Dataset Embedding: Updated ml_app to embed sentences of datasets instead of JSON itself, improving data matching accuracy.

  • Duplicate Dataset Detection: Implemented a new approach for duplicate dataset detection using embeddings, which improves performance and accuracy. This includes changes in processing datasets for both Tableau and PowerBI.

  • Airflow Configuration Optimization: Aligned with Airflow production recommendations, now using LocalExecutor on a single VM configuration.

  • Improved Similarity Metrics: Updated precision on similarity scores to 3 significant figures and relaxed duplication rules for a more accurate detection of duplicate tables.

  • Enhanced Dataset Tree Maps: Added new types, similarity scores, and children to the top level of dim_dataset_treemap, enabling better visualization and understanding of dataset structures.

  • Performance Enhancements: Numerous optimizations and refactors to improve the efficiency and speed of various processes, especially in similarity detection and embedding calculations.

  • Tableau and PowerBI Integration Improvements: Numerous updates and fixes to enhance integration with Tableau and PowerBI, including better handling of dataset similarities and license summaries.

  • Embedding Model Enhancements: Transitioned to a smaller FastText model for reduced memory footprint and updated Docker compose files for better service management.

  • CI/CD Pipeline Updates: Added ml_app to the CI/CD pipeline and updated Docker image names for consistency.

Bug Fixes and Refinements:

  • Syntax and Logic Fixes: Addressed various issues including fixing Snowflake syntax, resolving ambiguous alias names, and correcting incremental logic errors.

  • UI and UX Improvements: Adjustments to sorting and filtering logic in various models for a more intuitive user experience.

  • Data Handling Corrections: Fixed issues related to data embedding, including removing newline characters in sentences for embedding generation and adjusting data types and structures for better processing.

Last updated