0.1.2

Enhanced UI/UX in inventory and impact pages, lineage graph improvements, and new impact chart features.

What's Changed

Frontend Service

New Features and Enhancements:

• Inventory Table Update: Minor update to the inventory table columns for improved data representation and user experience.

• Grouped Bar Chart on Impact Page: Implemented a new grouped bar chart feature on the impact page, replacing the previous stacked bar chart for a clearer visualization of data.

• Impact Page Developments: Integration of the impact API and inclusion of impact chart data from Redux for enhanced data handling and visualization.

• Lineage Graph Enhancements: Significant improvements to the lineage graph, including:

  • Highlighting the full path on node click for better focus and understanding.

  • Custom styling for children and root nodes.

  • Addition of child nodes inside the parent node using ReactFlow subflow and dagre layout.

  • Workspace filter addition and default selection of the first workspace on the lineage graph page.

  • Customization of lineage node handles to match dagre layout.

UI/UX Improvements:

• Column Width Adjustment: Fixed the column width on the issueTracking table for better readability.

• Clock Icon Addition: Added a clock icon to the impact card for a more intuitive representation of time-related metrics.

• Potential Dev Hours Savings: Included potential development hours savings on the impact tab, providing a clearer indication of efficiency gains.

• Licensing Table Modifications: Made specific changes to the licensing table, including separate columns lists for PowerBI and Tableau.

Operational and Infrastructure Changes:

• Enhanced Hover Functionality: Updated the hover-over mechanism on metric cards to reduce accidental triggers.

• Workspace Filter and Input Field Enhancements: Improved workspace filter functionality and focus management on the lineage graph window.

Gateway Service

New Features and Enhancements:

• Inventory Page Update: Minor update to inventory page columns for better data representation and user experience.

• Recommendation Configuration Enhancement: Added asset_id, asset_name, and asset_type in Recommendation Config to provide more detailed recommendations.

• Snowflake Proxy Improvement: Updated the Snowflake proxy to deserialize JSON properly from the VARIANT column, enhancing data handling and accuracy.

• Usage Impact Chart Development: Introduced a method for get_usage_impact_chart in the snowflake_bi_proxy, along with a refactor of the query to match the latest design. This feature aims to provide better insights into usage impact through comprehensive charting.

Bug Fixes and Refinements:

• Bot User Creation Bug Fix: Resolved an issue causing new bot users not to be created on fresh builds due to a missing account_id foreign key in the UserOrmModel.

• SQL Query Refinement for Impact Charts: Updated and cleansed the SQL query used to retrieve data for impact charts, ensuring more accurate and relevant information.

Operational and Infrastructure Changes:

• Data Model Naming Correction: Fixed the naming of the database model used to gather data for usage impact charts.

• API Addition for Impact Charts: Added a new API to return impact charts data, enhancing the ability to visualize usage impact effectively.

• Column Additions for Inventory Page: Added dev_usage_ratio and link columns to the inventory page, providing more detailed and relevant information for inventory analysis.

• Engineering vs. Usage Hours Analysis: Introduced analysis for Engineering versus Usage Hours, enabling deeper insights into engineering efforts versus actual usage.

Diagnostics Service

New Features and Enhancements:

• Enhanced Dataset Embedding: Updated ml_app to embed sentences of datasets instead of JSON itself, improving data matching accuracy.

• Duplicate Dataset Detection: Implemented a new approach for duplicate dataset detection using embeddings, which improves performance and accuracy. This includes changes in processing datasets for both Tableau and PowerBI.

• Airflow Configuration Optimization: Aligned with Airflow production recommendations, now using LocalExecutor on a single VM configuration.

• Improved Similarity Metrics: Updated precision on similarity scores to 3 significant figures and relaxed duplication rules for a more accurate detection of duplicate tables.

• Enhanced Dataset Tree Maps: Added new types, similarity scores, and children to the top level of dim_dataset_treemap, enabling better visualization and understanding of dataset structures.

• Performance Enhancements: Numerous optimizations and refactors to improve the efficiency and speed of various processes, especially in similarity detection and embedding calculations.

• Tableau and PowerBI Integration Improvements: Numerous updates and fixes to enhance integration with Tableau and PowerBI, including better handling of dataset similarities and license summaries.

• Embedding Model Enhancements: Transitioned to a smaller FastText model for reduced memory footprint and updated Docker compose files for better service management.

• CI/CD Pipeline Updates: Added ml_app to the CI/CD pipeline and updated Docker image names for consistency.

Bug Fixes and Refinements:

• Syntax and Logic Fixes: Addressed various issues including fixing Snowflake syntax, resolving ambiguous alias names, and correcting incremental logic errors.

• UI and UX Improvements: Adjustments to sorting and filtering logic in various models for a more intuitive user experience.

• Data Handling Corrections: Fixed issues related to data embedding, including removing newline characters in sentences for embedding generation and adjusting data types and structures for better processing.