New Features and Bug fixes in Version 1.1
Date of Publication: 28/04/2022
- Processor Creation: Create a processor for the existing connection so that it is used for the Pushdown feature. In the current build, this Pushdown feature is enabled for Snowflake source only. In the upcoming builds, we will enable Pushdown for Redshift and SQL Server sources.
- Snowflake Pushdown: If data is migrating within the same snowflake system, DataFactory will be orchestrating, and processing logic is pushed to the snowflake system to get high throughput.
- S3 CSV and S3 Parquet as sources: Capability to Ingest/query the CSV and Parquet files in AWS S3 is implemented. In the upcoming build, we will expand to JSON format.
- UI Performance Optimizations: The initial load time of DataFactory UI and time consumed to switch between different modules in UI has been drastically brought down by 10 times.
- Connection Management: Mapping a connection under the Project hierarchy is implemented.
- Migrating Snowflake Merge to the new UI: The Snowflake Merge widget is switched to the new UI for a better look and feel.
- DB Explorer: To save Dataset in DB Explorer is implemented.
- Jobs: Ability to Send Email notifications at each Asset level within a job, pause a scheduled job, resume a paused job and finally, kill an Active session of a job are implemented.
Maintaining the Linage/dependency of a job on other different assets like Batch Pipeline and BDP.
Note: Groundwork to be completed before Linage activity.
- Projects: Initial version of Copy To pipeline is completed. Ability to migrate Pipeline (Batch and Bulk Data Pipeline) from project to project was successful.
- User Management
Bug fixes: (Info Required)
- Sessions (Pipeline & BDP)
Bug Fixes: Session history issues are fixed.