New Features and Bug fixes in Version 1.6
BSP:
- Kafka sink is implemented
- Detailed logs are enabled to provide table-level metrics at both the source and target systems
Enhancements:
- Provided an ability to view the underlying JSON configuration for BSP
- UX Interaction enhancement for BSP table selection flow is completed
Streaming:
- Mongo DB as a Source is implemented
- REST API Metadata (first level of Flattening) as a Source is implemented
Batch Pipeline:
- Oracle Merge: Ability to merge a source table to the target table by performing all three operations (INSERT, UPDATE, DELETE) at once
- REST API(Interactive Flatten):
In prior releases, forRESTAPIsource node the user has the flexibility to FULLFLATTEN the nested JSON response. In the current release, INTERACTIVE FLATTEN is enabled.
With this capability, the user could flatten at the first level and selectively choose to flatten the next level and further. And finally, selectively choose the fields that he is interested in
from multiple nested levels.
Job Clean-up Policy:
The user has the flexibility to provide clean-up routines similar to batch and BDP for both interim data and session data.
The clean-up policy that is specified at the job level will override clean up-policy setup for all underlying assets and will only be applicable to the sessions that were triggered from the job.
- Client issues with respect to the Jobs were addressed.
BDP
CICD flow would enable the Review and Approval process for the assets that have been created in DataFactory.
It is a capability that would allow us to version the assets in DataFactory working on similar lines of GIT workflow.
As part of Sprint 1.6, CICD flow has been initiated for the BDP and completed the following capabilities:
1. Notifications on the review process
2. Listing the session history by version
3 UI restrictions in the review process according to the state of the assets
4. Ability to list version history of asset
Users can use the CI/CD flow for batch pipeline end-to-end.
Features like rollback and version comparison will be covered in upcoming sprints.
DATA WRANGLER:
- Column to JSON/Array
Converting multiple column data into JSON format or Array format
- Change case transformation
Converting to upper and lower case
- Trim text
Removing white spaces
Remove special characters
Remove accents
- Sort column values
Sorting the grid table with a column by ascending or descending
AI-ML:
- Regression Flow
Added capability to create regression models
- Multi-classification Flow
Added capability to create Multi classification models
Confusion Matrix for Multi classification
- Model details
Details of all the inputs used in different stages of model creation
- Predict manual screen
Enhanced the prediction screen with the capability to give users visibility to available values for categories when doing manual input