What is DataFactory?
DataFactory is an off-the-shelf, no code, self-service data engineering platform that empowers CDO organizations to build end-to-end high-performance data pipelines with data preparation, data wrangling, ML Modeling, Reporting, and Scheduling capabilities coupled with a scalable data warehouse (Data Lakehouse on Delta Lake platform). Project Management and Admin modules are also bundled to accurately track and monitor every activity closely while the tool capabilities are being utilized.
Quick peek at product features of DataFactory
What are the DataFactory capabilities?
The major capabilities include:
- Data Migration
- Identifying and Capturing the changes made to source data
- Data Streaming or Real-time replication of data
- Push-down Transformations to yield much better performance
- Data-Driven Modelling
- Analytics all the way thru built-in Analytics Hub
- Orchestration module to automate the data transfer process
- Data Preparation, Cleansing, and enrichment of data thru Data Wrangling
- MLOps thru AI/ML Modeling module to empower Data Scientists
DataFactory tool comprises two login users.
- Administrator
- General User
Administrator- The Administrator/SuperAdmin of the DataFactory tool is the sole owner who can create Users, and grant/deny access to the application.
General User- A general user can only access those features based on the access rights granted by the Administrator/Superadmin.
- Bulk Data Pipeline (BDP)
The Bulk Data Pipeline (BDP) is a one-time data migration initiative to migrate data from multiple tables in a stretch and load the data into the corresponding table.
Assume an organization has decided to revamp its existing data storage system that contains plenty of tables pertaining to employees, clients, operations, logistics, HR, and finance verticals data. Usually, the traditional approach of migrating each table of data from the source system to the target system individually is a time taking process, instead, the BDP migrates multiple tables of data from the source to the target system in one go.
The performance of the BDP depends on the size of the table. With Packetization and Parallelism features in place, large amounts of data are migrated quickly.
What is Data Migration?
Data migration is the process of extracting data from one data storage system and loading it to another. It also involves data transformation between diverse formats and applications. This includes data extraction, preparation, transformation, and loading process.
- Batch Pipeline
What is a Batch Pipeline?
A Batch Pipeline is a process of moving data from one system to another system. In a broader sense, a pipeline pulls the data from the source, applies required transformation widgets, and then pushes the processed data to the sink.
A Batch Pipeline is termed a regular Pipeline. These pipelines are executed manually or repeatedly. During each run, the data is extracted from the data source, apply various transformations to the data, and finally, push the refined data to the Sink. This process is marked complete once all the data is processed.
Typically, the execution time of a batch pipeline depends on the source data size. The primary purpose of a batch pipeline is to keep data flowing to solve problems and make decision-making, that make lives convenient and smoother.
ETL: Stands for Extract, Transform and Load. In ETL the data ingestion process is slower since it transforms data on a separate processing server before the loading process. Ideally, ETL is used when source data needs to be refined and manipulated before loading to the target system.
ELT: Stands for Extract, Load, and Transform. In ELT the data ingestion process is faster since the data is not sent to a separate server for restructuring. The ELT is flexible and efficient to ingest large amounts of data and processes both structured and unstructured datasets.
Change Data Capture CDC Pipeline:
Change data capture pipeline is the process of detecting and capturing changes made to the data in a database and then consecutively pushing those real-time changes to a downstream system or widget.
In DataFactory, the CDC pipeline captures the data changes based on DATE and TIMESTAMP variables. The real-time process of capturing data changes in the source database and instantly moving them to the sink database keeps the systems in sync and intact. CDC pipeline is very efficient, as it captures data changes from the last sync rather than copying the entire database.
- Data Streaming or Real-time streaming of data
Data streaming is the continuous process of transfer of data from a source to a destination. Streaming data sources sends data frequently, sometimes multiple times per second, and in small quantities. In contrast, the traditional batch processing where operations run infrequently and transmit larger amounts of data every time.
Real-time data streaming is advantageous when new data is generated continuously. This real-time transactional data can be monitored by financial/banking organizations to identify irregularities and prevent any fraud occurrences. This dynamic data can be accessed to perform analytics and thereby generate meaningful insights.
- PUSH-DOWN Approach
A PUSHDOWN approach is a method of pushing the transformation logic to the underlying database. All the processing happens within the underlying data platforms to achieve huge performance gains, instead of processing the integrations within the DataFactory application.
In DataFactory, the underlying data platforms are MSSQL, SNOWFLAKE, and REDSHIFT.
When you enable the PUSH-DOWN approach in DataFactory, the application executes the Pipeline in the underlying data platforms, which is quite faster than executing the Pipeline in the Apache Spark engine.
Pre-requisites:
- Processor creation
- Process Engine
Key Benefits:
- Faster data migration and improved performance
- Significant Cost reduction
- Data platforms are enabled for MSSQL, REDSHIFT, and SNOWFLAKE data sources
- PUSHDOWN can be implemented on Transformation and Sink widgets
- The sink and Source data source should be the same
To consider an Active connection where your granted Create, Drop, Truncate, Insert and Update permissions on selected Schema
Enables PUSHDOWN approach
Presently, the Processor Engine is enabled for a few Data sources
- Data-Driven Modelling
The Data-Driven modeling is an intelligent system built within DataFactory that helps preview the existing data for the selected widget earlier while creating the pipeline. This model validates data while configuring the widget. At any point in time, you can Resume/Re-run at every pipeline configuration stage for better visibility and build effective pipelines.
- Analytics Hub
Organizations rely on Analytics and Reports provided by data practitioners. These analytics and reporting help organizations improve business operations and take informed decisions. But, how these critical decisions can be taken? It's only through quality data. With the right tools in place, you can get a complete picture of the data enrichment process from start to finish where you can pinpoint and focus on major roadblocks and mild slippages to avoid severe damage and loss.
Normal practice is to run the analytics on the final data available in the target system. The final number displayed in the dashboards helps the leadership team make timely critical decisions to improve processes and redefine standards.
The Analytics hub generates visual dashboards for the selected datasets. Connections and Datasets (analytics-enabled) are metrics to factor in before generating a visual and subsequently a dashboard. There are a couple of visual type graphs enabled. The analytics hub generates visuals on the analytics-enabled datasets subsequently a dashboard can be populated for the selected visual.
7. Orchestration module to automate the data transfer process
In IT organizations, the data journey starts from several data sources, where data gets ingested, integrated, transformed, processed, and loaded in various target data warehouse systems. At a glimpse, this data migration process sounds simple and easy, but unfortunately, it isn't the way your thinking. It's quite hectic and cumbersome to manually initiate and track these processes round the clock. To make this process automated, the orchestration mechanism is in place to avoid manual intervention.
The orchestration mechanism streamlined the data transfer process efficiently and effectively to cater to business needs and help build complex pipelines. Ideally, the right Orchestration tool should be able to perform:
- Schedule/Re-schedule the pipelines to run at desired time intervals without any glitches
- Create Pre/Postcondition rules on assets
- Send Email notification to the concerned team to notify success/failure of a pipeline run
- Provision an option to Pause/Resume of currently running pipelines in case of system maintenance or any last-minute unplanned events
DataFactory is a cumulative data platform with several built-in capabilities to cater all the data engineering needs.
- Data Preparation, Cleansing, and enrichment of data through Data Wrangling
Data wrangling is the method of transforming and mapping raw data form into another form with the aim of transforming it more accurately, appropriately, and valuable for a wide variety of uses and purposes such as analytics. It is the process of structuring, organizing, enriching, cleaning, and extracting data for various purposes.
Raw data is a piece of information that has to be processed or integrated into a system. This data can be in any format such as text, photos, databases, etc.
Data wrangling is also called Data munging, which is the most time-intensive aspect of data processing. According to Data Scientists, completion of this process takes 75% of the time. It’s time-intensive because accuracy is essential since this data is pulled from various sources and is often used for many purposes.
The Data Preparation stage is initiated when the data is available and ready to be processed. At this stage, the data is structured and cleaned to remove duplicates, redundancies, and unwanted data. Instead of juggling manually to organize this raw data, DataFactory helps to remove these discrepancies easily. Data Wrangling in DataFactory enables Data Privacy and Security that reduces data scams and frauds.
- MLOps through AI/ML Modeling module to empower Data Scientists
The AI/ML models are mathematical algorithms that are “trained” using data and human expert input to replicate a decision an expert would make when provided that same information.
- Artificial Intelligence: It is a technique that enables machines to mimic human behavior.
- Machine Learning: It is a subset of AI which use a statistical method to enable machines to improve with experience.
In DataFactory, Data can be imported from Files, Cloud, APIs, and Projects. A dataset is prepared from the data that is uploaded and imported from various data sources. This dataset is passed through Split Train and Test Data stage for a sampling process. The sampled dataset is evaluated using Estimators. Finally, when the evaluation is completed a Predictor model is sent for review and feedback. Additionally, you can perform different actions on different models.