> For the complete documentation index, see [llms.txt](https://gchandra.gitbook.io/big-data-and-tools-with-nosql/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://gchandra.gitbook.io/big-data-and-tools-with-nosql/big-data-overview/data-integration.md).

# Data Integration

There is a significant difference between how data is integrated or consumed in the Big Data world and the Traditional database world.

### ETL: Extract Transform Load

They are used with traditional RDBMS such as MySQL, SQLServer, and Oracle.

<figure><img src="https://content.gitbook.com/content/rtXPLjVTxuTGIysjCx89/blobs/GC6y4c1K5bSmxmDJLK8n/01_etl.png" alt=""><figcaption><p>Src: Leanmsbitutorial.com</p></figcaption></figure>

**ETL Tools - Examples**

* SSIS / SSDT: SQL Server Integration Services / SQL Server Data Tools
* Pentaho Kettle
* Talend Open Source Data Integrator
* Benetl for MySQL and PostgreSQL

<figure><img src="https://1471795080-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FrtXPLjVTxuTGIysjCx89%2Fuploads%2F2IoymwBEBGYc9Eus6Ux9%2Fimage.png?alt=media&amp;token=1e0e952c-574f-40b6-bad5-38bd777634f3" alt=""><figcaption><p>Img Src: https://towardsdatascience.com/how-i-redesigned-over-100-etl-into-elt-data-pipelines-c58d3a3cb3c</p></figcaption></figure>

### **ELT: Extract Load Transform**

Newer methodology compared to ETL and is used in the Big data world.

**Benefits**

* Ingest anything and everything as soon you get the data.
* Transform based on your needs.
* High-Speed loading
* Low Maintenance to load.

**Challenges**

* Security gaps are challenging to monitor as the objective is to load first.
* Compliance -- Increased pressure on Organizations to follow all regulations such as HIPAA, Do Not Call, Privacy policy, etc.
* Resource usage -- High storage costs money (On-Prem or Cloud)

### Data Flow

Raw data > Cleansed Data > Data Process\* > Data Warehousing > ML/BI/Analysis

\*Data goes through various pipelines and stages.