Qlik Talend Data Integration Solutions

Fa'amatalaga

• Igoa o oloa: Qlik Talend Data Integration Solutions
• Tu'ufa'atasiga Fa'avae: Databricks Lakehouse Platform
• Vaega: Suia Faʻamaumauga Faʻamaumauga (CDC), Suiga Tekonolosi

Fa'atonuga o le Fa'aaogaina o Mea

Qlik Talend Data Integration Solutions
Qlik Talend Data Integration solutions faʻavavevaveina le aʻoaʻoina o masini (ML), atamai faʻapitoa (AI), ma faʻataʻitaʻiga DataOps ma Suiga Faʻamaumauga Faʻamaumauga (CDC), ma tekonolosi suiga e faʻamautinoa ai le faʻaauauina o faʻamaumauga mai le tele o faʻamaumauga i le Databricks Lakehouse Platform ua saunia mo AI ma Fa'aaogāina fa'amatalaga.

Qlik Toe fai
I lenei fausaga, Qlik Replicate o loʻo faʻatinoina galuega nei:

1. Fa'atotonu le sini:
   • Fausia laulau fa'atatau ile DELTA fa'atasi ma fa'amaumauga sa'o fa'aliliu mai le puna.
   • Fa'atino se uta muamua/atoa mai le puna ma lafo fa'amaumauga i totonu o le fa'aputuga.
   • Auina atu Spark SQL i Databricks e utaina faʻamaumauga mai le faʻapipiʻi faʻapipiʻi ma faʻaliliu i laulau e faʻaaoga ai le faʻasologa o le delta.
2. Pu'e ma Fa'aoga Suiga:
   • Pu'e suiga e fa'aaoga ai le CDC fa'atatau i ogalaau mai le puna.
   • Tu'u ma FA'A'O suiga (Fa'aofi/Fa'afou/Ta'ese) i laulau DELTA fa'amoemoe (fa'aaogā le fa'aputuga fa'aputu e fai s vaeluagalemu.taging).

Qlik Cloud Data Integration
Qlik Cloud Data Integration o se iPaaS ofo lea e maua ai le tomai e fatuina ai laina paipa o faʻamaumauga e faʻatino ai le tele o galuega faʻapipiʻi faʻamaumauga e lagolago ai lau faʻataʻitaʻiga faʻamaumauga ma AI, ma mea e manaʻomia.

Galuega a le Qlik Cloud Data Integration:

• Galuega Tu'ueleele:
  • Vaavaaia le fesiitaiga lemu o faʻamatalaga mai faʻamatalaga eseese i le sone tulaʻi mai e faʻaaoga ai le Qlik Data Gateway - Faʻamatalaga Faʻamatalaga mo le mauaina o faʻamatalaga faʻamatalaga e ala i Suiga Faʻamaumauga Faʻamaumauga (CDC).
  • Fa'ataga mo toe fa'atulagaina fa'avaitaimi e ala i le fa'atinoina o uta atoa.
• Galuega Suiga:
  • Fa'atupuina suiga fa'amatalaga e mafai ona fa'aaogaina ma fa'avae tulafono i totonu ole laina fa'amaumauga.
  • Fa'atino suiga o laina-tulaga ma fa'amaumauga tufuga e fa'aaoga ai le SQL masani.
  • E mafai ona faʻaalia e pei o laulau pe faʻaalia e faʻamalosi views fa'aogaina suiga ile lele.

Fesili e Fai soo (FAQ)

• E mafai ona fa'aoga fa'atasi le Qlik Replicate ma le Qlik Cloud Data Integration?
  Ioe, o fofo uma e lua e tutusa ma e mafai ona fa'aoga fa'atasi pe fa'atasi e fa'atatau i le fa'aoga a le atina'e ma fa'amaumauga fa'amaumauga.
• O a nisi o galuega autu a Qlik Replicate?
  Qlik Replicate e mafai ona vave faʻatulagaina le faʻamoemoe e ala i le fatuina o laulau ile DELTA format, faia muamua uta, puʻeina suiga e faʻaaoga ai le CDC, ma faʻaoga suiga i laulau faʻatatau.
• O le a le faʻamoemoe o le Qlik Cloud Data Integration?
  Qlik Cloud Data Integration e mafai ai mo le fatuina o laina paipa o faʻamatalaga mo galuega faʻapipiʻi faʻamaumauga eseese e lagolago ai faʻamaumauga faʻamaumauga ma AI, ma manaʻoga Analytics.

Folasaga

Qlik Talend Data Integration solutions faʻavavevaveina le aʻoaʻoina o masini (ML), atamai faʻapitoa (AI), ma faʻataʻitaʻiga DataOps ma Suiga Faʻamaumauga Faʻamaumauga (CDC), ma tekonolosi suiga e faʻamautinoa ai le faʻaauauina o faʻamaumauga mai le tele o faʻamaumauga i le Databricks Lakehouse Platform ua saunia mo AI ma Fa'aaogāina fa'amatalaga.

Qlik Talend Data Integration Solutions

• Qlik Replicate ma Qlik Talend Cloud Data Integration o ni fofo se lua mai le Qlik e mafai ai e pisinisi ona faʻatautaia a latou faʻamaumauga i luga o punaoa eseese ma faʻavae. O le Qlik Replicate o se polokalama faʻapipiʻi faʻamaumauga e mafai ai e tagata faʻaoga ona toe faʻafoʻi ma faʻafouina faʻamaumauga i le taimi moni mai luga ole fale ma faʻapogai o ao ile fale teu oloa faʻamaumauga e aunoa ma le faʻasologaina tusi poʻo le tusitusi. Qlik Cloud Data Integration o se 'auʻaunaga faʻavae i luga ole ao e maua ai le gafatia e fatuina ai laina paipa faʻamaumauga e faʻatino ai galuega faʻapipiʻi faʻamaumauga eseese, e pei o le tulaueleele, resitalaina, suiga, ma le tuʻufaʻatasia o faʻamaumauga. O fofo uma e lua e lagolagoina le tele o faʻamatalaga faʻamatalaga ma taunuuga, e pei o fesoʻotaʻiga faʻamaumauga, faʻamaumauga tetele, SAP, Mainframes, cloud storage, ma SaaS applications. Ae ui i lea, o loʻo i ai ni eseesega i le va o latou i tulaga o foliga, tau, ma le faʻaogaina o filifiliga.
• O nisi o eseesega autu o:
  • Qlik Replicate o se oloa tutoʻatasi e mafai ona faʻapipiʻi i luga o le fale poʻo luga o le ao, aʻo Qlik Talend Cloud Data Integration o se auaunaga faʻatautaia atoatoa lea e taʻavale i luga o le Qlik Cloud platform.
  • Qlik Replicate e ofoina atu faʻamatalaga sili atu mo le toe faia o faʻamaumauga, e pei o le suiga o faʻamaumauga (CDC), faʻatuai faʻatasi, faʻasologa o fuafuaga, ma le faʻafitauli o feteʻenaʻiga. A o Qlik Talend Cloud Data Integration e taulaʻi atili i le suiga o faʻamatalaga ma le faʻatasi, e pei o le faʻamamaina o faʻamaumauga, faʻatamaoaigaina, faʻafefiloi, ma faʻasalalauga.
• I le aotelega, Qlik Replicate ma Qlik Talend Cloud Data Integration o ni fofo faʻatasi e mafai ona faʻaoga faʻatasi pe faʻapitoa e faʻatatau i le faʻaoga faʻaoga ma le faʻasologa o faʻamaumauga a le atinaʻe. O fofo uma e lua e faʻamoemoe e tuʻuina atu faʻamatalaga vave, faʻatuatuaina, ma mafai ona faʻaogaina faʻamatalaga mo manaʻoga faʻaonaponei AI ma Analytics.

Qlik Toe fai

• Qlik Replicate® fa'aauau ona fa'autometi fe'avea'i fa'amaumauga a le CDC mai le tele o fa'amaumauga (fa'ata'ita'iga, Oracle, Microsoft SQL Server, SAP, Mainframe, ma isi mea) i le Databricks Lakehouse Platform. E fesoasoani i le tagata faʻatau e aloese mai le siiina mamafa e fesoʻotaʻi ma le suʻeina ma le lima o faʻamaumauga, faʻafeiloaʻi e ala i le API / script, ona tipi lea, stagfa'atau, ma fa'aulufale mai.
• I lenei fausaga, Qlik Replicate o loʻo faʻatinoina galuega nei:
  1. Fa'anatinati le Sini
     • Fausia laulau fa'atatau ile DELTA fa'atasi ma fa'amaumauga sa'o fa'aliliu mai le puna
     • Fa'atino se uta muamua/atoa mai le puna ma lafo fa'amaumauga i totonu o le fa'aputuga
     • Auina atu Spark SQL i Databricks e utaina faʻamaumauga mai le faʻapipiʻi faʻapipiʻi ma faʻaliliu i laulau e faʻaaoga ai le faʻasologa o le delta.
  2. Pu'e ma Fa'aaoga Suiga
     • Pu'e suiga e fa'aaoga ai le CDC fa'atatau i ogalaau mai le puna
     • Tu'u ma FA'A'O suiga (Fa'aofi/Fa'afou/Ta'ese) i laulau DELTA fa'amoemoe (fa'aaogā le fa'aputuga fa'aputu e fai s vaeluagalemu.taging)

Qlik Cloud Data Integration

• Qlik Cloud Data Integration o se iPaaS (Integration Platform as a Service) ofo e maua ai le gafatia e fatuina ai laina paipa o faʻamatalaga e faʻatino ai le tele o galuega faʻapipiʻi faʻamaumauga e lagolago ai lau faʻataʻitaʻiga faʻamaumauga ma AI, ma mea e manaʻomia.
  • Pipa fa'amaumauga - E mafai ona e fa'aogaina taimi moni, fa'amautu suiga o fa'amaumauga o fa'amaumauga ma feso'ota'iga fa'amautu i puna'oa fa'amaumauga i luga ole fale i tua atu o pa puipui po'o le fa'aogaina o le gafatia atoa mo fa'amaumauga a le SaaS. O le taimi lava e te maua ai faʻamatalaga i luga ole laiga, e mafai ona e faʻaogaina suiga mo mea e fetaui lelei mo le faʻamoemoega poʻo le faʻaautomatika faʻataʻitaʻiga e pei o faʻamatalaga mart mea moni ma fua. Fafo views ma ola views e fa'atupuina mo fa'amatalaga fa'amatalaga. Qlik Cloud Data Integration e fa'atupuina ai fo'i le ituaiga atoa 2 fa'amaumauga o tala fa'asolopito (HDS).
  • Galuega Fa'aliliuga Fa'amatalaga - Toe fa'atusa fa'amaumauga mai so'o se fa'apogai talafeagai i so'o se taunu'uga lagolago. E mafai ona suia fa'amaumauga ma fa'aauau pea ona fa'afouina e fa'aoga ai Suiga Fa'amatalaga Pu'eina (CDC) metotia. O loʻo i ai foi le gafatia e tuʻuina atu faʻamatalaga i totonu o se vaituloto faʻamatalaga e tuʻuina atu faʻamatalaga i le Amazon S3, Azure Data Lake Storage, poʻo le Google Cloud Storage.
• O faʻamatalaga faʻamaonia mai le Qlik Cloud Data Integration e mafai ona faʻaaogaina mo le tele o faʻamoemoega:
  • Fa'agasolo taimi moni mai so'o se atina'e e aofia ai fa'amaumauga feso'ota'iga, SAP, Mainframe, ma SaaS talosaga.
  • Suiga o faʻamatalaga e faʻaaoga ai le ELT (Extract/Load/Transform) e faʻaaoga ai se auala e leai se tulafono e aunoa ma le manaʻomia mo isi fofo 3rd party.
  • Fausiaina otometi o datamarts mo auʻiliʻili i le Databricks Lakehouse.
  • Fa'aonaponei o lau fa'amaumauga fa'amaumauga e lagolago ai AI, Masini A'oa'oga ma isi fuafuaga.
• I lenei fausaga, Qlik Cloud Data Integration o loʻo faʻatinoina galuega nei:
  • Fa'anatinati le Sini.
    • Fausia laulau fa'atatau ile DELTA fa'atasi ma fa'amaumauga sa'o fa'aliliuina mai laulau fa'apogai
    • Fa'atino se uta muamua/atoa mai le puna e lafo atu ai fa'amaumauga ile fa'aputuga
      • SaaS talosaga - tuusao
      • Faʻamaumauga Faʻamaumauga - faʻaaogaina Data Gateway
    • Auina atu Spark SQL i Databricks e utaina faʻamaumauga mai le faʻapipiʻi faʻapipiʻi ma faʻaliliu i laulau e faʻaaoga ai le faʻasologa o le delta.
  • Pu'e ma Fa'aaoga Suiga.
    • Pu'e suiga e fa'aaoga ai le CDC fa'atatau i ogalaau mai le puna
      • SaaS talosaga - tuusao
      • Faʻamaumauga Faʻamaumauga - faʻaaoga Qlik Data Gateway
    • Tu'u ma FA'A'O suiga (Fa'aofi/Fa'afou/Ta'ese) i laulau DELTA fa'amoemoe (fa'aaogā le fa'aputuga fa'aputu e fai s vaeluagalemu.taging)
  • Fa'atino le Suiga Tuuina atu Sparksql i Databricks

O le isi auala e vaʻai ai i lenei fausaga e ala i le manatu o galuega ma galuega faʻapitoa

1. Galuega Tu'ueleele - va'ava'aia le fa'agasologa o fa'amatalaga mai fa'apogai eseese i le sone e tulaueleele ai. O le ata faʻataʻitaʻiga o loʻo faʻaalia ai le faʻaogaina o le Qlik Data Gateway - Faʻamatalaga Faʻamatalaga mo le mauaina o faʻamatalaga faʻamatalaga e ala i Suiga Faʻamaumauga Faʻamaumauga (CDC) ina ia mautinoa o loʻo tumau pea faʻamaumauga. E le gata i lea, e mafai ona faʻaogaina fesoʻotaʻiga punaʻoa Qlik Cloud Data Integration e faʻatino ai uta atoa, faʻatagaina mo le toe faʻatulagaina taimi.
2. Galuega Teuina – va'aia le fa'aogaina o fa'amaumauga i laulau teu, e aofia ai le fa'atūina ma le fa'afoeina o laulau ma fafo views. O lenei galuega taua o loʻo i ai se sao taua i le faʻamautuina o faʻamaumauga ma le faʻaogaina i totonu ole Qlik Cloud Data Integration environment. O le galuega o le teuina e le gata e pulea ai le taimi o le faʻaogaina o faʻamatalaga ae faʻamautinoaina foi le tuʻufaʻatasia o faʻamatalaga i totonu o fale teu oloa, faʻaleleia atili le lelei ma le faʻaogaina o le Qlik Cloud Data Integration platform.
3. Galuega Suiga – I totonu o lau paipa faʻamatalaga, e mafai ona e faʻatupuina suiga faʻamaumauga e mafai ona toe faʻaaogaina ma faʻavae tulafono. O nei suiga e mafai ona tu'ufa'atasia lelei i au fa'amaumauga i luga ole fa'aulufalega po'o le fa'atūina e avea ma galuega toe fa'aogaina o fa'amaumauga. O le fetuutuuna'i e fa'alautele atu i le fa'atinoina o suiga o laina-tulaga ma le tu'ufa'atasia o fa'amaumauga e fa'aaoga ai le SQL masani, lea e mafai ona fa'atinoina e pei o laulau pe fa'aalia o le malosi. views fa'aogaina suiga ile lele.
4. Galuega Fa'atauga Fa'amatalaga - A mae'a ona fa'auluina fa'amaumauga, e mafai ona fa'atupuina fa'atauga fa'amatalaga e fa'aaoga ai fa'amatalaga na maua mai i galuega a le Teuina po'o le Suia. Fa'afetaui i mana'oga fa'apisinisi, e tele fa'amatalaga fa'atauga e mafai ona faia. O le mea lelei, o nei fa'atauga fa'amatalaga e tatau ona avea ma fale teu mo fa'amaumauga tu'ufa'atasia, fa'aputuina mo fa'amoemoega au'ili'ili i totonu o se matagaluega fa'apitoa po'o se iunite o se fa'alapotopotoga, e pei o le Matagaluega o Fa'atauga pe fa'aalia fo'i o ni foliga e fa'aaogaina e faiga ML.

Ta'iala fa'atinoga

• E pei ona taʻua muamua i lenei pepa, Qlik Replicate ma Qlik Cloud Data Integration o ni fofo se lua e mafai ona faʻaoga faʻatasi pe faʻatasi. O le fa'ai'uga e fa'aoga fa'atasi pe ta'ito'atasi e fa'atatau ile fa'asologa o fa'amaumauga ma mana'oga ole atina'e. E taua le mafaufau i manaoga o tagata faatau ma faʻaoga mataupu e fuafua ai le fausaga sili ona lelei ma aoga.
  • Tasi i le tele o topologies e manaʻomia i le taimi nei le faʻaaogaina o le Qlik Replicate
  • O punaoa SaaS e lagolagoina naʻo le Qlik Cloud Data Integration
• Atonu e iai mataupu e mafai ona fa'aoga fa'atasi ai le Replicate ma le Qlik Cloud Data Integration. Mo se faʻataʻitaʻiga, e mafai ona faʻaaogaina le Replicate e fafaga ai faʻamatalaga i Databricks mai se punavai e le o lagolagoina nei e Qlik Cloud Data Integration. O faʻamaumauga faʻamaumauga e mafai ona faʻaaogaina e fai ma faʻaoga mo paipa faʻamaumauga na faia e faʻaaoga ai le Qlik Cloud Data Integration.

Databricks Data Intelligence Platform

• O le faavae o le Databricks Data Intelligence Platform o loʻo taoto i totonu o le fale vaituloto, o se faʻafefiloi fou o vaituloto faʻamaumauga ma fale teu oloa. O lenei faiga fou e faʻatatau i le faʻaitiitia o tau ma faʻavaveina le faʻataunuʻuina o faʻamaumauga ma sini AI.
• O le taliaina o mataupu faavae matala ma le tausisia o tulaga tatala, o le fale vaituloto e faʻafaigofieina faʻamaumauga faʻamaumauga e ala i le aveesea o pa puipui faʻasolopito e masani ona faʻalavelave ai le malo o faʻamaumauga ma AI. I le faia o lea mea, e ofoina atu ai se siosiomaga e sili atu le maopoopo ma lelei mo le puleaina ma le faʻaogaina o au punaoa faʻamatalaga.

Tuufaatasi

O se fausaga tu'ufa'atasi e aofia ai le tu'ufa'atasia, teuina, fa'agaioiga, pulega, fa'asoa, au'ili'ili, ma AI. O se auala fa'apitoa mo le fa'afoeina o fa'amaumauga fa'atulagaina ma fa'alilolilo. Se va'aiga atoatoa ile fa'asologa o fa'amaumauga ma fa'atupu mai le amataga e o'o i le fa'ai'uga. O se meafaigaluega tu'ufa'atasi e fa'aogaina ai le Python ma le SQL, api ma IDE, fa'aputuga ma fa'agasolo fa'agasolo, i luga o fa'alapotopotoga tetele uma.

Tatala

• I totonu o le auivi o Databricks, e fa'atumauina pea le fa'atonutonuina o fa'amaumauga, fa'amautinoaina le tuto'atasi mai fa'atonuga fa'apitoa ma fa'alapotopotoga fa'anatura tapunia.
• O le faʻavae o le fale vaituloto e faʻalagolago i luga o faʻalapotopotoga faʻapitoa faʻapitoa e pei o Apache Spark ™, Delta Lake, ma MLflow. E fiafia i le lagolago a le lalolagi e ala i le Databricks Partner Network. E le gata i lea, o le Delta Sharing feature o loʻo tuʻuina atu ai se fofo tatala mo le faʻasoaina ma le saogalemu o faʻamatalaga faʻamatalaga mai le vaituloto i soʻo se fausaga komepiuta. E ausia lenei mea e aunoa ma le manaʻomia o faʻamatalaga faʻamatalaga poʻo faʻamatalaga lavelave, Transform, Load (ETL) faʻagasologa.

Fa'asalaina

• O le fa'ata'ita'iga otometi mo le fa'atinoga ma le teuina o lo'o fa'atulaga ma'oti ina ia mautinoa le tau maualalo maualalo o le Puleaina (TCO) i totonu o fa'amaumauga o fa'amaumauga, fa'atasi ai ma le ausiaina o fa'amaumauga fa'amaumau-fa'amaumauga mo le teuina o fa'amaumauga ma le fa'aogaina o mataupu fa'aoga Artificial Intelligence (AI). E fa'alautele atu i le fa'aogaina o metotia fa'atupu e pei ole Large Language Models (LLMs).
• Po o le a lava le fua faʻalapotopotoga, Databricks ua faʻainisinia e faʻafetaui lelei manaʻoga faʻagaioiga o pisinisi, e amata mai i amataga e oʻo atu i atinaʻe o le lalolagi.

Faleteuoloa SQL x Fa'aputuga Fa'asologa Lautele

Qlik solutions e lagolagoina uma Databricks SQL Warehouses ma Compute Clusters. E lua auala eseese e fa'agasolo ai fa'amaumauga i le ao. I tulaga uma e lua o Qlik solutions o le a tuʻuina atu tulafono SparkSQL e faʻatautaia ai faʻamaumauga, ae le o le faʻalagolago i isi vaega lagolago (pei o api i Scala mo ex.ample). Ole filifiliga ile va ole Databricks SQL warehouses ma fa'aputuga fa'afuainumera lautele e fa'alagolago ile mana'oga ma sini o galuega ta'itasi. O nisi o mea e tatau ona mafaufau i ai:

Tele fa'amaumauga

• Ole taimi ole CDC: SQL Warehouses e vave ona vili i luga pe a fai i le taimi muamua, ma fuifui e mafai ona faagesegese e tali le poloaiga muamua.
• Scalability ma elasticity: SQL faleteuoloa e mafai ona otometi ona fua i luga pe lalo e faʻafetaui le manaʻoga o tagata faʻaoga tutusa ma fesili. E mafai fo'i ona fua fa'aputuga fa'afuainumera lautele ae mana'omia le tele o fa'aoga tusilima ma fa'alogo.
• Saogalemu ma pulega: O faleteuoloa SQL e tuʻuina atu i totonu o le puipuiga malu e pei o faʻamatalaga, faʻamaoniga, faʻatagaina, suʻega, ma le tausisia. E mafai fo'i ona fa'atino e fuifui fa'akomepiuta lautele ni faiga tau puipuiga ae mana'omia nisi fa'atonuga ma pulega.
• I se aotelega, o faleteuoloa o Databricks SQL ma fa'aputuga fa'afuainumera lautele o ni fofo mamana ma fa'atuatuaina mo le fa'agaioia o fa'amaumauga i le ao. Ae ui i lea, e eseese o latou malosiaga ma vaivaiga e tatau ona iloiloina ma le totoa aʻo leʻi filifilia le tasi i le isi.

Fautuaga mo Fa'aputuga Fa'atonu Lautele mo Qlik Solutions

FAAMANATU
O fautuaga o loʻo i lalo e mo faʻamatalaga ma e faʻavae i luga o poloketi ma POCs faʻatautaia e Qlik ma ana paaga e uiga i manaʻoga faʻapitoa. Ole tele o mea e pei ole feso'ota'iga topology, latency, fa'atulagaina o laulau, fa'afouina taimi, fa'aliliuga aveta'avale, ma isi mea e ono a'afia ai tulaga talafeagai mo lau fa'aoga patino. Ua fautuaina tagata fa'atau e fa'atino le fa'ata'atiaga ma le filigā e fa'amautu ai a latou fa'atonuga.

1. Taimi Taimi Databricks
   Siaki i taimi uma le Qlik Replicate ma le Qlik Cloud Data Integration (http://help.qlik.com) e iloa ai po o fea le Databricks Runtime e lagolagoina pe a e faʻapipiʻiina lau fuifui.
2. Databricks Runtime Version Lagolagoina Photon
   A'o e fa'atulagaina lau fuifui filifili le "Photon" mo le fa'asologa o taimi fa'agaienisi a Databricks e lagolagoina ai lau fa'amoemoe lautele. O le Photon o le masini su'esu'e i luga o Databricks, tusia ina ia fetaui tonu ma Apache Spark. O le Photon o se vaega o le ta'aloga maualuga e fa'agaoioia lau telefoni SQL ma DataFrame API o lo'o i ai nei ma fa'aitiitia ai lou tau atoa i le avega o galuega. Mo se isi talanoaga e uiga i Photon, faʻamolemole vaʻai i lenei pepa https://docs.databricks.com/runtime/photon.html
3. Filifili "Memory optimized - Delta cache faʻavave"
   A e faʻatulagaina lau fuifui, ia mautinoa e te filifilia le "Memory optimized - Delta cache accelerated" ituaiga Tagata faigaluega.
   (*) o le lisi o loʻo i luga e faʻavae ile Azure Databricks, e ono suia pe a e faʻaaogaina AWS poʻo GCP

   

4. Fa'atonu Filifiliga Fa'a-Otometi
   Fa'aopoopo se fa'aopoopo i lau fuifui e mafai ai le optimizeWrite ma fa'amalo le autoCompact. O le fa'agataina o le autoCompact e mana'omia e puipuia ai le fa'apipi'i fa'asologa mai le fa'aosoina e fa'afouga taimi moni a le CDC (lea e mafai ona ta'ita'iina ai le fa'atuputeleina o le taofiofia). Ina ia faia lena mea, faʻaopoopo laina o loʻo i lalo i lau Spark vaega o Advanced options o lau fuifui.
   aloiafi. fa'amatalaga bricks.delta.properties.defaults. auto-optimize.optimize Tusi sa'o
   spark.databricks.delta.properties.defaults.autoOptimize.autoCompact sese
   Faamolemole siaki https://docs.databricks.com/clusters/configure.html mo nisi fa'amatalaga e uiga i le fa'atulagaina o lau fuifui.
5. Fa'alelei Laupapa i taimi uma
   E taua le fa'atulagaina o se api e OPTIMIZE laulau i lau Delta Lake. O le a fa'aleleia atili ai le saoasaoa o fesili mo fa'amaumauga na taunu'u. Fa'amolemole fa'afeso'ota'i le pepa lenei: https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/file-mgmt mo sample itiiti o api e fa'asilisili ai laulau.
6. Autoscaling
   Ona o le fesuisuiai o le mamafa o galuega o loʻo tuʻuina mai e le CDC, o le fautuaga o le toeview your configuration based on the workload and testing with your tasks, monitoring and then increasing po o le poloa'igaasing based on the usage. Please refer to Databricks documentation (https://docs.databricks.com/clusters/clusters-manage.html#monitor-performance) mo le auala e mataʻituina ai le faʻatinoga o faʻasologa.

Fautuaga mo Faleteuoloa SQL mo Qlik Solutions

FAAMANATU

O fautuaga o loʻo i lalo e mo faʻamatalaga ma e faʻavae i luga o poloketi ma POCs faʻatautaia e Qlik ma ana paaga e uiga i manaʻoga faʻapitoa. Ole tele o mea e pei ole feso'ota'iga topology, latency, fa'atulagaina o laulau, fa'afouina taimi, fa'aliliuga aveta'avale, ma isi mea e ono a'afia ai tulaga talafeagai mo lau fa'aoga patino. Ua fautuaina tagata fa'atau e fa'atino le fa'ata'atiaga ma le filigā e fa'amautu ai a latou fa'atonuga.
SQL Warehouses e tele naua filifiliga e faʻapipiʻi i le tulaga faleteuoloa (faʻatusatusa i fuifui). Fa'atonuga avanoa.

1. Ituaiga Faleteuoloa
   • O le taimi na tusia ai lenei pepa e tolu ituaiga faleteuoloa. Fa'amolemole va'ai i lenei pepa O a SQL Warehouses? mo se talanoaga lautele e uiga ia i latou
   • Mai se faʻatinoga ma faʻatusatusaga, o le fautuaga lautele o le faʻaaogaina o se faleteuoloa Serverless e faʻateleina ai le faʻatinoga lautele o lau galuega. O nisi siosiomaga ma tala e le o iai lenei filifiliga, i lenei tulaga, o le fautuaina o se faleteuoloa Pro.
2. Fuaina
   Fetuuna'i lenei ta'iala e fa'atuputeleina e fa'atatau i galuega fa'amaufa'ailoga "Numera maualuga o laulau e uta fa'atasi" (Fa'atusa) po'o le "Numera maualuga o feso'ota'iga fa'amaumauga" (Qlik Cloud Data Integration). Ole tulafono lautele ole limamatua ole tasi le fuifui faleteuoloa e fa'agasolo ai le 2 i le 3 laulau po'o feso'ota'iga tutusa.
3. Fuafuga Tele
   • O lenei fa'amaufa'ailoga e fa'alagolago tele i fa'amaumauga o lo'o fa'agasolo. Ole tele o ta'otoga e mafai ona a'afia ai le fa'atinoga lautele e pei ole tele o laulau, numera ma ituaiga koluma o laulau ta'itasi, fa'afouina taimi, ma isi.
   • O le fautuaga lautele e amata i se lapopoa e lelei le tau faʻamoemoeina x faʻatinoga (pei o se ala mo example) ma faia ni su'ega e fetu'una'i ai lenei ta'oto (i luga pe lalo) fa'atusatusa i le laina fa'avae.

Fa'atusatusaga fa'atinoga i le va o SQL Warehouses x General Compute Clusters

FAAMANATU
O lenei su'ega sa faia i totonu o se si'osi'omaga falesu'esu'e ma e le'o fa'atusalia ai so'o se si'osi'omaga moni ola. E ono fesuisuia'i fa'ai'uga e fa'atatau i le ituaiga puna'oa, topology, voluma, tele fa'amaumauga, fa'asologa o fa'amaumauga autu, ma isi fesuiaiga.

Siosiomaga:

• Punavai
  • PostgreSQL o loʻo tamoe i luga o se VM
  • 1 laulau ma 7 koluma ma 36 miliona faamaumauga
  • Ki autu (1 koluma)
  • Su'ega fa'amaumauga a le CDC ma le profile lalo
• Sini (Databricks on Azure) – Stagi luga ole ADLS gen2
• Qlik Replicate Novema 2023 (2023.11.0.149) i luga ole Windows
• Fa'agasologa
  • E lua galuega (puna lava e tasi ma fa'ai'uga 'ese'ese)
  • Amata le uta atoa ile galuega 1 ma fa'atali mo le mae'a.
  • Amata le uta atoa ile galuega 2 ma fa'atali mo le mae'a.
  • Amata le SQL script fa'atusa suiga. O galuega uma e lua e fa'agasolo tutusa ma tu'uina atu i fuafuaga eseese i luga o Databricks
    I'uga:
    I le faʻaaogaina o le Qlik Enterprise Manager e aoina uma fuainumera, na matou matauina e leai se eseesega tele i le faʻatinoga i le va o le faʻaaogaina o SQL Warehouses ma Clusters. Mai le tau, SQL Warehouses e masani ona sili atu le taugofie, lea e mafai ona faʻaalia ai lo latou faʻaogaina i luga o fuifui.

Fautuaga mo Qlik Cloud Data Integration

1. Filifiliga laulau i se Galuega
   Fautuaina e Databricks le tu'u'ese'ese o laulau tetele pe lautele (tele koluma) e faia ai le tele o gaioiga mo a latou galuega. O le fa'aogaina o lenei auala e fa'afaigofie ai ona mata'ituina le fa'atinoga po'o le fa'asoaina o se faleteuoloa i se galuega fa'apitoa.
2. Suiga i se Galuega
   Afai o lau faʻamoemoega o le faʻamalieina o le faʻaogaina o faʻamatalaga sili ona fefaʻatauaʻiga o loʻo faʻaogaina i totonu o le vaituloto, e fautuaina e faʻaitiitia suiga i le tulaga o galuega. O lenei faiga e mafai ai e oe ona faʻasalalau faʻamaumauga e pei ona i ai i totonu o le vaituloto ona faʻaogaina uma ai lea o avanoa faʻainisinia faʻamatalaga e faʻatino ai suiga. O lenei faiga o loʻo taʻua faʻapitoa e faʻaliliuina se ETL (Extract-Transform-Load) i se ELT (Extract-Load-Transform).
3. Faleoloa Fa'asolopito Fa'asolopito (ituaiga 2) ile Teuga / Suiga
   Qlik Cloud Data Integration o le a le mafai ona fatuina le Tala Fa'asolopito Faʻamatalaga (Type 2) aseta e taofia ma pulea faʻamaumauga o loʻo iai nei ma tala faʻasolopito i le aluga o taimi, faʻavae i luga o le ituaiga 2 SCD (Slowly Changing Dimension) manatu. O fa'amaumauga uma o se fa'amaumauga o lo'o taofia, e aofia ai le tapeina, fa'atasi ai ma aso e fa'ailoa mai ai le vaitaimi na ola ai fa'amaumauga ta'itasi. Afai e le talafeagai lenei faʻamatalaga, e mafai ona e faʻamalo lenei vaega i le Teuina poʻo le Suia vaega o le faʻatulagaina o galuega. Ole mea lea ole a fa'asaoina ai punaoa (avanoa ma komepiuta) mai lau avanoa faigaluega.
4. Fa'atino i Galuega Suiga
   O faʻamatalaga e maua mai se galuega suiga e mafai ona faʻaalia i ni auala se lua:
   • Views – lea o le a fa'atino uma ai fesili e faasaga i lenei faalapotopotoga e faasaga i uluai laulau mai le sone e teu ai
   • Laupapa - lea o le a faʻatumuina ai laulau i se auala faʻatulagaina e faʻatatau i faʻamaumauga mai le sone teuina.
     O le filifiliga i le va oi latou o le a faʻavae i luga o le tele o taimi e maua ai faʻamatalaga suiga. O fa'amaumauga e le masani ona suia o le a fa'aalia views, e tatau ona fa'atumauina pea fa'amaumauga e sili atu ona suia e pei o laulau. Afai e iai se manaʻoga e faʻaoga uma, o loʻo i ai le filifiliga e lua (pe sili atu) galuega suiga, tasi e faʻavae i luga views ma isi (s) fa'avae i luga o laulau.
5. Fa'aaogaina Live Views
   • Ola views tu'ufa'atasia fa'amaumauga mai laulau suiga e le'i fa'aogaina i le taimi nei po'o laulau muamua. O lenei fa'aoga e mafai ai e tagata fa'aoga ona maua fa'amaumauga ma le fa'aitiitia o le latency e aunoa ma le mana'omia o le fa'aogaina soo o suiga.
   • O le fa'atuai o le fa'aogaina o le tu'ufa'atasiga e ta'ita'i atu ai fo'i i le fa'asaoina o tau ma fa'aitiitia ai mana'oga mo le fa'agaoioiga i luga o le fa'atonuga.
   • E le gata i lea, ola views ofo atu le advantagu o le le mana'omia o le vaega fa'akomepiuta ia fa'agaioia i taimi uma. E mafai ona fa'aleleia atili le fa'agaoioiga ona ua le toe mana'omia le fa'aogaina o suiga ile aso atoa. O fa'amaumauga fa'ato'a fa'aofiina e vave ona maua i le fa'asalalauga views pe a mafai ona maua i le laulau suiga ma o le galuega e teu ai e ono fa'aitiitia le fa'asaoina o fuifui/fale teu oloa.
6. Taimi Taimi I le Tulaga So'oga
   Ina ia mautinoa le faʻaogaina lelei o le Qlik Cloud Data Integration, e taua tele le faʻapipiʻiina o se meatotino i totonu e taʻua o le executeTimeout ma se tau e sili atu nai lo le 300. O lenei faʻatulagaga e faʻamautinoa ai o le Qlik Cloud Data Integration system o le a tausia se taimi faʻatali a itiiti ifo i le 5 minute aʻo leʻi lesitala. se toilalo. O le mea la lea, e lava le taimi mo le faleteuoloa e amata ai pe afai sa i ai muamua i se tulaga taofi. E taua tele lenei mea i faʻaaliga e manaʻomia ai e le faleteuoloa se taimi umi amata.

Fautuaga mo Qlik Replicate

1. Filifiliga laulau i se Galuega
   Fautuaina e Databricks le tu'u'ese'ese o laulau tetele pe lautele (tele koluma) e faia ai le tele o gaioiga mo a latou galuega. O le fa'aogaina o lenei faiga e fa'afaigofie ai ona mata'ituina le fa'atinoga po'o le fa'asoaina o se fuifui i se galuega fa'apitoa.
2. Suiga i se Galuega
   Afai o lau faʻamoemoe o le faʻamalieina o le faʻaogaina o faʻamatalaga sili ona fefaʻatauaʻiga o loʻo faʻaogaina i totonu o le Lakehouse, e fautuaina e faʻaitiitia suiga i le tulaga o galuega. O lenei faiga e mafai ai e oe ona faʻasalalau faʻamaumauga e pei ona i ai i totonu o le Lakehouse ona faʻaogaina uma ai lea o avanoa faʻainisinia faʻamatalaga e faʻatino ai suiga. O lenei faiga o loʻo taʻua faʻapitoa e faʻaliliuina se ETL (Extract-Transform-Load) i se ELT (Extract-Load-Transform).
3. File Fuafuaga Tele
   • O loʻo i ai le Qlik Replicate parameter i le tulaga fesoʻotaʻiga e mafai ona faʻateleina le faʻaogaina o faʻamaumauga. Ua ta'ua o le Maximum file tele (MB) ma o loʻo i lalo o le Advanced settings o lau fesoʻotaʻiga
   • Ole tau fa'aletonu ole 100Mb ma o lenei fa'ailoga e fa'aalia ai le file tele o lo'o tu'uina atu i le stagvaega a'o le'i utaina ile laulau. E mafai ona e vaʻai i lalo le aʻafiaga o le suia o lenei parakalafa mo se laulau ma 100M faʻamaumauga (pe tusa 3.8 GB faʻamaumauga i luga o le puna). E leai se "tulafono auro" mo lenei parakalafa, ae masani lava, e sili atu file tele faʻateleina le faʻatinoga o le faʻaliliuina o faʻamatalaga lea e taua tele i le taimi muamua o le uta atoa.
   • Fa'avasegaga fa'aputu:
   • Laupapa Punavai (Azure RDS Mysql) 

     
     As shown above, there was a very good improvement when increasing le file lapo'a mai le tau fa'aletonu (100MB) i le 500MB, e ui o fa'aopoopoga fa'aopoopo i tua atu o le 500MB i lenei su'ega e itiiti le a'afiaga i le fa'atinoga.
4. Fa'atonu Fa'atonu
   Qlik Toe fa'afo'i suiga laititi mo le tu'uina atu i le Databricks Delta ma le fa'atulagaina o le fa'atulagaina mo se galuega e a'afia ai le tele o le micro-batch na lafo i Databricks.
   • Suia le Faiga Fa'agaioiga: Na'o le Batch Optimized Apply e lagolagoina mo le Databricks targets.
   • Fa'aoga suiga tu'ufa'atasi i le tele o laulau fa'atasi: This option configures the number of threads that will work in parallel to upload and apply data to Databricks. The default value is 5, with a maximum of 50. Increasing this value can improve your throughput when there are many tables with CDC in a given batch however it may require additional cluster resources. Please review o tapula'a o lenei faiga i le Repliates fesoasoani taiala.
   • Fa'aoga suiga tu'ufa'atasi i fa'atulagaga vaeluaga: Fa'atulaga le taimi ma le tele o le micro-batch.
   • Umi atu nai lo (sekona): This specifies the minimum amount of time to wait between each application of batch changes. The default value is 1 and typically is too low a value for Databricks delta apply processes. Increasing this value decreases the frequency with which changes are applied to the target while increasing the size of the batches, essentially creating larger batches at the expense of some additional latency. It is recommended to start with a value of 60 and increase even further if some additional latency is acceptable. In some cases waiting for larger batches can improve throughput and latency
   • Ae itiiti ifo i le (sekona): O lenei tau o loʻo faʻamaoti mai ai le maualuga o le taimi e faʻatali ai i le va o talosaga taʻitasi o suiga o vaega (aʻo leʻi faʻaalia se taimi malolo). I se isi faaupuga, o le latency aupito maualuga taliaina. O le tau fa'aletonu o le 30. O lenei tau e fuafua ai le maualuga o le taimi e fa'atali ai a'o le'i fa'aogaina suiga pe a uma ona ausia le tau o le Umi atu (sekona). E fautuaina e fa'atulaga lenei tau i le 120 (tu'ufa'atasia ma le umi atu nai lo le tau o le 60 ma fa'aoga le tau e sili atu pe a talia le tele o le latency.
   • Fa'amalosi e fa'aaoga se vaega pe a sili atu le manatua (MB): o lenei fa'atulagaga o lo'o fa'amaoti mai ai le aofa'i aupito maualuga o le manatua e fa'aoga mo le fa'agaioia muamua ile faiga fa'aoga fa'alelei o le Batch. O le tau le aoga o le 500. Mo le maualuga o le aofaʻi, seti lenei tau i le maualuga maualuga o le manatua e mafai ona e tuʻuina atu i le Qlik Replicate. E fautuaina e amata i se tau o le 2000 ma mafaufau e faʻalogo maualuga pe a lava punaoa i luga o le Qlik Replicate server.
   • Fa'aoga Suiga e fa'aaoga ai le Fa'atasi: e mafai ai e le galuega ona faʻaaoga tulafono SQL MERGE e faʻavave ai le ingestion i le Lakehouse.
5. Vaevae Laulau tetele
   • Databricks e maua ai le avanoa e vaelua ai laulau Delta. E fautuaina le vaeluaina o laulau tetele e mafai ona avea ma fagu i le faagasologa o talosaga. Qlik Replicate e le o lagolagoina i le taimi nei le faʻatulagaina o vaeluaga autu i totonu o le galuega. Ole la'au fa'atatau Delta e tatau ona faia e le Qlik Replicate ona toe faia lea i koluma vaeluaga talafeagai. Afai e faʻamatalaina se laulau e vaelua, e fautuaina e seti le galuega e fai ai se TRUNCATE mo uta atoa.
   • E ui o le vaeluaga o se manatu tuusa'o, o le fuafuaina o koluma vaeluaga sili ona lelei (s) e mana'omia ai se malamalamaga mausali i le auala e suia ai faʻamatalaga e le talosaga. E le fautuaina le vaeluaina o le ki autu ona o atugaluga cardinality. O laulau tetele e mana'omia le vaeluaga e masani lava
     “fefaatauaiga” i le natura – eg fa'amatalaga fa'atau. E masani lava ona filifilia se koluma o aso po'o le fa'aopoopoina o se koluma YEAR_MONTH i fa'amaumauga fa'atatau i totonu o le Replicate e maua ai se auala lelei mo le vaeluaga. Lalo o se example a'afiaga o le vaeluaga i le fa'aogaina o fuifui ma fa'apea ai le le tumau. I lenei example, o se laulau e tusa ma le 68 miliona punaoa laina / 655 GB o faʻamaumauga o loʻo faʻagasoloina se gaosiga o galuega CDC. O le vaeluaina o le laulau delta e faʻaaoga ai se koluma DATE na maua ai le 73% faʻaitiitiga i le latency ma se faʻaititia tele o le mafaufau ma le CPU faʻaaogaina i luga o le fuifui.

Fa'aaogāga Fa'aputuga – E Le'i Vaevaeina

Fa'aopoopo I – Fausia So'oga

E matua fautuaina lava e te siakiina pepa o loʻo maua ile help.qlik.com mo manaʻoga lata mai ma faʻapipiʻi lagolago mo lau siosiomaga Databricks
Ina ia fatuina se fesoʻotaʻiga faʻamatalaga i le Qlik Replicate i se faʻataʻitaʻiga Databricks e te manaʻomia:

1. Server Hostname
2. Taulaga
3. HTTP Auala
4. Faailoga

Fa'apotopotoga
O faʻamatalaga 1, 2, ma le 3 e mafai ona maua mai i le Databricks console e ala i le alu i le faʻapipiʻi faʻapipiʻi poʻo lau SQL Endpoint configuration, ma i lalo o le Advanced Options section o le ae mauaina le JDBC/ODBC tab

Databricks SQL Faleteuoloa
O faʻamatalaga 1, 2, ma le 3 e mafai ona maua mai le Databricks console e ala i le alu i le SQL Warehouse vaega i lalo o le "Faʻamatalaga fesoʻotaʻiga" tab

Ina ia maua se faʻailoga avanoa, e tatau ona e alu i le User
Vaega fa'atulagaina o lau fa'amafanafanaga Databricks ma fa'aoga le fa'amau Fa'ailoga fou.

E taua le teuina o le fa'ailoga ua gaosia i se nofoaga saogalemu aua e le mafai ona e toe aumai pe a uma ona e tapunia lenei talanoaga.

Pepa / Punaoa

Qlik Talend Data Integration Solutions [pdf] Taiala mo Tagata Fa'aoga Faʻamatalaga Faʻamatalaga Faʻamatalaga Tale, Faʻamatalaga Faʻafesoʻotaʻi Faʻamatalaga, Fofo Faʻatasi, Fofo

Fa'asinomaga

• Tusi Taiala