What is everyone using to extract data from datalake/cloud? Is it necessary to use datalake to extract data from MT cloud to an external database or can you also bypass datalake to extract data directly?
We are using a combination of streaming pipelines and Compass APIs. We use the streaming pipelines for our real-time analytics on our master data and transactional tables. Streaming pipelines completely bypasses the Data Lake ONCE the initial load is sent to your Postgres or SQL Server ODS. The initial loads do use the Data Lake but once the initial load is complete, the streaming services directly from M3 work well to maintain the data. We use Postgres in Azure as our ODS and it's quite cost Read more...effective.
For our financial and detailed transaction information like inventory transactions where the data load is huge we do use the Fabric Compass APIs to access that data. Typically every 30 minutes, every few hours or daily depending on the data being collected and how it's used in reporting. For that we use Python to access the Compass API's and post the query results directly to a SQL Server ODS.
We then use MS SQL Server Integration Services (SSIS) to collect the M3 data in our ODS databases (we have many other 3rd party sources) and perform good old-fashioned ETL to an analytics database in SQL Server. Then Power BI takes over. Show less...
Streaming pipelines and Streaming services are the most popular right now, from Infor. There are other means to using JDBC drivers as well. The best approach would depend on your objectives and latency duration for the data.
There are a few options depending on where you want to extract data to. Among them are streaming pipelines offered by Infor to export to Azure or AWS postgres or MS SQL, something like Precog that allows for extracting to a variety of different destinations, or something like our connector to go to Microsoft Fabric.
You can also either use the Compass JDBC driver or the API to roll your own solution.
Some of the above can go more or less directly from the application Read more...to the destination, bypassing the Infor Data Lake. Show less...
Share a Message
For our financial and detailed transaction information like inventory transactions where the data load is huge we do use the Fabric Compass APIs to access that data. Typically every 30 minutes, every few hours or daily depending on the data being collected and how it's used in reporting. For that we use Python to access the Compass API's and post the query results directly to a SQL Server ODS.
We then use MS SQL Server Integration Services (SSIS) to collect the M3 data in our ODS databases (we have many other 3rd party sources) and perform good old-fashioned ETL to an analytics database in SQL Server. Then Power BI takes over. Show less...
Share a Message
Share a Message
You can also either use the Compass JDBC driver or the API to roll your own solution.
Some of the above can go more or less directly from the application Read more...to the destination, bypassing the Infor Data Lake. Show less...
Share a Message