Data management tools & concepts in D365 F&O and LCS
In my last two posts, we looked at the various available data management concepts, tools available within the Data Management Framework, and LCS. We also looked at how to manage the data and configuration by leveraging with the discussed tools.
In this post, I will discuss the end-to-end picture of all the tools available in D365 F&O and LCS that you can use to manage the data and configuration in the implementation and beyond.
1. Toolsets Recap
Different toolsets and frameworks are available for data management in a Dynamics 365 Finance and Operations implementation project.
1.1. Data Import/Export framework
Dynamics 365 Finance and Operations provides a comprehensive functionality within the application and LCS to manage:
- Configurations
- Data Migration
Source: Microsoft
1.2 Levels of Aggregation and Abstraction
Data entities provide a conceptual abstraction and encapsulation (de-normalized view) of the underlying table schema that represents the data concepts and functionalities. The diagram below shows the levels of aggregation and abstraction available in the data management framework.
Once the data entities and the related entities are identified, they can be grouped in a data package, where you can assign the dependencies. These data packages are created using a data project.
Use and load the default data templates as provided by Microsoft in Dynamics 365 Finance and Operations or creating own templates by identifying and assigning your required data entities and load them in D365 F&O. These data templates can then be reused in a different environment with the necessary data by creating data projects.
Apart from the data management workspace available, you can leverage the LCS to create data packages. These tools primarily start with data packages. These are the same data packages that you can create in D365 F&O, Enterprise edition, and add in the LCS asset library to import data and configuration in an environment. Additionally, you can make use of the LCS Configuration and Data Manager and create your own LCS Process Data Package to copy configurations.
2. Integration scenarios
2.1 Synchronous service (OData)
Data entities enable public application programming interfaces (APIs) on entities to be exposed, which enables synchronous services. Synchronous services are available for the following purposes:
- Office integration – Microsoft Data Connector - Open in Excel and Export in Excel experience in Dynamics 365 F&O
- Third-party mobile apps Integration with third-party mobile apps through APIs.
2.1.1 Asynchronous integration
Data entities also support asynchronous integration through a data management pipeline. This enables asynchronous and high-performing data insertion and extraction scenarios. Here are some examples:
- Interactive file-based import/export
- Recurring integrations (file, queue, and so on)
2.1.2 Business intelligence
- Aggregate data
- Standardized key performance indicators (KPIs)
3. Database Operations
Working on an entire database could be a requirement in a project especially when creating a new environment and integrating it with the data in golden configuration or spinning other environments and you can achieve this by some additional tools available and discussed as follows:
3.1 Customer / Partner managed
One must acknowledge the differences in the database platform when using Azure SQL and SQL on the premises. The process involves the following information flow when moving the database from Azure SQL to Microsoft SQL Server (on-premise).
3.2 Dynamics Service Engineering (DSE) – Service Requests
- Database refresh request
- Data point-in-time restore request
LCS offers tools for initiating database operations such as backup, restore, and so on, using the support request from LCS.
For more details, please refer the following links:
- https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/database/dbmovement-operations
- https://docs.microsoft.com/en-us/azure/sql-database/sql-database-copy?tabs=azure-powershell
4. Data Sharing framework
Cross company data sharing simplifies the management of master data for customers with multiple legal entities. For example, if your implementation project is dealing with multiple legal entities, but some parameter and setup data is shared across the legal entities, you can make use of the cross-company sharing feature to share the data.
Data sharing is managed by defined policies that are saved in data packages. Templates that Microsoft has tested and supports are available as downloadable data packages on Microsoft Dynamics Lifecycle Services (LCS). Policies let you control the following aspects of data sharing:
- The fields that are replicated
- The entities that participate in the replication
For example, leverage the customer and vendor master data from the data packages available in LCS. This feature is available for customers on version 8.0 and later.
Make use of the configuration and data manager to import the package in an applicable environment [e.g. sandbox] and enable cross-company data sharing from the introduced policy in Dynamics 365 Finance and Operations environment.
4.1 Limitations
- Sharing of transactional records for reporting or management purposes, such as orders and consolidations
- Sharing across deployments
- It supports the replication of fewer than one million total records per job. (This total is calculated as the number of shared records × the number of shared companies.)
- Only one level of child relationships is exposed. To protect data consistency, replication doesn't occur if another level is required.
For more details, please refer the following links:
- https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/sysadmin/cross-company-data-sharing
- https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/tasks/configure-financial-cross-company-data-sharing
5. Copy the company configuration within the existing environment
There are two options for copying configuration data in Finance and Operations:
- To move data between instances, you must first export it from one company and then import it to another company.
- To move data from one legal entity to another legal entity in the same instance, you can use the Copy into legal entity feature.
You can create a data project and select multiple legal entities as applicable, copy number sequence across the entities, and select default or own designed data templates for copying the parameter and reference data.
For more details, please refer the following link:
https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/copy-configuration
6. Copy data between environments
If your data import-export process is validated, then make use of database copy functionality for data movement to an environment, including production.
If the requirement is for an incremental load, you can use the data management tools such as data entities, data packages within Microsoft Dynamics 365 for Finance and Operations. You can also utilize the Refresh database functionality to copy the transactional and financial databases into the sandbox UAT environment.
For more details, please refer the following link:
https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/database/database-refresh
7. Recommendations
- Have the right data management strategy as it has a direct impact on all users who will use the system.
- There are several data management techniques and tools; it is highly essential to select the right device that fits the purpose.
- Identify the data entities in scope for your project as you configure
- Make use of the default templates provided by Microsoft and design data packages based on the scope of implementation
- Use PDP to create new legal entities and just modify change data or new shared data.
- Make use of the data-sharing framework to copy parameter and reference data between legal entities
- Leverage the LCS and other useful tools like DevOps to manage the project tasks, design documents, code, build, and other artifacts and document handling through tools like SharePoint online and LCS project document libraries [especially for testing by integrating with RSAT[regression suite automation tool].
I hope this series will help you in adopting a better strategy for data management by understanding the key data management concepts and leveraging the right data management tools.