Requirements:
- Source and destination system impact
- Processing time windows and performance
- Destination system state consistency
- Hard and soft exception handling and restartability needs
- Environment architecture model, distributed hardware, or scaled-up servers
- Solution architecture requirements, such as flexibility of change or OEM targeted solutions
- Modular and configurable solution needs
- Manageability and administration requirements
Data Flow Design Practices
- Limit synchronous processes.
- Monitor the memory use of blocking and semi-blocking transformations.
- Reduce staging and disk I/O.
- Reduce reliance on an RDBMS.
Leveraging the Data Flow
Data Integration and Correlation
Data Cleansing and Transformation
Troubleshooting Data Flow Performance Bottlenecks
The pipeline execution reports (reviewed earlier in the chapter) are a great way to identify which component in your Data Flow is causing a bottleneck. Another way to troubleshoot Data Flow performance is to isolate transformations and sources by themselves.
Windows Performance Monitor