Changes/Enhancements in IBM Datastage 8.5, 8.7 and 9.1


1. Design & Runtime Performance Changes:
DS8.5: Implemented by Internal code change. Design and Runtime performance is better than 8.1, 40% performance improvement in job open, save, compile etc.
DS8.7: Improvements in Xmeta. Significant Performance improvement in Job Open, Save, Compile etc.
DS9.1: No Change

2. PX Engine Performance Changes :
DS8.5: Not Exist
DS8.7: Improved partition/sort insertion algorithm. XML parsing performance is improved by 3x or more for large XML files.
DS9.1: No Change

3. Added View Job Log in Designer client
DS8.5: Not Exist
DS8.7: New Feature has been added (Menu -> View -> Job Log) .
             Job log is now viewed in Designer client.
DS9.1: No Change

4. Added Stop/Reset buttons In Designer Client:
DS8.5: Not Exist
DS8.7: Stop/ Reset button added to Compile and Run buttons for the DS jobs.
DS9.1: No Change

5. Interactive Parallel Job Debugging
DS8.5: Not Exist
DS8.7: Breakpoints with conditional logic per link and node. 
          (Link -> Rclick -> Toggle Breakpoint)
          The running job can be continued or aborted by using multiple breakpoints with                 conditional logic per link and node. (row data or job parameter values can be                   examined by breakpoint conditional logic)
DS9.1: No Change


6. Added Vertical Pivot : Pivot Enterprise Stage
DS8.5: Extended to current horizontal parallel pivot. Enhanced pivot stage to support vertical     pivoting. (mapping multiple input rows with a common key, to a single output row containing multiple columns)
DS8.7: No Change
DS9.1: No Change


7. Balance Optimization :
DS8.5: Balanced Optimization is that to redesign the job automatically with maximize performance by minimizing the amount of input and output performed, and by balancing the processing against source, intermediate, and target environments. The Balanced Optimization enables to take advantage of the power of the databases without becoming an expert in native SQL.
DS8.7: No Change
DS9.1: Balanced Optimization for Hadoop.


8. Transformer Enhancements
DS8.5: Looping in the transformer, Multiple output rows to be produced from a single input row. 1. New input cache: SaveInputRecord(), GetSavedInputRecord().
         2. New System Variables: @ITERATION, @Loop Count, @EOD(End of data flag for last row).
         3. Functions : LastRowInGroup(InputColumn).
         4. Null Handling more Options.
DS8.7: No Change
DS9.1: New Transformation Expressions has been added. EREPLACE :  Function to replace substring in expression with another substring. If not specified occurrence, then each occurrence of substring will be replaced.


9. Big Data File Stage :
DS8.5: Not Exist
DS8.7: Big Data File Stage for Big Data sources (Hadoop Distributed File System-HDFS).
DS9.1: New Enhancement on
1. The IBM Big Data Solution Integrate and manage the full variety, velocity and volume of data.
2. New Hadoop-based Big Data Support Any to Big Data.
3. Big Data Integration with DataStage.

10. Added Java Integration Stage
DS8.5: Not Exist
DS8.7: Not Exist
DS9.1: Java code and creates baseline for upcoming big data source support.

11. Added Encryption Techniques
DS8.5: Not Exist
DS8.7: Encrypted because of security reasons.
1. Strongly encrypted credential files for command line utilities.
2. Strongly encrypted job parameter files for dsjob command.
3. Encryption Algorithm and Customization.
DS9.1: No Change

12. Added Dual-stack protocol Support
DS8.5: Not Exist
DS8.7: IPv6 Support: Information Server is fully compatible with IPv6 addresses and can support dual-stack protocol implementations. (Env  Variable: APT_USE_IPV4.)
DS9.1: No Change

13. Added Unstructured text stage
DS8.5: Not Exist
DS8.7: Not Exist
DS9.1: Excel read capabilities on all platforms with rich features to support ranges, multiple worksheets and New Unstructured data read.

14. Added DBMS Connector Boost
DS8.5: Not Exist
DS8.7: Not Exist

DS9.1: New big buffer optimizations which has increased bulk load performance in DB2 and Oracle Connector by more than 50% in many cases