In this post I will share my experience on the Migration process performed for Electricity and Gas Data, using Emigall tool-set for Utilities. I will walk through the road map we followed during the different stages of the migration life cycle. The primary focus will be on the use of the Emigall tool-set in the different phases of the migration cycle.

Before we proceed further, please read this fine print. All the views provided below are my personal opinion and doesn't necessarily reflect my employer's. I work for my employer as an IS-U CRB ABAP consultant with a focus in Utilities Industry but every customer's requirement is unique and you should definitely seek for professional opinion before making your business decisions. Treat the inputs in this blog series as just opinions, nothing more, nothing less.

SAP Data Migration Approach:

SAP recommends the Rapid Data Migration and Data Quality for the migration of non-SAP data to the SAP system.

They depict the migration process as below

SAP Rapid DM and DQ provide a migration framework, including documentation, templates, methodology, tools and expertise to analyze, extract, cleanse, validate, upload and reconcile legacy data and accelerate your migration project.

Details can be found in the SAP links as below

http://service.sap.com/bestpractices

https://websmp103.sap-ag.de/rds-dm2u

Migration Life cycle:

There are many resources available online on the Data Migration process, but here I will cover the following stages as road-map for Data migration life-cycle and how we used Emigall tool-set in each phase.

1. Data Scanning/Profiling
2. Data Identification.
3. Data Mapping.
4. Data transformation.
5. Data Load.
6. Data Cleansing.
7. Data Reconciliation.

In my next blog I will write on the approaches taken for performance tuning the Emigall Loads.

1. Data Scanning or Profiling

1.1     Migration objects streaming

The object streaming happens in the Blue print Design Phase of the project. The Business streams are identified based on functional processes and data are categorized under specific streams. In our implementation we had profiled the data into the below streams.

1. Front Office
2. Device Management
3. FICA
4. Billing
5. Service Location Master Data
6. Address/Regional Structure
7. Service Management

For the streams identified the next step is to classify each object within each stream at a higher level. Business Owner, functional and the Migration team go through the requirements and based on the outcome:

1. Standard SAP Migration objects from the ISMW are identified.
2. Custom objects are built to cater to the additional requirements.

An example list of objects that we identified and the associated stream is attached here

1.2 Object dependencies:

Dependencies set the relationships among objects. If the objects are migrated out of dependency, it could lead to serious issues with the data not meeting the required functionality. The worst case would be re-loading of the entire data all over again or a series of manual corrections.

The objects within each stream have dependencies and are also related to each other across the stream, leading to Cross Dependencies. These relationships are outcome of functional and business requirements.

We found that setting up the object relationship earlier in the blueprint phase worked well for us. A team of Business Owners, functional and Data Load team worked on establishing the relationships.

1.3 Object sequencing

Once the objects relationships are established, the cross stream objects need to be sequenced appropriately. This sequence is then followed during the loads. From the objects identified and the dependencies set the following sequencing was used for the Data loads.

Based on the object sequencing the load strategy was designed. The strategy had to ensure parallel loads of the objects and the best possible use of the background  work processes . Once the object dependencies and sequencing are identified the next stage is the Data Identification.

2.  Data Identification

The Emigall objects were identified, set up, and sliced and diced for loads in this step. The identified Emigall Objects were scrutinized to generate the required automation structures and the fields. The details of the Emigall configuration and the specific object configurations can be found in the SAP documentation from the below linkGuidelines_ISMW

Following the below steps worked well for us in setting up the Emigall objects:

1. Data mockup was carried out from SAP GUI screen through the transaction codes, for e.g. FPP1 for creating the Contract Partner. This step helps in answering the below questions,

                what are the mandatory fields in the transaction

                what fields are expected to have values from the legacy system

                what are expected values for the fields, e.g. ID types

                how the final data should look like.

   2. From step 1 the Emigall Object was looked up for generating the automation structures and fields within each structure.

   3. Once the Emigall object is generated along with the data field, sample data file is mocked in Emigall, by creating Data import files from the Emigall transaction.

   
   4. Data loads are carried from the data import file. The load result should exactly mimic the data created from the front end in step1.

   5. Steps 1 to 4 can be iterative depending upon the Business scenarios and identified scope. Based on the scenarios, the field processing types ( rules, conversion set , initial values etc) are identified in this step.

   6. For objects that cannot be handled by standard SAP, custom objects are created. The guidelines documentation describes the creation of Custom objects in EmigallGuidelines_ISMW

These steps were carried out in presence of the functional folks and the data load team for each object.

3.  Data Mapping

This stage identifies the Mapping rules between the source system data and the target system. The target field values in Emigall are identified and appropriate mapping rules are established. As an example the Connection object Regional Structure data could be numeric codes in the source system while it’s configured as alpha numeric in the target system.

Emigall provides, known as processing types, to implement the mapping through rules, conversion set and fixed value. We considered these processing types only when the mapping rules were simple one to one mapping and did not have table fetch in SAP. More details and techniques can be found in the Guidelines_ISMW.

For complex rules involving look up tables and cross references, it worked well for us to implement them in the Data Transformation step.

4.  Data Transformation

Data Transformation is the meat of the Data Migration process. Files are generated at this stage based on the Emigall Data structures identified and the mapping rules defined. The outputs of the Data Identification and Data Mapping steps are supplied to the Data Transformation along with the extracted files from legacy system.

The Rapid Data Solution for Migration in Utilities details out the transformation process using BOBJ DS. Data Migration for Utilities

We ensured that the file extraction and subsequent transformation process were designed in such a way that the lag time/wait time for loading activity was minimal.

We considered having 3 mutual streams of Business Partner, Service Location Master, and Device Management at the Master data level ensuring some room of parallel loads. The transformation process was sequenced in such a way, that the files were transformed and ready for loading activity in either of the above streams without encountering idle time.

5.  Data Loading

The data loading activity was achieved in the following ways depending upon business requirements.

1. Standard / Custom  Emigall objects.
2. Custom Reports/BDC
3. LSMW.

We considered building custom Emigall object rather than developing ABAP reports,  when standard SAP objects did not meet our requirements. The custom Emigall objects can reuse the capability of Emigall. Emigall is the powerful tool equipped with the exhaustive error logs, load restart capability, data analysis and faster means of loading which can be put to best possible use rather than building a custom report and testing them.

5.1.       Things to consider before the actual execution of Emigall loading:

1. File Conversion: Standard file conversion program REMIG_FILE_TRANSFORM was used to convert the files into Emigall compatible (Migration) format. If the files are not delivered in the format expected by this program then custom program was developed for the purpose.
2. File Size:Once the file is converted to Emigall compatible format standard program REMIG_PARSE_FILE was used to break down the file to smaller files of certain packet size. The smaller the file size the better the performance of the load jobs.
3. Job Parameters:Appropriate object parameters were set before loading, for each object. The Commit interval for different objects differs. SAP recommends commit interval for each object was followed.Guidelines_ISMW.
4. Job scheduler: Standard data import utility REMIG_JOB_SUBMIT was used to split the files and schedule the jobs by defining the work processes and the application server.
5. Work Processes: The higher the number of Work Processes the higher the capacity to run parallel loads.

It worked well for us when the Data Load team worked with the Basis to determine the appropriate number of background work processes for each object load. We executed 2 to 3 different object loads in parallel with 100-110 background Work Processes in total.

Further details on the scheduling strategy can be found in the Guidelines_ISMW.

5.2        Job Monitoring tools:

The required tips and tricks for monitoring the jobs can be found in the Performance Guidelines for Emigall Perf_Cookbook but I would like to point out certain specific parameters we considered during our execution.

1. Process Overview: The transactions SM66 and SM50 were monitored extensively throughout the data migration. Fine tuning was performed based on the observations in these transactions.
2. Lock table overview DB01: Lock tables in active and wait state was monitored and traced for the jobs. Basis team performed monitoring the handling of the lock entries.
3. DB Stats: The DB stats for the tables were monitored and updated consistently by the Basis team when the jobs were running. For intense jobs such as Device Technical Install and Billing install the same tables (EABL, EABLG, EASTL etc) gets updated frequently and required stats updated. Perf_Cookbook has the details on the required DB stats.
4. Performance trace: During the test runs on high volume loads, the performance trace (ST05) was activated and monitored to see the DB tables being hit and the performance on these tables. Appropriate indexes were created / updated on certain tables depending upon the trace result.
5. OS level: Basis monitoring the OS level parameters provides insights on the CPU Utilization, I/O load, network response, memory utilized and DB management and help to determine correct packet size and Work Processes to be used.We executed this exercise to determine the correct figures for Business Partner and Meter Read loads, which were processing intensive. This helped in bringing down the load time to fit the cut-over window.
6. Error Statistics: Emigall provides a powerful tool to monitor the jobs. The throughput of individual jobs and the error count is monitored from this tool. EMIGSTATS transaction code is the door to this tool.

5.3.      Approaches to Load:

Two approaches to loads were considered

1. Using Distributed Import
2. Manual break down and scheduling

5.3.1     Using Distributed Import

Emigall provides the means to automate the distribution of the data import files during loads. It takes the migration converted file names for conversion and the Error file name. The split for the error files is automatically handled by the Standard SAP Master Program.

A screen shot of the auto scheduler

5.3.2       Manual break-down and scheduling.

For a larger volume file size the manual break down of the file into packet size and the scheduling using the standard program worked well for us.

It has the below steps

1. Break down the file using the program REMIG_FILE_PARSE. Provide the packet size and names for the smaller files.

    2.  Once the file break down is done, run the scheduler program with appropriate inputs to start the scheduler. The program name is REMIG_JOB_SUBMIT.

    3. The scheduler can be used to schedule the jobs by selecting the application server name, date and time for the job execution.

We found the manual scheduling approach better suited for high volume loads. Also we were able to reduce the error count to less than 1% of the total file, so the restart of the entire file was very easy and the restart produced a single file in a short time.

Moreover it provided a better way to tag team with the basis and allocate the jobs to the desired servers without impact the run of the other parallel jobs. More details on the load scheduling can be found out in the Guidelines_ISMW.

5.4     Error Analysis and Migration Statistics:

Detail coverage of the error analysis and looking at the Migration statistics is given in the Emigall guidelines Guidelines_ISMW.

Below I will mention the steps we took to analyze the error messages coming out as load rejects

1. The errors encountered for each of the load job was analyzed and statistics read from the error logs.

    2.   To get details on the errors we can highlight the error message and click on the long text to get the SAP suggestions on the error message. When the long text messages were quite obvious for the corrections to be done, this method worked very well to give us the first pass on the errors.

     3.   If the error long texts details were not sufficient, the next step was to run the particular record in a separate file in the online mode by setting the below parameter. This makes the execution of the data as if it is run as a BDC online. Uncheck the flag “W/o screens” and run the data file for the error record to find the error on the online mode.

       4.  In cases when debugging was essential debug mode was used, when the 1^st three steps above did not suffice.

5.5.     Load Performance

Improving the Performance of the Load is the key to the effective Conversion completion in the cut-over window.We had to apply series of Performance measurement techniques to arrive at a total load time fitting the cut-over window. The load time only bettered for every load from the regression tests through the Mocks and the preferred one was during the go live.I will mention the tip and tricks that we approached to improve the load performance in a separate blog.

 
6.  Data Cleansing

After the initial set of data was loaded into SAP, the Emigall rejected data due to cleansing issues were corrected in Emigall through set of defining rules and were loaded into SAP by restarting the Error file load in Emigall.

This approach was taken to push in the Master Data of Business Partner, Devices, Installations and the Partner relationship and Business Partner Contacts.

Using Emigall to cleanse data was chosen as the time and effort that would have required to clean the affected data in the legacy system was considerable. Writing simple rules in Emigall and re loading just the error files was an easier way of approaching the migration.

It’s very important to load as much of the data as possible, especially Master data into SAP, as it would give way for the downstream to get into the SAP. Else the downstream objects will be seen failing due to missing Higher Level Objects (HLO’s).

7.  Data Reconciliation

In this section I am mentioning about the data validation/reconciliation part that was performed by the load team to maintain consistency in the number of records.

Basically, with our number we looked to match the below equation

Number of records in the    =   Data loaded in SAP     +        Data rejected

Transformations file                  Through Emigall                      in Emigall

The above 3 parameters can be retrieved from the Emigall Statistics as below.

The records migrated by Emigall get entered into the migration table TEMKSV per object under the user specific company. As soon as the migration is performed for each object the table count on TEMKSV has to be performed.

A validation check has also to be made on the SAP DB table using query or a simple number count on the SAP tables involved.

In the end, load team can reconcile the data records as below for the end to end validation on the number of records created.

Data loaded in SAP        =          Number of records       =       Number of records in

Through Emigall                         in TEMKSV                              DB table in SAP.

Additionally we considered developing custom reports to perform the reconciliation of the financial data between  the legacy system and SAP. FICA Documents, Payments and Security Deposits were reconciled, ensuring that there is a penny to penny match in the migrated data.

I will continue on my next blog with the steps we took to improve the load performance.