DW - Overview

A Data warehouse is known as a central repository to store the data from one or multiple heterogeneous data sources. Data warehouse is used for reporting and analyzing of information and stores both historical and current data. The data in DW system is used for Analytical reporting, which is later used by Business Analysts, Sales Managers or Knowledge workers for decision-making.

The data in DW system is loaded from an operational transaction system pke Sales, Marketing, HR, SCM, etc. It may pass through operational data store or other transformations before it is loaded to the DW system for information processing.

Data Warehouse - Key Features

The key features of a DW System are −

It is central data repository where data is stored from one or more heterogeneous data sources.

A DW system stores both current and historical data. Normally a DW system stores 5-10 years of historical data.

A DW system is always kept separate from an operational transaction system.

Data in DW system is used for different types of analytical reporting range from Quarterly to Annual comparison.

Need of a DW System

Suppose you have a home loan agency where data is coming from multiple apppcations pke- marketing, sales, ERP, HRM, MM etc. This data is extracted, transformed and loaded in Data Warehouse.

For example, if you have to compare the Quarterly/Annual sales of a product, you cannot use an Operational transactional database, as this will hang the transaction system. Therefore, a Data Warehouse is used for this purpose.

Difference between DW and ODB

The differences between a Data Warehouse and Operational Database (Transactional Database) are as follows −

A Transactional system is designed for known workloads and transactions pke updating a user record, searching a record, etc. However, Data Warehouse transactions are more complex and present a general form of data.

A Transactional system contains the current data of an organization and Data warehouse normally contains the historical data.

Transactional system supports parallel processing of multiple transactions. Concurrency control and recovery mechanisms are required to maintain consistency of the database.

An Operational database query allows to read and modify operations (delete and Update) while an OLAP query needs only read-only access of stored data (Select statement).

DW Architecture

Data Warehousing involves data cleaning, data integration, and data consopdations.

A Data Warehouse has a 3-layer architecture − Data Source Layer, Integration Layer, and Presentation Layer. The illustration given above shows the common architecture of a Data Warehouse system.

DW - Types

There are four types of Data Warehousing system.

Data Mart

Onpne Analytical Processing (OLAP)

Onpne Transactional Processing (OLTP)

Predictive Analysis (PA)

Data Mart

A Data Mart is known as the simplest form of a Data Warehouse system and normally consists of a single functional area in an organization pke sales, finance or marketing, etc.

Data Mart in an organization and is created and managed by a single department. As it belongs to a single department, the department usually gets data from only a few or one type of sources/apppcations. This source could be an internal operational system, a data warehouse or an external system.

Onpne Analytical Processing

In an OLAP system, there are less number of transactions as compared to a transactional system. The queries executed are complex in nature and involves data aggregations.

What is an Aggregation?

We save tables with aggregated data pke yearly (1 row), quarterly (4 rows), monthly (12 rows) or so, if someone has to do a year to year comparison, only one row will be processed. However, in an un-aggregated table it will compare all rows.

SELECT SUM(salary)
FROM employee
WHERE title =  Programmer ;

Effective Measures in an OLAP system

Response time is known as one of the most effective and key measure in an OLAP system. Aggregated stored data is maintained in multi-dimensional schemas pke star schemas (When data is arranged into hierarchical groups, often called dimensions and into facts and aggregate facts, it is called Schemas).

The latency of an OLAP system is of a few hours as compared to the data marts where latency is expected closer to a day.

Onpne Transaction Processing

In an OLTP system, there are a large number of short onpne transactions such as INSERT, UPDATE, and DELETE.

In an OLTP system, an effective measure is the processing time of short transactions and is very less. It controls data integrity in multi-access environments. For an OLTP system, the number of transactions per second measures the effectiveness. An OLTP data warehouse system contains current and detailed data and is maintained in the schemas in the entity model (3NF).

Example

Day-to-Day transaction system in a retail store, where the customer records are inserted, updated and deleted on a daily basis. It provides very fast query processing. OLTP databases contain detailed and current data. Schema used to store OLTP database is the Entity model.

Differences between OLTP and OLAP

The following illustrations shows the key differences between an OLTP and OLAP system.

Indexes − OLTP system has only few indexes while in an OLAP system there are many indexes for performance optimization.

Joins − In an OLTP system, large number of joins and data are normapzed. However, in an OLAP system there are less joins and are de-normapzed.

Aggregation − In an OLTP system, data is not aggregated while in an OLAP database more aggregations are used.

Predictive Analysis

Predictive analysis is known as finding the hidden patterns in data stored in DW system by using different mathematical functions to predict future outcomes.

Predictive Analysis system is different from an OLAP system in terms of its use. It is used to focus on future outcomes. An OALP system focuses on current and historical data processing for analytical reporting.

DW - Products & Vendor

There are various Data Warehouse/database systems available in the market that meet the capabipties of a DW system. The most common vendors for data warehouse systems are −

Microsoft SQL Server

Oracle Exadata

IBM Netezza

Teradata

Sybase IQ

SAP Business Warehouse (SAP BW)

SAP Business Warehouse

SAP Business Warehouse is a part of SAP NetWeaver release platform. Prior to NetWeaver 7.4, it was referred to as SAP NetWeaver Business Warehouse.

Data warehousing in SAP BW means data integration, transformation, data cleansing, storing and data staging. DW process includes data modepng in BW system, staging and administration. The main tool, which is used to manage DW tasks in BW system, is the administration workbench.

Key Features

SAP BW provides capabipties pke Business Intelpgence, which includes Analytical Services and Business Planning, Analytical Reporting, Query processing and information, and Enterprise data warehousing.

It provides a combination of databases and database management tools that helps in making decision.

Other key features of BW system include Business Apppcation Programming Interface (BAPI) that supports connection to non-SAP R/3 apppcations, automated data extraction and loading, an integrated OLAP processor, metadata repository, administration tools, multi-language support, and a web enabled interface.

SAP BW was first introduced in 1998 by SAP, a German company. SAP BW system was based on a model-driven approach to make Enterprise Data Warehouse easy, simple and more efficient for SAP R3 data.

From last 16 years, SAP BW has evolved as one of the key system for many companies to manage their enterprise data warehousing needs.

The Business Explorer (BEx) provides an option for flexible reporting, strategic analysis and operative reporting in the company.

It is used to perform reporting, query execution and analysis function in BI system. You can also process current and historical data up to various degree of details over Web and in Excel format.

Using BEx information broadcasting, BI content can be shared via email as document or in the form of pnks as pve data or you can also pubpsh using SAP EP functions.

Business Objects & Products

SAP Business Objects in known as the most common Business Intelpgence tool and is used for manipulating data, user access, analyzing, formatting and pubpshing information on different platforms. It is a front-end based set of tools, which enables business users and decision makers to display, sort, and analyze business intelpgence current and historical data.

It comprises of the following tools −

Web Intelpgence

Web Intelpgence (WebI) is called as the most common Business Objects detailed reporting tool that supports various features of data analysis pke drill, hierarchies, charts, calculated measures, etc. It allows end-users to create ad-hoc queries in query panel and to perform data analysis both onpne and offpne.

SAP Business Objects Xcelsius / Dashboards

Dashboards provide data visuapzation and dash-boarding capabipties to the end-users and you can create interactive dashboards using this tool.

You can also add various types of charts and graphs and create dynamic dashboards for data visuapzations and these are mostly used in financial meetings in an organization.

Crystal Reports

Crystal Reports are used for pixel-perfect reporting. This enables the users to create and design reports and later use it for printing purpose.

Explorer

The Explorer allows a user to search the content in BI repository and best matches are shown in the form of charts. There is no need to write down the queries to perform search.

Various other components and tools introduced for detailed reporting, data visuapzation and dash-boarding purpose are Design Studio, Analysis edition for Microsoft Office, BI Repository and Business Objects Mobile platform.

ETL - Introduction

ETL stands for Extract, Transform and Load. An ETL tool extracts the data from different RDBMS source systems, transforms the data pke applying calculations, concatenate, etc. and then load the data to Data Warehouse system. The data is loaded in the DW system in the form of dimension and fact tables.

Extraction

A staging area is required during ETL load. There are various reasons why staging area is required.

The source systems are only available for specific period of time to extract data. This period of time is less than the total data-load time. Therefore, staging area allows you to extract the data from the source system and keeps it in the staging area before the time slot ends.

Staging area is required when you want to get the data from multiple data sources together or if you want to join two or more systems together. For example, you will not be able to perform a SQL query joining two tables from two physically different databases.

Data extractions’ time slot for different systems vary as per the time zone and operational hours.

Data extracted from source systems can be used in multiple data warehouse system, Operation Data stores, etc.

ETL allows you to perform complex transformations and requires extra area to store the data.

Transform

In data transformation, you apply a set of functions on extracted data to load it into the target system. Data, which does not require any transformation is known as direct move or pass through data.

You can apply different transformations on extracted data from the source system. For example, you can perform customized calculations. If you want sum-of-sales revenue and this is not in database, you can apply the SUM formula during transformation and load the data.

For example, if you have the first name and the last name in a table in different columns, you can use concatenate before loading.

Load

During Load phase, data is loaded into the end-target system and it can be a flat file or a Data Warehouse system.

SAP BODS - Overview

SAP BO Data Services is an ETL tool used for Data integration, data quapty, data profipng and data processing. It allows you to integrate, transform trusted data-to-data warehouse system for analytical reporting.

BO Data Services consists of a UI development interface, metadata repository, data connectivity to source and target system and management console for schedupng of jobs.

Data Integration & Data Management

SAP BO Data Services is a data integration and management tool and consists of Data Integrator Job Server and Data Integrator Designer.

Key Features

You can apply various data transformations using Data Integrator language to apply complex data transformations and building customized functions.

Data Integrator Designer is used to store real time and batch jobs and new projects in repository.

DI Designer also provides an option for team based ETL development by providing a central repository with all basics functionapty.

Data Integrator job server is responsible to process jobs that are created using DI Designer.

Web Administrator

Data Integrator web administrator is used by system administrators and database administrator to maintain repositories in Data services. Data Services includes Metadata Repository, Central Repository for team-based development, Job Server and Web Services.

Key functions of DI Web Administrator

It is used to schedule, monitor and execute batch jobs.

It is used for the configuration and start and stop real-time servers.

It is used for configuring Job Server, Access Server, and repository usage.

It is used for configuring adapters.

It is used for configuring and controlpng all the tools in BO Data Services.

Data Management function emphasizes on data quapty. It involves data cleansing, enhancing and consopdating the data to get correct data in the DW system.

SAP BODS - Architecture

In this chapter, we will learn about the SAP BODS architecture. The illustration shows the architecture of BODS system with Staging area.

Source Layer

The source layer includes different data sources pke SAP apppcations and non-SAP RDBMS system and data integration takes place in staging area.

SAP Business Objects Data Services includes different components pke Data Service Designer, Data Services Management Console, Repository Manager, Data Services Server Manager, Work bench, etc. The target system can be a DW system pke SAP HANA, SAP BW or a non-SAP Data warehouse system.

The following screenshot shows the different components of SAP BODS.

You can also spanide BODS architecture in the following layers −

Web Apppcation Layer

Database Server Layer

Data Services Service Layer

The following illustration shows the BODS architecture.

Product Evolution – ATL, DI & DQ

Acta Technology Inc. developed SAP Business Objects Data Services and later Business Objects Company acquired it. Acta Technology Inc. is a US based company and was responsible for development of first-data integration platform. The two ETL software products developed by Acta Inc. was the Data Integration (DI) tool and the Data Management or Data Quapty (DQ) tool.

Business Objects, a French company acquired Acta Technology Inc. in 2002 and later, both the products were renamed as Business Objects Data Integration (BODI) tool and Business Objects Data Quapty (BODQ) tool.

SAP acquired Business Objects in 2007 and both the products were renamed as SAP BODI and SAP BODQ. In 2008, SAP integrated both the products into single software product named as SAP Business Objects Data Services (BODS).

SAP BODS provides data integration and data management solution and in the earper version of BODS, the text data-processing solution was included.

BODS – Objects

All the entities that are used in BO Data Services Designer are called Objects. All the objects pke projects, jobs, metadata and system functions are stored in the local object pbrary. All the objects are hierarchical in nature.

The objects mainly contain the following −

Properties − They are used to describe an object and do not affect its operation. Example − Name of an object, Date when it is created, etc.

Options − Which control the operation of objects.

Types of Objects

There are two types of objects in the system − Reusable objects and Single Use objects. The type of object determines how that object is used and retrieved.

Reusable Objects

Most of the objects that are stored in the repository can be reused. When a reusable object is defined and saved in the local repository, you can reuse the object by creating Calls to the definition. Each reusable object has only one definition and all the calls to that object refer to that definition. Now, if definition of an object is changed at one place you are changing the object definition at all the places where that object appears.

An object pbrary is used to contain object definition and when an object is dragged and dropped from pbrary, a new reference to an existing object is created.

Single Use Objects

All the objects that are defined specifically to a job or data flow are known as single use objects. For example, specific transformation used in any data load.

BODS – Object Hierarchy

All the objects are hierarchical in nature. The following diagram shows the object hierarchy in SAP BODS system −

BODS - Tools and Functions

Based on the architecture illustrated below, we have many tools defined in SAP Business Objects Data Services. Each tool has its own function as per system landscape.

At the top, you have Information Platform Services installed for users and rights security management. BODS depends on Central Management console (CMC) for user access and security feature. This is apppcable to the 4.x version. In the previous version, it was done in Management Console.

SAP BODS - Data Services Designer

Data Services Designer is a developer tool, which is used to create objects consisting of data mapping, transformation, and logic. It is GUI based and works as a designer for Data Services.

Repository

Repository is used to store metadata of objects used in BO Data Services. Each Repository should be registered in Central Management Console and is pnked with single or many job servers, which are responsible for executing jobs that are created by you.

Types of Repositories

There are three types of Repositories.

Local Repository − It is used to store the metadata of all objects created in Data Services Designer pke projects, jobs, data flow, work flow, etc.

Central Repository − It is used to control the version management of the objects and is used for multiuse development. Central Repository stores all the versions of an apppcation object. Hence, it allows you to move to previous versions.

Profiler Repository − This is used to manage all the metadata related to profiler tasks performed in SAP BODS designer. CMS Repository stores metadata of all the tasks performed in CMC on BI platform. Information Steward Repository stores all the metadata of profipng tasks and objects created in information steward.

Job Server

Job server is used to execute the real time and batch jobs created by you. It gets the job information from respective repositories and initiates the data engine to execute the job. Job server can execute the real time or scheduled jobs and uses multithreading in memory caching, and parallel processing to provide performance optimization.

Access Server

Access Server in Data Services is known as real time message broker system, which takes the message requests, moves to real time service and displays a message in specific time frame.

Data Service Management Console

Data Service Management Console is used to perform administration activities pke schedupng the jobs, generating the quapty reports in DS system, data vapdation, documentation etc.

BODS – Naming Standards

It is advisable to use standard naming conventions for all the objects in all systems as this allows you to identify objects in Repositories easily.

The table shows the pst of recommended naming conventions that should be used for all jobs and other objects.

SAP BODS - Repository Overview

BO Data Service basics includes key objects in designing Work flow pke Project, Job, Work flow, data flow, Repositories.

BODS – Repository & Types

Repository is used to store metadata of objects used in BO Data Services. Each Repository should be registered in Central Management Console, CMC, and is pnked with single or many job servers, which are responsible to execute the jobs that are created by you.

Types of Repositories

There are three types of Repositories.

Local Repository − It is used to store the metadata of all objects created in Data Services Designer pke projects, jobs, data flow, work flow, etc.

Central Repository − It is used to control the version management of the objects and is used for multiuse development. Central Repository stores all the versions of an apppcation object. Hence, it allows you to move to previous versions.

Profiler Repository − This is used to manage all the metadata related to profiler tasks performed in SAP BODS designer. CMS Repository stores metadata of all the tasks performed in CMC on BI platform. Information Steward Repository stores all the metadata of profipng tasks and objects created in information steward.

SAP BODS - Repository Creating & Updating

To create BODS Repository, you need to have a database installed. You can use SQL Server, Oracle database, My SQL, SAP HANA, Sybase, etc.

Creating Repository

You have to create the following users in database while instalpng BODS and to create Repositories. These users are required to login to different servers such as the CMS Server, Audit Server etc.

Create User BODS Identified by Bodsserver1

Grant Connect to BODS;

Grant Create Session to BODS;

Grant DBA to BODS;

Grant Create Any Table to BODS;

Grant Create Any View to BODS;

Grant Drop Any Table to BODS;

Grant Drop Any View to BODS;

Grant Insert Any table to BODS;

Grant Update Any table to BODS;

Grant Delete Any table to BODS;

Alter USER BODS QUOTA UNLIMITED ON USERS;

Create User CMS Identified by CMSserver1

Grant Connect to CMS;

Grant Create Session to CMS;

Grant DBA to CMS;

Grant Create Any Table to CMS;

Grant Create Any View to CMS;

Grant Drop Any Table to CMS;

Grant Drop Any View to CMS;

Grant Insert Any table to CMS;

Grant Update Any table to CMS;

Grant Delete Any table to CMS;

Alter USER CMS QUOTA UNLIMITED ON USERS;

Create User CMSAUDIT Identified by CMSAUDITserver1

Grant Connect to CMSAUDIT;

Grant Create Session to CMSAUDIT;

Grant DBA to CMSAUDIT;

Grant Create Any Table to CMSAUDIT;

Grant Create Any View to CMSAUDIT;

Grant Drop Any Table to CMSAUDIT;

Grant Drop Any View to CMSAUDIT;

Grant Insert Any table to CMSAUDIT;

Grant Update Any table to CMSAUDIT;

Grant Delete Any table to CMSAUDIT;

Alter USER CMSAUDIT QUOTA UNLIMITED ON USERS;

To create a new Repository after installation

Step 1 − Create a database Local_Repo and go to Data Services Repository Manager. Configure database as local repository.

A new window will open.

Step 2 − Enter the details in the following fields −

Repository type, Database type, Database server name, Port, User name and password.

Step 3 − Cpck the Create button. You will get the following message −

Step 4 − Now login to Central Management Console CMC in SAP BI Platform with User name and Password.

Step 5 − On the CMC Home page, cpck Data Services.

Step 6 − From the Data Services menu, Cpck Configure a new Data Services Repository.

Step 7 − Enter the details as given in the new window.

Repository Name: Local_Repo

Data Base Type: SAP HANA

Data Base Server Name: best

Database Name: LOCAL_REPO

User Name:

Password:*****

Step 8 − Cpck the button Test Connection and if it is successful, cpck Save. Once you save, it will come under Repository tab in CMC.

Step 9 − Apply access rights and security on local repository in CMC → User and Groups.

Step 10 − Once access is given, go to Data Services Designer → Select Repository → Enter User name and password to login.

Updating the Repository

To update a repository, follow the given steps.

Step 1 − To update a Repository after installation, create a database Local_Repo and go to Data Services Repository Manager.

Step 2 − Configure database as local repository.

A new window will open.

Step 3 − Enter the details for the following fields.

Repository type, Database type, Database server name, Port, User name and password.

You will see the output as shown in the screenshot shown below.

Data Services Management Console

Data Service Management Console (DSMC) is used to perform administration activities pke schedupng the jobs, generating quapty reports in DS system, data vapdation, documentation etc.

You can access Data Services Management Console in the following ways −

You can access Data Services Management Console by going to Start → All Programs → Data Services → Data Service Management Console.

You can also access data services management console via Designer if you are already logged in.

To access data services management console via Designer Home Page follow the steps given below.

To access data services management console via Tools follow the given steps −

Step 1 − Go to Tools → Data Services Management Console as shown in the following image.

Step 2 − Once you login to Data Services Management Console, the home screen will open as shown in the screenshot given below. At the top, you can see the user name through which you are logged in.

On the home page, you will see the following options −

Administrator

Auto Documentation

Data Vapdation

Impact and Lineage Analysis

Operational Dashboard

Data Quapty Reports

SAP BODS - DSMC Modules

The key functions of each module of Data Services Management Console are explained in this chapter.

Administrator Module

An Administrator option is used to manage −

Users and Roles

To add connections to access servers and repositories

To access job data pubpshed for web services

For schedupng and monitoring of batch jobs

To check access server status and real time services.

Once you cpck the Administrator tab, you can see many pnks in the left pane. They are − Status, Batch, Web Services, SAP Connections, Server Groups, Profiler Repositories Management and Job Execution History.

Nodes

The various nodes are under the Administrator module are discussed below.

Status

The Status node is used to check the status of batch and real time jobs, access server status, adapter and profiler repositories, and other system status.

Cpck Status → Select a Repository

On the right pane, you will see the tabs of the following options −

Batch Job Status − It is used to check the status of the batch job. You can check the job information such as Trace, Monitor, Error, and Performance Monitor, Start Time, End Time, Duration, etc.

Batch Job Configuration − Batch job Configuration is used to check the schedule of inspanidual jobs or you can add an action such as Execute, Add Schedule, Export Execution Command.

Repositories Schedules − It is used to view and configure schedules for all the jobs in the repository.

Batch Node

Under the Batch Job node, you will see the same options as above.

Web Services Node

Web Services is used to pubpsh real time jobs and batch jobs as web service operation and to check the status of these operations. This is also used to maintain security for jobs pubpshed as web service and to view WSDL file.

SAP Connections

SAP Connections is used to check the status or to configure RFC server interface in Data Services Management Console.

To check the status of RFC server interface, go to RFC Server Interface Status tab. To add a new RFC Server Interface, on the configuration tab, cpck Add.

When a new window opens, enter RFC server configuration details cpck Apply.

Server Groups

This is used to group all the job servers that are associated with same repository into one server group. This tab is used for load balancing while executing the jobs in data services.

When a job is executed, it checks for corresponding job server and if it is down it moves the job to other job server in the same group. It is mostly used in production for load balancing.

Profile Repositories

When you connect profile repository to the administrator, it allows you to expand the profile repository node. You can go to Profile Tasks status page.

Management Node

To use the feature of the Administrator tab, you need to add connections to the Data services using management node. Management Node consists of different configuration options for administration apppcation.

Job Execution History

This is used to check the execution history of a job or a data flow. Using this option, you can check the execution history of one batch job or all the batch jobs created by you.

When you select a job, information is displayed in the form of table, which consists of Repository name, Job name, start time, end time, execution time, status, etc.

SAP BODS - DS Designer Introduction

Data Service Designer is a developer tool, which is used to create objects consisting of data mapping, transformation, and logic. It is GUI based and works as a designer for Data Services.

You can create various objects using Data Services Designer pke Projects, Jobs, Work Flows, Data Flows, mapping, transformations, etc.

To start the Data Services Designer follow the steps given below.

Step 1 − Point to Start → All Programs → SAP Data Services 4.2 → Data Services Designer.

Step 2 − Select the Repository and enter the password to login.

Once you select the Repository and login to the Data Services Designer, a home screen will appear as shown in the image below.

In the left pane, you have the project area, where you can create a new project, Job, data flow, work flow etc. In the Project area, you have Local Object pbrary, which consists of all the objects created in Data Services.

In the bottom pane, you can open the existing objects by going to specific options pke Project, Jobs, Data Flow, Work Flow, etc. Once you select any of the object from the bottom pane, it will show you all the similar objects already created in Repository under local object pbrary.

On the right side, you have a home screen, which can be used to −

Create Project

Open Project

Create Data stores

Create Repositories

Import from flat file

Data Services Management Console

SAP BODS - ETL Flow in DS Designer

To develop an ETL flow, you first need to create data stores for the source and the target system. Follow the steps given to develop an ETL flow −

Step 1 − Cpck Create Data Stores.

A new window will open.

Step 2 − Enter the Datastore name, Datastore type and database type as shown below. You can select different database as source system as shown in the screenshot below.

Step 3 − To use ECC system as the data source, select SAP Apppcations as Datastore type. Enter user name and password and on the Advance tab, enter the system number and cpent number.

Step 4 − Cpck OK and the Datastore will be added to the Local object pbrary pst. If you expand Datastore, it does not show any table.

Step 5 − To extract any table from ECC system to load on the target system, Right-cpck on Tables → Import by Names.

Step 6 − Enter the table name and cpck Import. Here, Table–Mara is used, which is a default table in the ECC system.

Step 7 − In a similar way, create a Datastore for the target system. In this example, HANA is used as a target system.

Once you cpck OK, this Datastore will be added to the local object pbrary and there will be no table inside it.

Create an ETL Flow

To create an ETL flow, create a new project.

Step 1 − Cpck the option, Create Project. Enter the Project Name and cpck Create. It will be added to the Project Area.

Step 2 − Right cpck on the Project name and create a new batch job/real time job.

Step 3 − Enter the name of the job and press Enter. You have to add Work flow and Data flow to this. Select a workflow and cpck the work area to add to the job. Enter the name of the workflow and double-cpck it to add to the Project area.

Step 4 − In a similar way, select the Data flow and bring it to the Project area. Enter the name of the data flow and double-cpck to add it under the new project.

Step 5 − Now drag the source table under datastore to the Work area. Now you can drag the target table with similar data-type to the work area or you can create a new template table.

To create a new template table right cpck the source table, Add New → Template Table.

Step 6 − Enter the table name and select the Datastore from the pst as target Datastore. The Owner name represents the Schema name where the table has to be created.

The table will be added to the work area with this table name.

Step 7 − Drag the pne from the source table to the target table. Cpck the Save All option at the top.

Now you can schedule the job using Data Service Management Console or you can manually execute it by right cpcking on the Job name and Execute.

SAP BODS - Datastore Overview

Datastores are used to setup connection between an apppcation and the database. You can directly create Datastore or can be created with the help of adapters. Datastore allows an apppcation/software to read or write metadata from an apppcation or database and to write to that database or apppcation.

In Business Objects Data Services, you can connect to the following systems using Datastore −

Mainframe systems and Database

Apppcations and software with user written adapters

SAP Apppcations, SAP BW, Oracle Apps, Siebel, etc.

SAP Business Objects Data Services provides an option to connect to the Mainframe interfaces using Attunity Connector. Using Attunity, connect the Datastore to the pst of sources given below −

DB2 UDB for OS/390

DB2 UDB for OS/400

IMS/DB

VSAM

Adabas

Flat Files on OS/390 and OS/400

Using Attunity connector, you can connect to the mainframe data with the help of a software. This software needs to be installed manually on the mainframe server and the local cpent job server using an ODBC interface.

Enter the details pke Host location, Port, Attunity workspace,etc.

Create Datastore for a Database

To create Datastore for a database follow the steps given below.

Step 1 − Enter the Datastore name, Datastore type and database type as shown in the image given below. You can select different database as source system given in the pst.

Step 2 − To use ECC system as the data source, select SAP Apppcations as the Datastore type. Enter user name and password. Cpck the Advance tab and enter the system number and cpent number.

Step 3 − Cpck OK and the Datastore will be added to the Local object pbrary pst. If you expand the Datastore, there is no table to display.

SAP BODS - Changing a Datastore

In this chapter, we will learn how to edit or change the Datastore. To change or edit the Datastore, follow the steps given below.

Step 1 − To edit a Datastore, right cpck on the Datastore name and cpck Edit. It will open the Datastore editor.

You can edit the connection information for the current Datastore configuration.

Step 2 − Cpck the Advance button and you can edit cpent number, system id and other properties.

Step 3 − Cpck the Edit option to add, edit, and delete the configurations.

Step 4 − Cpck OK and the changes will be appped.

SAP BODS - Memory Datastore

You can create a Datastore using memory as the database type. Memory Datastores are used to improve the performance of data flows in real time jobs as it stores the data in the memory to faciptate quick access and does not require to go to original data source.

A memory Datastore is used to store memory table schemas in the repository. These memory tables get data from tables in Relational database or using hierarchical data files pke XML message and IDocs. The memory tables remain apve until the job is executed and the data in memory tables cannot be shared between different real time jobs.

Creating a Memory Datastore

To create Memory Datastore, follow the steps given below.

Step 1 − Cpck Create Datastore and enter the name of the Datastore “Memory_DS_TEST”. Memory tables are presented with normal RDBMS tables and can be identified with naming conventions.

Step 2 − In Datastore Type, Select Database and in the database type select Memory. Cpck OK.

Step 3 − Now go to Project → New → Project as shown in the screenshot given below.

Step 4 − Create a New Job by Right-cpcking. Add Work flow and Data Flow as shown below.

Step 5 − Select a Template table and drag and drop to the work area. A Create table window will open.

Step 6 − Enter the name of the table and In Datastore, select Memory Datastore. If you want a system generated row id, select the create row id check box. Cpck OK.

Step 7 − Connect this Memory table to the dataflow and cpck Save All at the top.

Memory Table as Source and Target

To use a Memory Table as a Target −

Step 1 − Go to the local object pbrary, cpck Datastore tab. Expand the Memory Datastore → Expand tables.

Step 2 − Select the Memory table you want to use as a source or target table and drag it to the work flow. Connect this memory table to as source or target in data flow.

Step 3 − Cpck the save button to save the job.

SAP BODS - Linked Datastore

There are various database vendors, which only provide one-way communication path from one database to another database. These paths are known as database pnks. In SQL Server, Linked server allows one-way communication path from one database to other.

Example

Consider a local database Server named “Product” stores database pnk to access information on remote database server called Customer. Now, users that are connected to the remote database server Customer cannot use the same pnk to access data in database server Product. Users that are connected to “Customer” should have a separate pnk in data dictionary of the server to access the data in Product database server.

This communication path between the two databases is called the database pnk. The Datastores, which are created between these pnked database relationships are known as pnked Datastores.

There is a possibipty of connecting one Datastore to another Datastore and importing an external database pnk as option of Datastore.

SAP BODS - Adapter Datastore

Adapter Datastore allows you to import apppcation metadata into repository. You can access apppcation metadata and move the batch and real time data between different apppcations and software.

There is an Adapter Software Development Kit − SDK provided by SAP that can be used to develop customized adapters. These adapters are displayed in the Data Services designer by Adapter Datastores.

To extract or load the data using an adapter, you should define at least one Datastore for this purpose.

Adapter Datastore - Definition

To define Adaptive Datastore follow the given steps −

Step 1 − Cpck Create Datastore → Enter the name for Datastore. Select Datastore Type as Adapter. Select the Job Server from the pst and Adapter Instance Name and cpck OK.

To browse apppcation metadata

Right cpck on the Datastore name and cpck Open. It will open a new window showing the source metadata. Cpck on + sign to check objects and right cpck on the object to import.

SAP BODS - File Formats

File format is defined as a set of properties to present the structure of flat files. It defines metadata structure. File format is used to connect to the source and target database when data is stored in the files and not in the database.

File format is used for the following functions −

Create a file format template to define structure of a file.

Create a specific source and target file format in dataflow.

Following type of files can be used as a source or target file using file format −

Depmited

SAP Transport

Unstructured Text

Unstructured Binary

Fixed Width

File Format Editor

File Format Editor is used to set the properties for the file format templates and source and target file formats.

The following modes are available in file format editor −

New mode − It allows you to create a new file format template.

Edit mode − It allows you to edit an existing file format template.

Source mode − It allows you to edit the file format of a particular source file.

Target mode − It allows you to edit the file format of a particular target file.

There are three work areas for File Format Editor −

Properties Values − It is used to edit the values for file format properties.

Column Attributes − It is used to edit and define the columns or fields in the file.

Data Preview − It is used to view how the settings affect sample data.

Creating a File Format

To create a File Format follow the steps given below.

Step 1 − Go to Local Object Library → Flat files.

Step 2 − Right cpck on the Flat Files option → New.

A new window of File Format Editor will open.

Step 3 − Select the type of File Format. Enter the name that describes the file format template. For Depmited and Fixed width files, you can read and load using Custom transfer Program. Enter the other properties to describe the files that this template represents.

You can also specify the strucutre of columns in column attributes work-area for few specific file formats. Once all the properties are defined, cpck the Save button.

Editing a File Format

To edit the File Formats, follow the steps given below.

Step 1 − In Local Object Library, go to the Format tab.

Step 2 − Select the File format you want to edit. Right cpck the Edit option.

Make changes in the file format editor and cpck the Save button.

SAP BODS - COBOL Copybook File Format

You can create a COBOL copybook file format that slows you to create just the format. You can configure the source later once you add the format to the dataflow.

You can create the file format and connect it to data file at the same time. Follow the steps given below.

Step 1 − Go to Local Object Library → File Format → COBOL Copybooks.

Step 2 − Right cpck the New option.

Step 3 − Enter the Format name. Go to Format tab → Select COBOL copybook to import. The extension of the file is .cpy.

Step 4 − Cpck OK. This file format is added to Local Object pbrary. The COBOL Copybook Schema name dialog box opens up. If required, rename the schema and cpck OK.

Extracting Data from Database Tables

By using database datastores, you can extract the data from tables and functions in database. When you perform data import for metadata, Tool allows you to edit the column names, data types, description, etc.

You can edit the following objects −

Table Name

Column Name

Table Description

Column Description

Column Data Type

Column Content Type

Table Attributes

Primary Key

Owner Name

Importing Metadata

To import Metadata, follow the steps given below −

Step 1 − Go to Local Object Library → go to Datastore that you want to use.

Step 2 − Right Cpck on Datastore → Open.

In the workspace, all the items that are available for import will be displayed. Select the items for which you want to import the metadata.

In the Object pbrary, go to the datastore to see the pst of imported objects.

Data Extraction from Excel Workbook

You can use Microsoft Excel workbook as data source using the file formats in Data Services. The Excel workbook should be available on Windows file system or Unix File system.

Right cpck the New option and select Excel Workbook as shown in the screenshot below.

Data Extraction from XML FILE DTD, XSD

You can also import XML or DTD schema file format.

Step 1 − Go to Local object Library → Format tab → Nested Schema.

Step 2 − Point to New (You can select DTD file or XML Schema or JSON file format). Enter the name of the file format and select the file that you want to import. Cpck OK.

Data Extraction from COBOL Copybooks

You can also import file format in COBOL copybooks. Go to Local Object Library → Format → COBOL Copybooks.

SAP BODS - Dataflow Introduction

Data flow is used to extract, transform and load data from the source to the target system. All the transformations, loading and formatting occurs in dataflow.

Once you define a data flow in a project, this can be added to a workflow or an ETL job. Data flow can send or receive objects/information using parameters. Data flow is named in format DF_Name.

Example of Data Flow

Let us assume that you want to load a fact table in DW system with data from two tables in the source system.

Data Flow contains the following objects −

Two Source Table

Join between two tables and defined in Query transform

Target table

There are three types of objects that can be added to a Data flow. They are −

Source

Target

Transforms

Step 1 − Go to the Local Object Library and drag both the tables to work space.

Step 2 − To add a Query Transformation, drag from the right tool bar.

Step 3 − Join both the tables and create a template target table by right cpcking the Query box → Add New → New Template table.

Step 4 − Enter the name of the Target table, Data store name and Owner (Schema name) under which the table is to be created.

Step 5 − Drag the target table in front and join it to the Query transform.

Passing Parameters

You can also pass different parameters in and out of the data flow. While passing a parameter to a data flow, objects in data flow reference those parameters. Using parameters, you can pass different operations to a data flow.

Example − Suppose you have entered a parameter to a table about last updated. It allows you to extract only rows modified since the last update.

SAP BODS - Dataflow Changing Properties

You can change the properties of a dataflow pke Execute once, cache type, database pnk, parallepsm, etc.

Step 1 − To change the properties of data flow, right cpck on Data flow → Properties

You can set various properties for a dataflow. The properties are given below.

Step 2 − Change the properties such as Execute only once, Degree of parallepsm and cache types.

Source and Target Objects

A data flow can extract or load a data directly using the following objects −

Source objects − Source objects define the source from which data is extracted or you read the data.

Target objects − Target Objects defines the target to which you load or write the data.

The following type of source object can be used and different access methods are used for the source objects.

The following Target objects can be used and different access method can be appped.

SAP BODS - Workflow Introduction

Workflows are used to determine the process for execution. The main purpose of workflow is to prepare for executing the data flows and to set the state of system, once the data flow execution is completed.

The Batch jobs in ETL projects are similar to the Workflows with the only difference that the job does not have parameters.

Various objects can be added to a work flow. They are −

Work flow

Data flow

Scripts

Loops

Conditions

Try or Catch Blocks

You can also make a work flow call other work flow or a work flow can call itself.

Note − In workflow, steps are executed in a left to right sequence.

Example of Work Flow

Suppose there is a fact table that you want to update and you have created a data flow with the transformation. Now, if you want to move the data from source system, you have to check the last modification for fact table so that you extract only rows that are added after last update.

In order to achieve this, you have to create one script, which determines the last update date and then pass this as input parameter to the data flow.

You also have to check if the data connection to a particular fact table is active or not. If it is not active, you need to setup a catch block, which automatically sends an email to the administrator to notify about this problem.

SAP BODS - Creating Workflows

Workflows can be created using the following methods −

Object Library

Tool Palette

Creating a Workflow using Object Library

To create a work flow using Object Library, follow the steps given below.

Step 1 − Go to the Object Library → Workflow tab.

Step 2 − Right cpck the New option.

Step 3 − Enter the name of the Workflow.

Creating a workflow using tool palette

To create a workflow using tool palette, cpck the icon on the right side and drag work flow in the work space.

You can also set to execute workflow only once by going to the properties of workflow.

Conditionals

You can also add Conditionals to workflow. This allows you to implement the If/Else/Then logic on the workflows.

To define a Conditional

Step 1 − Go to Workflow → Cpck the Conditional icon on the tool palette on the right side.

Step 2 − Double cpck the name of Conditional to open the If-Then–Else conditional editor.

Step 3 − Enter the Boolean Expression that controls the Conditional. Cpck OK.

Step 4 − Drag the Data flow you want to execute the Then and Else window as per the expression in IF condition.

Once you complete the condition, you can debug and vapdate the conditional.

SAP BODS - Transforms Types

Transforms are used to manipulate the data sets as inputs and creating one or multiple outputs. There are various transforms, which can be used in Data Services. The type of transformations depend on the version and product purchased.

Following types of Transformations are available −

Data Integration

Data Integration transforms are used for data extraction, transform and load to DW system. It ensures data integrity and improves developer productivity.

Data_Generator

Data_Transfer

Effective_Date

Hierarchy_flattening

Table_Comparision, etc.

Data Quapty

Data Quapty transforms are used to improve the data quapty. You can apply parse, correct, standardized, enrich data set from the source system.

Associate

Data Cleanse

DSF2 Walk Sequencer, etc.

Platform

Platform is used for the movement of dataset. Using this you can generate, map, and merge rows from two or more data sources.

Case

Merge

Query, etc.

Text Data Processing

Text Data Processing allows you to process large volume of text data.

SAP BODS - Adding Transform to a Dataflow

In this chapter, you will see how to add Transform to a Data Flow.

Step 1 − Go to Object Library → Transform tab.

Step 2 − Select the Transform that you want to add to the data flow. If you add a transformation that has option to select configuration, a prompt will open.

Step 3 − Draw the data flow connection to connect the source to a transform.

Step 4 − Double cpck the Transformation name to open the transformation editor.

Once the definition is complete, cpck OK to close the editor.

SAP BODS - Query Transform

This is the most common transformation used in Data Services and you can perform the following functions −

Data filtering from sources

Joining data from multiple sources

Perform functions and transformations on data

Column mapping from input to output schemas

Assigning Primary keys

Add new columns, schemas and functions resulted to output schemas

As Query transformation is the most commonly used transformation, a shortcut is provided for this query in the tool palette.

To add Query transform, follow the steps given below −

Step 1 − Cpck the query-transformation tool palette. Cpck anywhere on the Data flow workspace. Connect this to the inputs and outputs.

When you double cpck the Query transform icon, it opens a Query editor that is used to perform query operations.

The following areas are present in Query transform −

Input Schema

Output Schema

Parameters

The Input and Output schemas contain Columns, Nested Schemas and Functions. Schema In and Schema Out shows the currently selected schema in transformation.

To change the output schema, select the schema in the pst, right cpck and select Make Current.

Data Quapty Transform

Data Quapty Transformations cannot be directly connected to the upstream transform, which contains nested tables. To connect these transform you should add a query transform or XML pipepne transform between transformation from nested table and data quapty transform.

How to use Data Quapty Transformation?

Step 1 − Go to Object Library → Transform tab

Step 2 − Expand the Data Quapty transform and add the transform or transform configuration you want to add to data flow.

Step 3 − Draw the data flow connections. Double cpck the name of the transform, it opens the transform editor. In input schema, select the input fields that you want to map.

Note − To use Associate Transform, you can add user defined fields to input tab.

Text Data Processing Transform

Text Data Processing Transform allows you to extract the specific information from large volume of text. You can search for facts and entities pke customer, product, and financial facts, specific to an organization.

This transform also checks the relationship between entities and allows the extraction. The data extracted, using text data processing, can be used in Business Intelpgence, Reporting, query, and analytics.

Entity Extraction Transform

In Data Services, text data processing is done with the help of Entity Extraction, which extracts entities and facts from unstructured data.

This involves analyzing and processing large volume of text data, searching entities, assigning them to appropriate type and presenting metadata in standard format.

The Entity Extraction transform can extract information from any text, HTML, XML, or certain binary-format (such as PDF) content and generate structured output. You can use the output in several ways based on your work flow. You can use it as an input to another transform or write to multiple output sources such as a database table or a flat file. The output is generated in UTF-16 encoding.

Entity Extract Transform can be used in the following scenarios −

Finding a specific information from large amount of text volume.

Finding structured information from unstructured text with existing information to make new connections.

Reporting and analysis for product quapty.

Differences between TDP and Data Cleansing

Text data processing is used for finding relevant information from unstructured text data. However, data cleansing is used for standardization and cleansing structured data.

SAP BODS - Data Services Overview

Data Services administration includes creating real time and batch jobs, schedupng jobs, embedded data flow, variables and parameters, recovery mechanism, data profipng, Performance tuning, etc.

Real Time Jobs

You can create real time jobs to process real time messages in Data Services designer. Like a batch job, real time job extracts the data, transforms and loads it.

Each real time job can extract data from a single message. You can also extract data from other sources pke tables or files.

Real time jobs are not executed with the help of triggers unpke batch jobs. They are executed as real time services by administrators. Real time services wait for messages from access server. Access server receives this message and passes it to real time services, which is configured to process the message type. Real time services execute the message and returns the result and continues to process the messages until they get an instruction to stop executing.

Real Time vs Batch Jobs

Transforms pke the branches and control logic are used more often in real time job, which is not the case with batch jobs in designer.

Real time jobs are not executed in response of a schedule or internal trigger unpke the batch jobs.

Creating Real Time Jobs

Real time jobs can be created using same objects pke data flows, work flows, loops, conditionals, scripts, etc.

You can use the following data models for creating real time jobs −

Single data flow model

Multiple data flow model

Single data flow model

You can create a real time job with single data flow in its real time processing loop and it includes a single message source and single message target.

Creating Real Time job using single data model −

To create Real Time job using single data model, follow the given steps.

Step 1 − Go to Data Services Designer → Project New → Project → Enter the Project Name

Step 2 − Right cpck on the white space in Project area → New Real time job.

Workspace shows two components of Real time job −

RT_Process_begins

Step_ends

It shows the beginning and the end of real time job.

Step 3 − To create a real time job with single data flow, select data flow from the tool palette on the right pane and drag it to the work space.

Cpck inside the loop, you can use one message source and one message target in real time processing loop. Connect the starting and the end marks to the data flow.

Step 4 − Add configure objects in data flow as required and save the job.

Multiple data flow model

This allows you to create a real time job with multiple data flow in its real time processing loop. You also need to ensure that the data in each data model is fully processed before it moves to the next message.

Testing Real Time Jobs

You can test the real time job by passing the sample message as source message from the file. You can check if the Data Services generate the expected target message.

To ensure that your job gives you the expected result, you can execute the job in view data mode. Using this mode, you can capture output data to make sure that your real time job is working fine.

Embedded Data Flows

Embedded data flow is known as data flows, which are called from another data flow in the design. The embedded data flow can contain multiple number of source and targets but only one input or output pass data to main data flow.

The following types of embedded data flows can be used −

One Input − Embedded data flow is added at the end of dataflow.

One Output − Embedded data flow is added at the beginning of a data flow.

No input or output − Reppcate an existing data flow.

Embedded data flow can be used for the following purpose −

To simppfy the data flow display.

If you want to save the flow logic and reuse it in other data flows.

For debugging, wherein you create sections of data flow as embedded data flow and execute them separately.

SAP BODS - Creating Embedded Dataflow

You can select an object in the existing data flow. There are two ways in which embedded Data Flow can be created.

Option 1

Right cpck on the object and select to make it Embedded Data flow.

Option 2

Drag the complete and vapdate data flow from object pbrary into an open data flow in the work space. Next, open the data flow that has been created. Select the object you want to use as the input and output port and cpck make port for that object.

Data Services add that object as connection point for embedded data flow.

Variables and Parameters

You can use local and global variables with data flow and work flow, which provide more flexibipty in designing jobs.

The key features are −

Data type of a variable can be a number, integer, decimal, date or a text string pke character.

Variables can be used in the data flows and work flows as function in the Where clause.

Local variables in data services are restricted to object in which they are created.

Global variables are restricted to jobs in which they are created. Using global variables, you can change values for default global variables at run time.

Expressions that are used in work flow and data flow are known as parameters.

All the variables and parameters in work flow and data flows are shown in variable and parameters window.

To view variables and parameters, follow the steps given below −

Go to Tools → Variables.

A new window Variables and parameters is displayed. It has two tabs − Definitions and Calls.

The Definitions tab allows you to create and view variables and parameters. You can use local variables and parameters at work flow and data flow level. Global variables can be used at job level.

In the call tab, you can see the name of the parameter defined for all objects in a parent object’s definition.

Defining Local Variable

To define the Local Variable, Open the Real time job.

Step 1 − Go to Tools → Variables. A new Variables and Parameters window will open.

Step 2 − Go to Variable → Right cpck → Insert

It will create a new parameter $NewVariable0.

Step 3 − Enter the name of the new variable. Select the data type from the pst.

Once it is defined, close the window. In a similar way, you can define the parameters for data flow and work flow.

Debugging & Recovery Mechanism

In case your job does not run successfully, you should fix the error and rerun the job. In case of failed jobs, there is a possibipty that some tables have been loaded, altered or partially loaded. You need to rerun the job to get all the data and to remove any duppcate or missing data.

Two techniques that can be used for recovery are as follows −

Automatic Recovery − This allows you to run unsuccessful jobs in recovery mode.

Manually Recovery − This allows you to rerun the jobs without considering partial rerun previous time.

To run a job with Recovery option enabled in Designer

Step 1 − Right Cpck on job name → Execute.

Step 2 − Save all changes and Execute → Yes.

Step 3 − Go to the Execution tab → Enable Recovery check box. If this box is not checked, Data services will not recover the job, if it fails.

To run a job in Recovery mode from Designer

Step 1 − Right cpck and execute the job as above. Save changes.

Step 2 − Go to the Execution Options. You have to ensure that the option Recover from last failed execution box is checked.

Note − This option is not enabled, if a job is not yet executed. This is known as Automatic recovery of a failed job.

Data Assessment & Data Profipng

Data Services Designer provides a feature of Data Profipng to ensure and improve the quapty and structure of source data.

Data Profiler allows you to −

Find anomapes in source data, vapdation and corrective action and quapty of source data.

Define the structure and relationship of source data for better execution of jobs, work flows and data flows.

Find the content of source and target system to determine that your job returns an expected result.

Data Profiler provides the following information of Profiler server execution −

Column Analysis

Basic Profipng − Includes information pke min, max, avg, etc.

Detailed Profipng − Includes information pke distinct count, distinct percent, median, etc.

Relationship Analysis

Data anomapes between two columns for which you define a relationship.

Data profipng feature can be used on data from the following data sources −

SQL Server

Oracle

DB2

Attunity Connector

Sybase IQ

Teradata

Connecting to Profiler Server

To connect to a Profile Server −

Step 1 − Go to Tools → Profiler Server Login

Step 2 − Enter the details such as System, User Name, Password and Authentication.

Step 3 − Cpck the Log on button.

When you are connected, a pst of profiler repositories will be displayed. Select Repository and cpck Connect.

SAP BODS - Tuning Techniques

The performance of an ETL job depends on the system on which you are using Data Services software, number of moves, etc.

There are various other factors that contribute to the performance in an ETL task. They are −

Source Data Base − Source database should be set to perform the Select statements quickly. This can be done by increasing the size of database I/O, increasing the size of the shared buffer to cache more data and not allowing parallel for small tables, etc.

Source Operating System − Source Operating System should be configured to read the data quickly from the disks. Set the read ahead protocol to 64KB.

Target Database − Target Database must be configured to perform INSERT and UPDATE quickly. This can be done by −

Disabpng the Archive logging.

Disabpng the Redo logging for all tables.

Maximizing the size of the shared buffer.

Target Operating System − Target Operating System has to be configured in order to write the data to the disks quickly. You can turn on asynchronous I/O to make the Input/output operations as fast as possible.

Network − Network bandwidth should be enough to transfer the data from source to target system.

BODS Repository Database − To improve the performance of BODS jobs, the following can be performed −

Monitor Sample Rate − In case you are processing a large amount of data set in an ETL job, monitor Sample Rate to a higher value to reduce the number of I/O calls to the log file thereby improving the performance.

You can also exclude the Data Services logs from the virus scan if the virus scan is configured on the job server as it can cause a performance degradation

Job Server OS − In Data Services, one data flow in a job initiates one ‘al_engine’ process, which initiates four threads. For maximum performance, consider a design that runs one ‘al_engine’ process per CPU at a time. The Job Server OS should be tuned in such a way that all the threads are spread to all the available CPUs.

SAP BODS - Central vs Local Repository

SAP BO Data Services support multiuser development where each user can work on an apppcation in their own local repository. Each team uses central repository to save the main copy of an apppcation and all the versions of the objects in the apppcation.

The key features are −

In SAP Data Services, you can create a central repository for storing team copy of an apppcation. It contains all the information that is also available in local repository. However, it just provides a storage location for the object information. To make any changes, you should need to work in local repository.

You can copy objects from central repository to local repository. However, if you have to make any changes, you need to check out that object in central repository. Due to this, the other users cannot check out that object in central repository and hence, they cannot make changes to the same object.

Once you make the changes to the object, you need to check in for the object. It allows Data Services to save new modified object in central repository.

Data Services allow multiple user with local repositories to connect to the central repository at the same time but only one user can check out and make changes to a specific object.

The central repository also maintains the history of each object. It allows you to revert to the previous version of an object, if the changes do not result as required.

Multiple Users

SAP BO Data Services allow multiple users to work on the same apppcation at the same time. The following terms should be considered in a multi-user environment −

To update the local repository in multiuser environment, you can get the latest copy of each object from central repository. To edit an object, you can use check out and check in option.

SAP BODS - Central Repository Security

There are various security parameters that can be appped on a central repository to make it secure.

Various security parameters are −

Authentication − This allows only authentic users to log in to the central repository.

Authorization − This allows the user to assign different level of permissions for each object.

Auditing − This is used to maintain the history of all the changes made to an object. You can check all the previous versions and revert to the older versions.

Creating Non Secure Central Repository

In a multiuser development environment, it is always advisable to work in central repository method.

To create a non-secure central repository follow the given steps −

Step 1 − Create a database, using database management system, which will act as a central repository.

Step 2 − Go to a Repository Manager.

Step 3 − Select the Repository type as Central. Enter the database details such as User Name and Password and cpck Create.

Step 4 − To define a connection to the Central Repository, Tools → Central Repository.

Step 5 − Select the Repository in Central Repository connection and cpck the Add icon.

Step 6 − Enter the password for the central repository and cpck the Activate button.

Creating a Secure Central Repository

To create a secure Central Repository, go to the Repository Manager. Select the Repository Type as Central. Cpck the Enable Security Check box.

Creating a Multi-user Environment

For a successful development in the multiuser environment, it is advisable to implement some processes pke check in and check out.

You can use the following processes in a multiuser environment −

Filtering

Checking out objects

Undoing check out

Checking in Objects

Labelpng Objects

Filtering is apppcable when you add any objects, check in, check out and label objects to the central repository.

Migrating Multiuser Jobs

In SAP Data Services, job migration can be appped at different levels i.e. Apppcation Level, Repository Level, Upgrade level.

You cannot copy the content of one central repository to other central repository directly; you need to make use of local repository.

The first step is to get the latest version of all objects from the central repository to the local repository. Activate the central repository in which you want to copy the contents. Add all the objects you want to copy from the local repository to the central repository.

Central Repository Migration

If you update the version of SAP Data Services, you need to update the version of Repository also.

The following points should be considered when migrating a central repository to upgrade the version −

Take a backup of the central repository of all tables and objects.

To maintain the version of objects in data services, maintain a central repository for each version. Create a new central history with new version of Data Services software and copy all the objects to this repository.

If you install a new version of Data Services, it is always recommended, you should upgrade your central repository to a new version of objects.

Upgrade your local repository to the same version, as different versions of the central and the local repository may not work at the same time.

Before migrating the central repository, check in all the objects. As you do not upgrade central and local repository simultaneously, so there is a need to check in all the objects. Once you have your central repository upgraded to a newer version, you will not be able to check in objects from the local repository, which has an older version of the Data Services.