Flat File Upload – Considerations -

Preliminary Program

Applies to:
SAP Business Intelligence 7.0. For more information, visit the Business Intelligence homepage.

Summary
The advantage of uploading flat files is the flexibility. This means that all kinds of sources of data can be
used to provide data for the SAP BI application. Flat files have also some particularities. Amongst them are
easy changes in the file structure, redundant data, lines without data and information in the filename. To
handle these particularities some considerations can be useful.

Author: Guy Mathys

Company: Aureus Financials
Created on: 15 August 2009

Author Bio
Guy Mathys enables companies to obtain information from SAP Business Intelligence (SAP
NetWeaver BW - SAP BI) for the last five years.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 1
 Flat File Upload – Considerations - Preliminary Program

Table of Contents
Reason of a preliminary program ....................................................................................................................... 3 
Functionalities ..................................................................................................................................................... 3 
Details of the functionalities ................................................................................................................................ 4 
Get a file. ......................................................................................................................................................... 4 
Read data record ............................................................................................................................................ 5 
Define delimiter ............................................................................................................................................... 5 
Check on allowed characters .......................................................................................................................... 5 
Check on records with all fields empty............................................................................................................ 5 
Check on comma as field separator and as decimal. ..................................................................................... 6 
Read data of the record into a table................................................................................................................ 6 
Check field types ............................................................................................................................................. 7 
Include filename in a field ............................................................................................................................... 7 
Write the record............................................................................................................................................... 8 
Other considerations........................................................................................................................................... 8 
Related Content .................................................................................................................................................. 9 
Help.sap.com .................................................................................................................................................. 9 
Articles............................................................................................................................................................. 9 
Blogs ............................................................................................................................................................... 9 
Disclaimer and Liability Notice .......................................................................................................................... 10 

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 2
 Flat File Upload – Considerations - Preliminary Program

Reason of a preliminary program

A program before starting the loads prevents some errors during the loading process. This is important
because an error can stop the activation of the loads of succeeding files in the same Data Store Object. A
check program can also generate alerts.

Functionalities
A preliminary can handle several functionalities. The first group is getting the data. Then there is a checking
and converting the characters, fields and their separators and checking and skipping records. A whole file
can also be rejected. The last group is writing an improved file ready for upload. This all can be done
generating error messages.
1. Get a file.
This is to get a file based on the directory WA_DRCTR and file mask WA_FLMSK.
2. Read data record
Opens the dataset reads a record, closes the dataset .
3. define delimiter
The delimiter can be used to add fields later.
4. Check on allowed characters
Put the allowed characters in a work area. Replaces the not allowed character by a 0.
5. Check on records with all fields empty
Checks if the first 5 characters contains not a delimiter. If it does not contain delimiters it continues.
6. Check on comma as field separator and as decimal.
Some csv files have comma as a field separator. on top of that the comma is also used a the decimal sign.
For instance some records are like
Field1,”123,45”,Field3
This removes the inverted comma and changes the decimal comma in a decimal point.
7. Read data of the record into a table
This is useful to manipulate data in a table. This can be the check of field types.
8. Check field types
This checks the field type date , numeric, character of a field based on parameters in an internal table.
9. Include filename in a field
This enables to load the filename into a DSO and use it or part of it; For instance to exploit the date
timestamp that is in it.
10. Write the record
This writes a check record in the new file. This new file will be used to load.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 3
 Flat File Upload – Considerations - Preliminary Program

Details of the functionalities

The basic idea is that the preliminary program takes the input file, checks and modifies it and writes it to
another file. This other is used to load in SAP BI.
These parts of ABAP are not real code. They are meant to explain the functionalities.

Get a file.
This is to get a file based on the directory WA_DRCTR and file mask WA_FLMSK.
* The first file that matches the pattern

CALL 'C_SAPGPARAM'
ID 'NAME' FIELD WA_DRCTR
ID 'VALUE' FIELD WA_DIR_VALUE.
CALL 'C_DIR_READ_FINISH' " just to be sure
ID 'ERRNO' FIELD WA_FILE_ERRNO
ID 'ERRMSG' FIELD WA_FILE_ERRMSG.
CALL 'C_DIR_READ_START'
ID 'DIR' FIELD WA_DRCTR
ID 'FILE' FIELD WA_FLMSK
ID 'ERRNO' FIELD WA_FILE_ERRNO
ID 'ERRMSG' FIELD WA_FILE_ERRMSG.
IF SY-SUBRC <> 0.
ENDIF.
DO.
CALL 'C_DIR_READ_NEXT'
ID 'TYPE' FIELD WA_FILE_TYPE
ID 'NAME' FIELD WA_FILENAME
ID 'LEN' FIELD WA_FILE_LEN
ID 'OWNER' FIELD WA_FILE_OWNER
ID 'MTIME' FIELD WA_FILE_MTIME
ID 'MODE' FIELD WA_FILE_MODE
ID 'ERRNO' FIELD WA_FILE_ERRNO
ID 'ERRMSG' FIELD WA_FILE_ERRMSG.
IF SY-SUBRC = 0.
IF WA_FILE_TYPE(1) = 'f' OR " regular file
WA_FILE_TYPE(1) = 'F'.
EXIT.
ENDIF.
ELSEIF SY-SUBRC = 1.
RAISE FILE_NOT_FOUND.
EXIT.
ELSE.
ENDIF.
ENDDO.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 4
 Flat File Upload – Considerations - Preliminary Program

Read data record

Opens the dataset , reads a record, closes the dataset .
OPEN DATASET WA_DIRFILENAME FOR INPUT IN TEXT MODE ENCODING DEFAULT.
DO.

CATCH SYSTEM-EXCEPTIONS CONVT_CODEPAGE = 1.

READ DATASET WA_DIRFILENAME INTO WA_LINE.
IF SY-SUBRC <> 0. EXIT. ENDIF.
ENDCATCH.
…
ENDDO
…

CLOSE DATASET WA_DIRFILENAME.

Define delimiter
The delimiter can be used to add fields later.
IF WA_SPRTR IS INITIAL.
IF WA_LINE CA C_SMCLMN.
WA_SPRTR = C_SMCLMN.
ELSE.
WA_SPRTR = C_CMM.
ENDIF.
ENDIF.

Check on allowed characters

Put the allowed characters in a work area. Replaces the not allowed character by a 0.
CALL FUNCTION 'RSKC_ALLOWED_CHAR_GET'
IMPORTING
E_ALLOWED_CHAR = WA_ALLOWED_CHAR
EXCEPTIONS
OTHERS = 1.
…
* check on allowed characters
TRANSLATE WA_LINE TO UPPER CASE.
DO.
IF WA_LINE cn WA_ALLOWED_CHAR.
WA_LINE+sy-fdpos(1) = 0.
ELSE.
EXIT.
ENDIF.
ENDDO.

Check on records with all fields empty

Checks if the first 5 characters contains not a delimiter. If it does not contain delimiters it continues.
The case is that there are records with only delimiters and no data.
* check on records with all fields empty
IF WA_LINE(5) CN WA_SPRTR.
….
Write new data record
…

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 5
 Flat File Upload – Considerations - Preliminary Program

ENDIF.

Check on comma as field separator and as decimal.

Some csv files have comma as a field separator. on top of that the comma is also used a the decimal sign.
For instance some records are like
Field1,”123,45”,Field3
This removes the inverted comma and changes the decimal comma in a decimal point.
* check on " and change decimal comma in decimal point
WA_SWNVRTDCMM = 0.
DESCRIBE FIELD WA_LINE :LENGTH WA_CLEN IN CHARACTER MODE.
WA_COUNT3 = 0.
DO WA_CLEN TIMES.
IF WA_LINE+WA_COUNT3(1) = C_INVERTED_COMMA.
WA_LINE+WA_COUNT3(1) = SPACE.
IF WA_SWNVRTDCMM = 0.
WA_SWNVRTDCMM = 1.
ELSE.
WA_SWNVRTDCMM = 0.
ENDIF.
ENDIF.
IF WA_LINE+WA_COUNT3(1) = C_COMMA AND WA_SWNVRTDCMM = 1.
WA_LINE+WA_COUNT3(1) = C_POINT.
ENDIF.
WA_COUNT3 = WA_COUNT3 + 1.
IF WA_LINE+WA_COUNT3(10) = SPACE.
EXIT.
ENDIF.
ENDDO.

Read data of the record into a table

This is useful to manipulate data in a table; This can be the check of field types.
* read into table
WA_RECORD = WA_LINE.
CALL FUNCTION 'RSDS_CONVERT_CSV'
EXPORTING
I_DATA_SEP = WA_SPRTR
I_ESC_CHAR = C_SCP
I_RECORD = WA_RECORD
I_FIELD_COUNT = WA_FLDCNT
IMPORTING
E_T_DATA = WA_DATA
EXCEPTIONS

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 6
 Flat File Upload – Considerations - Preliminary Program

ESCAPE_NO_CLOSE = 1
ESCAPE_IMPROPER = 2
CONVERSION_ERROR = 3.

Check field types

This checks the field type date , numeric, character of a field based on parameters in an internal table.
* check field types
IF WA_SWITHBEGIN <> C_B.
WA_COUNT = 1.
WHILE WA_COUNT <= WA_FLDCNT.
LOOP AT IT_PARAMETER_S ASSIGNING <FS_PARAMETER>.
IF <FS_PARAMETER>-/BIC/FLDNM = WA_COUNT.
READ TABLE WA_DATA INDEX WA_COUNT INTO WA_DATA2.
CASE <FS_PARAMETER>-/BIC/TPFLD.
WHEN C_NUM.
IF WA_DATA2 CN C_NUMERIC.
PERFORM FILE_STRUCTURE_ERROR.
ENDIF.
WHEN C_DATS.
IF WA_DATA2(4) < C_1900 AND
WA_DATA2+6(4) < C_1900.
PERFORM FILE_STRUCTURE_ERROR.
ENDIF.
ENDCASE.
ENDIF.
ENDLOOP.
WA_COUNT = WA_COUNT + 1.
ENDWHILE.
ENDIF.
WA_SWITHBEGIN = C_C.

Include filename in a field

This enables to load the filename into a DSO and use it or part of it. For instance to exploit the date
timestamp that is in it.
* add filename into data
CONCATENATE WA_LINE
WA_SPRTR
WA_FILENAME
INTO WA_LINE.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 7
 Flat File Upload – Considerations - Preliminary Program

Write the record

This writes a check record in the new file. This new file will be used to load.
* write new data file
TRANSFER WA_LINE to WA_DIRFILENAMENEW.
….
CLOSE DATASET WA_DIRFILENAMENEW.

Other considerations
Amongst other considerations that will be explained later are:
• Using DSO for redundant data
• Making data uniform
• Using parameters
• Using file masks
• Sizing of data packages
• Limiting number of processes
• Converting xml to an internal table.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 8
 Flat File Upload – Considerations - Preliminary Program

Related Content
Help.sap.com
Creating DataSources for File Source Systems
http://help.sap.com/saphelp_nw2004s/helpdata/en/43/01ed2fe3811a77e10000000a422035/frameset.htm

Articles
Gaurav Paul , Data Upload from Flat File to Business Intelligence 7.0 System
https://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/60debae1-84dd-2b10-e7bf-
bdedf1eabdf9&overridelayout=true
Aveek Ghose ,Flat File Interfaces in SAP
https://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/c0341dc1-515c-2910-f99b-
b575e9f74671&overridelayout=true
Anurag Krishna Dev , Making Flat File Data Sources Delta Capable By Using Routine In Info package
Selection.
https://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/a0b3f0e2-832d-2c10-1ca9-
d909ca40b54e&overridelayout=true

Blogs
Loading Text Files(.TXT) into BW with HEX option
https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4676
Hierarchy Upload from Flat files
https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/3113
Binary File in BW: what I can do with control file
https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4803
How-to... Load a File into SAP NetWeaver BI Integrated Planning (Part 1)
https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/6671
How-to... Load a File into SAP NetWeaver BI Integrated Planning (Part 2)
https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/7708

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 9
 Flat File Upload – Considerations - Preliminary Program

Disclaimer and Liability Notice

This document may discuss sample coding or other information that does not include SAP official interfaces and therefore is not
supported by SAP. Changes made based on this information are not supported and can be overwritten during an upgrade.
SAP will not be held liable for any damages caused by using or misusing the information, code or methods suggested in this document,
and anyone using these methods does so at his/her own risk.
SAP offers no guarantees and assumes no responsibility or liability of any type with respect to the content of this technical article or
code sample, including any liability resulting from incompatibility between the content within this document and the materials and
services offered by SAP. You agree that you will not hold, or seek to hold, SAP responsible or liable with respect to the content of this
document.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com

© 2009 SAP AG 10