NON TARIFF

MEASURES
PROCESS
ARCHITECTURE
IT&S SUPPORT REQUEST
V1, 29/01/2013

Author: Vincenzo Petrucci

OVERVIEW

This document is a general overview of the Non-Tariff Measures (NTM) process in

MacMap application and a user guide for running loading data process.

The scope of this document is providing the elements for understanding and
managing the whole process from the database level towards the web application.

The picture below provides the NTM architecture.

HOW TO RUN ETL

NTM team provides two files in folder

- Country data file

- Group file

Country data file need to be loaded in development environment. Group file does not
need to be loaded because groups affected are existing in the table
WMDS.NTM_COUNTRY_GROUPS (development instance sql server
witsmacmapdev),
WMDO.NTM_COUNTRY_GROUPS (production instance sql server prod).
The files are in the folder:

K:\MAS CAMAD\MAcMap data\DATABASE\ntm\ntm end

The name of the file is for example: KZ_NTM_2012_OK.txt

Move manually the file, on MacMap development server, in the folder:
C:\WMD_DATA\01_Source_DATA\05_NTM\01_In

In the local machine:

- open Visual Studio

- edit the package Load NTM data.dtsx
- change the value of the variable data_source_file_path with the name of
current Country File
- Save and deploy the package
- From SQL Server Management Studio:
select MAX(UPDATE_DATE) from T_NTM_LOAD;

On development server:

- Open Sql Server Management Studio

- Run the package Load NTM data.dtsx

If no errors test the MacMap web application using the modules

Quick Search -> Find non-tariff measures, so check if the country is available in
the drop-down list so proceed with the other parameters.
Whenever the data are loaded, ask NTM / Marketing team to test the data quality:

Example:
HOT MOVE TO PROD, CERTIFIED DATA:

- Connect witsmacmapdev server in Remote Desktop

- Run the following query in SQL Server (opening a sql connection from dev to
prod server):
select DEV.UPDATE_DATE_DEV, DEV.TABLE_NAME from
(
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'R_NTM_WORD' AS TABLE_NAME
from WMDO.DBO.R_NTM_WORD
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'R_NTM_TRANSLATION' AS
TABLE_NAME from WMDO.DBO.R_NTM_TRANSLATION
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_LOAD' AS TABLE_NAME
from WMDO.DBO.T_NTM_LOAD
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_AFFECTED_COUNTRY' AS
TABLE_NAME from WMDO.DBO.T_NTM_AFFECTED_COUNTRY
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_AFFECTED_COUNTRY_GROUP'
AS TABLE_NAME from WMDO.DBO.T_NTM_AFFECTED_COUNTRY_GROUP
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_AFFECTED_PRODUCT' AS
TABLE_NAME from WMDO.DBO.T_NTM_AFFECTED_PRODUCT
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_LEGISLATION_NEW_2' AS
TABLE_NAME from WMDO.DBO.T_NTM_LEGISLATION_NEW_2
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_CREDIT' AS TABLE_NAME
from WMDO.DBO.T_NTM_CREDIT
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_DATE' AS TABLE_NAME from
WMDO.DBO.T_NTM_DATE
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_REF_FRAMEWORK' AS
TABLE_NAME from WMDO.DBO.T_NTM_REF_FRAMEWORK
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_DEV, 'T_NTM_COLLECTION_NEW' AS
TABLE_NAME from WMDO.DBO.T_NTM_COLLECTION_NEW
) DEV
WHERE NOT EXISTS
(SELECT * FROM
 (select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD, 'R_NTM_WORD' AS TABLE_NAME
from [192.168.101.245].WMDO.DBO.R_NTM_WORD
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD,
'R_NTM_TRANSLATION' AS TABLE_NAME from [192.168.101.245].WMDO.DBO.R_NTM_TRANSLATION
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD, 'T_NTM_LOAD' AS
TABLE_NAME from [192.168.101.245].WMDO.DBO.T_NTM_LOAD
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD,
'T_NTM_AFFECTED_COUNTRY' AS TABLE_NAME from
[192.168.101.245].WMDO.DBO.T_NTM_AFFECTED_COUNTRY
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD,
'T_NTM_AFFECTED_COUNTRY_GROUP' AS TABLE_NAME from
[192.168.101.245].WMDO.DBO.T_NTM_AFFECTED_COUNTRY_GROUP
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD,
'T_NTM_AFFECTED_PRODUCT' AS TABLE_NAME from
[192.168.101.245].WMDO.DBO.T_NTM_AFFECTED_PRODUCT
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD,
'T_NTM_LEGISLATION_NEW_2' AS TABLE_NAME from
[192.168.101.245].WMDO.DBO.T_NTM_LEGISLATION_NEW_2
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD, 'T_NTM_CREDIT'
AS TABLE_NAME from [192.168.101.245].WMDO.DBO.T_NTM_CREDIT
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD, 'T_NTM_DATE' AS
TABLE_NAME from [192.168.101.245].WMDO.DBO.T_NTM_DATE
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD,
'T_NTM_REF_FRAMEWORK' AS TABLE_NAME from [192.168.101.245].WMDO.DBO.T_NTM_REF_FRAMEWORK
UNION
select DISTINCT UPDATE_DATE AS UPDATE_DATE_PROD,
'T_NTM_COLLECTION_NEW' AS TABLE_NAME from [192.168.101.245].WMDO.DBO.T_NTM_COLLECTION_NEW
) PROD
WHERE DEV.UPDATE_DATE_DEV=PROD.UPDATE_DATE_PROD
)
AND

DEV.UPDATE_DATE_DEV IS NOT NULL

The result of this query gives all data of the NTM tables in development and not
updated in production.

For example:

UPDATE_DATE_DEV TABLE_NAME
2013-01-28 17:29:20.7000000 T_NTM_AFFECTED_PRODUCT
2013-01-28 17:29:21.4600000 T_NTM_LEGISLATION_NEW_2
2013-01-28 17:29:22.5130000 T_NTM_DATE
2013-01-28 17:29:23.2400000 T_NTM_LOAD
2013-01-28 17:29:29.4530000 T_NTM_COLLECTION_NEW
2013-01-28 17:29:31.2330000 R_NTM_WORD
2013-01-28 17:29:32.2530000 R_NTM_TRANSLATION

Which means that we need to update in production those tables with where
condition UPDATE_DATE

- On local machine in Visual Studio edit the package Load R_NTM_Word Dev
to PROD.dtsx
- In the Data Flow, edit OLE DB Source step (for each table) and add in the
query the dates for table R_NTM_WORD and then R_NTM_TRANSLATION.
See picture below as example:
- Save and deploy the package on witsmacmapdev
- On the local machine in Visual Studio edit the package for loading the
datawarehouse
Load Product,credit,legislation,date,load,collection dev to prod.dtsx and
per every step (loading tables) insert in the query all dates in according to
what explained before. So for example if we want to update T_NTM_LOAD
table, open the step Load T_NTM_load and in the ste OLE DB Source put
in the query the UPDATE_DATEs involved (in the previous example is only
one 2013-01-28 17:29:23.2400000 but , for sure, they could be more.
See the pictureS below:
OLE DB Source for T_NTM_LOAD:

Use the same pattern for the other tables.

- Save and deploy package
- On witsmacmapdev server run the packages now modified:

Load R_NTM_Word Dev to PROD.dtsx

Load Product,credit,legislation,date,load,collection dev to
prod.dtsx