Total Pageviews

Saturday 15 March 2014

Information on Errbuf and Retcode Parameters

Errbuf and Retcode are two OUT type mandatory parameters used in PL/SQL procedures which are called through concurrent requests.

Purpose of Errbuf and Retcode Paramters?
A Concurrent Request is submitted with the help of Concurrent Manager. At the end of execution of concurrent request, the status of the request is sent to concurrent manager through these two mandatory parameters.

Errbuf is a parameter which is used to store error message when ever a program gets into an exception block.
The content of Errbuf can be seen in the ‘Completion Text‘ field of concurrent request.

Retcode is a parameter which is used to record the status of the concurrent request. Retcode has 3 possible values

0 – Success
1 – Success but finished with Warning (yellow color)
2 – Error (red color)
Note: Errbuf must be as first parameter and Retcode must be as second parameter in PL/SQL procedure.

Usage of Errbuf and Retcode Paramters?
Using these two parameters in proper way will help to debug the issue easily.
We can pass values to these parameters in exception blocks like shown in the below example

EXCEPTION
      WHEN OTHERS
      THEN
         retcode := 2;
         errbuf:= 'Unexpected Error in When Others of XYZ Procedure'||SQLERRM;
   END;

Friday 14 March 2014

AP List

APLIST

Ledger Information



The ledger that is assigned to the operating unit in which the invoice_id belongs is chosen and information on this ledger is displayed under Ledger Definition.
In R12, a Subledger Accounting Method (SLAM) is assigned to each ledger.
This is displayed in the report under Ledger Definition/Subledger Accounting Options
There are Oracle Seeded SLAMs for Accrual Accounting - Standard Accrual and Encumbrance Accrual. Check if the SLAM associated to the ledger is a seeded one or custom.

Information on both primary and secondary ledgers are presented.


Invoice Data

AP_INVOICES_ALL


Holds invoice header information
Main columns: Invoice_id, invoice_num, invoice_amount,invoice_date,invoice_type_lookup_code, source,
amount_paid, payment_status_flag

If the invoice is cancelled, the cancellation columns that should be populated are:

cancelled_amount
cancelled_by
cancelled_date

The temp_cancelled_amount is set during the cancellation process, and should normally be null if cancellation completed correctly.

The source column shows how the invoice was entered into the system, e.g Manual Invoice Entry, Oracle Project Accounting, Withholding Tax.

If more than one row is returned in the ap_invoices_all table in the aplist, the other
invoices are typically invoices paid by the same check.


AP_INVOICE_LINES_ALL
Link using Invoice_id
Holds invoice line information

1) Line_type_lookup_code -> ITEM, TAX, MISCELLANEOUS,
FREIGHT, AWT,  PREPAY
2) Line_source ->
            MANUAL LINE ENTRY
            HEADER MATCH
            AUTO WITHHOLDING
            PREPAY APPL
            ETAX  -> When Tax calculated by E-Business Tax engine
            IMPORTED
            HEADER FREIGHT
            PO PRICE ADJUSTMENT

3) Discarded_flag - N, Y
When a line is created, the discarded_flag = N.
If the line is discarded, the distributions for the line are reversed, the line_amount is set to 0 and the discarded_Flag is set to 'Y'.

4) Cancelled_flag - 'N', 'Y'
When a line is created, the cancelled_flag = N
If invoice is cancelled, the cancelled_flag on the line is set to Y, distributions are reversed and the line amount is set to 0.


5) Generate_Dists - N, Y, D
N- Do not generate dists
Y - Generate Dists
      If flag is Y, the distrbutions for the line will be generated automatically.
      If the flag is Y, but there is not enough information for the application to generate the     
      invoice distributions, the invoice will go on 'INSUFFICIENT LINE INFO' hold.
D - Done.
      The flag moves to D if the distributions have been generated or if the distribution  
      manually entered.


6) Tax_classification_code
Tax classification code provided for an ITEM line is one way to calculate tax.

7) Accounting_date and Period_name
   Important to note in case of troubleshooting accounting issues.
   Sweep should move the accounting_date and period_name on invoice lines also.

8) Amount
    Verify that the line amount is equal to the sum of its distributions.
  

AP_INVOICE_DISTRIBUTIONS_ALL

Hold Invoice distribution informationLink using Invoice_id.
Each row in invoice distributions table should belong to a corresponding line in ap_invoice_lines_all

1)      Invoice_id
2)      Invoice_line_number -> Determine which line this dist belongs to
3)      Distribution_line_number
4)      Amount
5)      Dist_code_combination_id -> Distribution account
6)      line_type_lookup_code ->
            ITEM
            NONREC_TAX - Nonrecoverable Tax
            REC_TAX - Recoverable Tax
            ERV - Exchange Rate Variance
            TRV - Tax Rate Variance
            TIPV-  Tax IPV
            MISCELLANEOUS
            FREIGHT
            AWT - Withholding Tax
            IPV - Invoice Price Variance
            ACCRUAL - When matched to PO set to Accrue on Receipt
            RETROACCRUAL
            PREPAY - Prepay Application and Unapplication
            RETAINAGE - When invoice matched to PO with retainage
7)      accounting_date
8)      period_name
9)      Accrual_posted_flag - Y for posted, N- Unposted
10)  posted_flag - Y for posted, N- Unposted
11)  match_status_flag - A for Validated, T - Tested, N- Not Validated
12)  reversal_flag - null or Y for reversed.
13)  accounting_event_id - Links to Event_id of XLA_EVENTS table for Actual event
14)  bc_event_id - Links to Event_id of XLA_EVENTS table for the budgetary control event
15)  historical_flag - If the distribution existed prior to R12 upgrade, this flag will be Y
16)  assets_addition_flag
a.       U – “Untested” - Mass Additions Create has not been run with parameters to select this line
b.      N – “No” - Mass Additions Create ran on the invoice line, but it did not meet the criteria to be an asset
c.       Y – “Yes” - Mass Additions Create ran and the item was sent to the fa_mass_additions_gt table to be prepared and posted in FA.

AP_HOLDS_ALL

Stores information on holds on the invoice if any
Link using Invoice_id
Columns: Hold_lookup_code, hold_date, hold_reason, release_lookup_code, release_reason
Unreleased holds have null release_lookup_code and release_reason.


AP_PAYMENT_SCHEDULES_ALL

Holds Payment Schedule information for the invoice
Link using Invoice_id
1) Amount_remaining
2) gross_amount
3) due_date
4)discount_date
5)hold_flag - N or Y(Payment cannot be made)
6)Payment_status_flag - N(Not Paid), Y (Paid), P(Partial)

One row is created in this table by default when invoice is saved.
If an invoice is partially paid, payment_status_flag is set to ‘P’ and amount_remaining is set to the amount left to be paid. If multiple payment schedules are created, there will be more rows.

Tax Data

ZX_LINES_SUMMARY

Holds the summarized information of the taxlines.
This is the information displayed in invoice workbench as line type Tax
All the tax lines from zx_lines that have the same summarization criteria (mainly regime to rate information) will have a corresponding single record in zx_lines_summary

Created during Tax calculation by the Tax engine.
Link to invoice using application_id = 200 and trx_id = invoice_id.
Can also use summary_tax_line_id to link to ap_invoice_lines_all.summary_tax_line_id

Columns:
Summary_tax_line_id
Application_id
Entity_code = ‘AP_INVOICES’
Event_Class_Code
Trx_id
Trx_number
Cancel_flag
Self_assess_flag

Different values for entity_code and event_class_code

ENTITY_CODE                    EVENT_CLASS_CODE
------------------------------ -------------------
AP_INVOICES                    EXPENSE REPORTS
AP_INVOICES                    PREPAYMENT INVOICES
AP_INVOICES                    STANDARD INVOICES

 

ZX_LINES

Holds the Tax Details records for the invoice. Created by the tax engine.
Link to the invoice is using
application_id = 200 and entity_code = 'AP_INVOICES' and trx_id = invoice_id

Columns:
Tax_line_id
Application_id
Entity_code
Event_class_code
Event_type_code
Trx_id -> invoice_id
Trx_line_id
Tax_Regime_code
Tax_status_code
Tax_rate_code
Cancel_flag - 'N' , 'Y'
Self_Assessed_flag - 'N', 'Y'


ZX_LINES_DET_FACTORS

Table stores transaction related attributes for calculating tax /reporting.
Each record in this table represents a transaction line.
Products (Payables) can insert/modify records in this table during calls to tax engine and passes the information to Etax.

Columns:
trx_id - invoice_id
trx_number - invoice_num
line_level_action - action at the transaction line level (create , update, apply_from, unapply_from, delete, cancel etc)
trx_line_type - same as line type in invoice - Item, Misc, Freight)
Line_amt - Transaction Line amount

 

ZX_REC_NREC_DIST
Table holds detailed Recoverable and Non-Recoverable Tax distributions.

For every line in ZX_LINES there can be one or more lines in ZX_REC_NREC_DIST
that stores the Recoverable and Non-recoverable amounts

Data in zx_rec_nrec_dist are mapped to zx_lines using tax_line_id, tax_line_number, summary_tax_line_id


Rec_nrec_tax_dist_id

Application_id

Entity_code

Event_class_code

Event_type_code

Tax_event_class_code

Tax_event_type_code

Trx_id

Trx_line_id

Trx_line_dist_id

recoverable_flag - Y, N
rec_nrec_rate - % recovery

recoverable_flag is 'N' - no recovery (0%recoverable)
recoverable_flag is 'Y' and rec_nrec_rate tells what % is recoverable


AP_SELF_ASSESSED_TAX_DIST_ALL

Link using invoice_id
If there is a Self assessed tax on the invoice, the distributions for the self assessed taxes are stored in this table.
accounting_event_id,accounting_date,period_name,self_assessed_flag = 'Y',invoice_distribution_id,dist_code_combination_id,amount,reversal_flag

Self assessed tax amounts are not included in the invoice amount on ap_invoices_all.


GENERAL
In all the tables columns record_type_code and historical_flag can be verified to identify if its migrated data or E-Tax created data. For upgraded data,

Record_type_code = ‘MIGRATED’
Historical_flag = ‘Y’

When tax calculation is successful - following tables will be populated
zx_lines_summary, zx_lines and zx_lines_det_factors

When tax calculation is attempted but no taxes applicable only zx_lines_det_factors will have data zx_lines and zx_lines_summary will not be populated

Incase of taxless invoice eg. No tax setup at all then no zx tables will be populated.

zx_rec_nrec_dist is populated during determine_recovery - triggered through validation / tax details - distribution button / all distribution button/
or changing some tax info on tax lines (from the detail tax window)


Payments Data


AP_DOCUMENTS_PAYABLE
View to invoice/payment information
Links to the invoice/check through the following
calling_appd_id = 200 ,calling_app_doc_unique_ref1 = Check_id, calling_app_doc_unique_ref2 = invoice_id
caling_app_doc_unique_ref4 = invoice_payment_id
call_app_pay_service_req_code -> PPR name or the Quick Payment ID


AP_INVOICE_PAYMENTS_ALL

Link using invoice_id
Primary key: Invoice_payment_id
Holds payment information. If payment is voided, a reversal rows is created in this table
Main columns: Invoice_id, invoice_payment_id, check_id, accounting_Date, accounting_event_id, accrual_posted_flag, posted_flag


AP_PAYMENT_HISTORY_ALL

Link using check_id
Primary key: payment_history_id
Holds information related to the payment transactions.
Columns: check_id, payment_history_id, transaction_type, accounting_date, accounting_event_id

The different values for transaction_type are:

TRANSACTION_TYPE
-----------------
PAYMENT CREATED
PAYMENT CLEARING
MANUAL PAYMENT ADJUSTED
REFUND CANCELLED
REFUND RECORDED
PAYMENT ADJUSTED
PAYMENT CLEARING ADJUSTED
PAYMENT CANCELLED
PAYMENT UNCLEARING
PAYMENT MATURITY

For every payment event, there is a record created in this table.


AP_PAYMENT_HIST_DISTS

Link using payment_history_id
Primary Key: PAYMENT_HIST_DIST_ID
Columns: Invoice_payment_id, payment_history_id, pa_dist_lookup_code, invoice_distribution_id, pay_dist_lookup_code, accounting_event_id, bank_curr_amount, inv_dist_amount

Different types of lines are
PAY_DIST_LOOKUP_CODE
-----------------------------
CASH
BANK CHARGE
FINAL PAYMENT ROUNDING
PAYMENT TO CLEARING ROUNDING
DISCOUNT
AWT
EXCHANGE RATE VARIANCE

AP_CHECKS_ALL

Primary key: check_id
Columns: Check_id, check_number, amount, check_date, status_lookup_code, void_date
Different values for the status
STATUS_LOOKUP_CODE
-----------------------
NEGOTIABLE
VOIDED
OVERFLOW
STOP INITIATED
SET UP
CLEARED BUT UNACCOUNTED
CLEARED
RECONCILED UNACCOUNTED
RECONCILED
ISSUED

AP_INV_SELECTION_CRITERIA_ALL

The table is populated during Payment Process Request

AP_SELECTED_INVOICES_ALL

The invoices selected for the payment process request is populated in this table

IBY_DOCS_PAYABLE_ALL

Payments assembled during PPR is stored in this IBY table.
Links to Payment process Request using payment_service_request_id
Links to invoice_id using  calling_app_id = 200(Payables)
and  calling_app_doc_unique_ref2 (invoice_id)


IBY_PAYMENTS_ALL

After Build Payments runs, proposed payments are stored in this table
Links to PPR using payment_service_request_id
Links to invoice_id through iby_docs_payable_all.payment_id

IBY_PAY_INSTRUCTIONS_ALL
Continuing the PPR process after build payments will create the payment instruction
Links to PPR using payment_service_request_id

Accounting Data

XLA.XLA_TRANSACTION_ENTITIES

Link using SOURCE_ID_INT_1 that stores the invoice_id or check_id
Columns: application_id = 200, entity_id, source_id_int_1,legal_entity_id, entity_code,upg_batch_id

The different entity_code for payables are
ENTITY_CODE
------------
AP_PAYMENTS
MANUAL
AP_INVOICES

If entity_code = 'AP_INVOICES', source_id_int_1 = invoice_id
If entity_code = 'AP_PAYMENTS', source_id_int_1 = check_id

 In all XlA tables, Upg_batch_id is null for R12 transactions and not null for pre-upgrade
 transactions.


XLA_EVENTS

Link using entity_id
Stored accounting event information for invoices and payments.
Primary_key : event_id

Event_id in this table should exist in accounting_event_id in one of the base transaction tables (ap_invoice_distributions_all, ap_invoice_payments_all, ap_payment_history_all) or should exist as bc_event_id on the invoice distribution table.

Columns: application_id = 200, event_id, entity_id, event_type_code, event_date, event_status_code, process_status_code, budgetary_control_flag, upg_batch_id

Different values for event_type_code
EVENT_TYPE_CODE
----------------------------
CREDIT MEMO VALIDATED
PREPAYMENT UNAPPLIED
PAYMENT MATURED
INVOICE VALIDATED
PREPAYMENT VALIDATED
DEBIT MEMO VALIDATED
PAYMENT CREATED
PREPAYMENT APPLIED
MANUAL
PAYMENT UNCLEARED
PAYMENT CLEARED
MANUAL PAYMENT ADJUSTED
INVOICE CANCELLED
CREDIT MEMO CANCELLED
INVOICE ADJUSTED
REFUND CANCELLED
REFUND RECORDED
PREPAYMENT CANCELLED
PAYMENT ADJUSTED
PAYMENT CLEARING ADJUSTED
PAYMENT CANCELLED

Different Values for event_status_code
-----------------------
U - Not accounted
P - Accounted
I - Incomplete

Different Values for Process_status_code
-------------------------
U - Not Processed
P - Processed
I - Incomplete
D - Draft

Budgetary_control_flag -> Y for budgetary control events created by Invoice validation and Invoice accounting if budgetary control is enabled and Encumbrance AAD is used.

In an instance with encumbrance AAD, when invoice is validated, two events are created, an actual INVOICE VALIDATED event and an encumbrance INVOICE VALIDATED event. The actual event remains unprocessed until accounting runs. The budgetary control event is created and processed (funds reserved) during invoice validation. At the end of validation, if funds reserve is successful, there will be a header and lines for this budgetary control event. This event_id exists as bc_event_id on the corresponding invoice distribution.

XLA_AE_HEADERS

Links using event_id
Primary_key: ae_header_id
Columns: event_id, ae_header_id, application_id =200, entity_id, event_type_code, accounting_date, period_name, gl_transfer_status_code, je_category_name, accounting_entry_status_code, balance_type_code

gl_transfer_status_code
--------------------
N - Not transferred to GL
Y - Transferred to GL

accounting_entry_status_code
--------------------------
I - In Error
R - Related event in error
D - Draft
F - Final

Balance_type_code
----------------
A - Actual
E - Encumbrance


The accounting header created for a budgetary_control event will have balance_type_code = 'E'. Also, when invoice is accounted, accounting should reverse the invoice encumbrance created during Invoice validation. An Invoice Validated actual event in an encumbrance instance will have two headers. One is a header for actual accounting (balance_type_code='A') and another for encumbrance accounting (balance_type_code = 'E')

XLA_AE_LINES

Links using ae_header_id
Has accounting journal lines for the corresponding event. In R12, the accounting is not rolled back in case of errors. The following can be checked in lines
Sum of accounted_dr should match sum of accounted_cr
The lines should have a valid code_combination_id.


XLA_DISTRIBUTION_LINKS

Link using source_distribution_id_num_1 = invoice_distribution_id
Stores distribution level accounting information that is then summarised in xla_ae_lines
Columns: application_id, event_id, ae_header_id, ae_line_num

EVENT_CLASS_CODE                               EVENT_TYPE_CODE
------------------------------                                   -----------------------------
PAYMENTS                                                 PAYMENTS_ALL
PREPAYMENT APPLICATIONS              PREPAYMENT APPLICATIONS_ALL
PREPAYMENTS                                           PREPAYMENTS_ALL
CREDIT MEMOS                                          CREDIT MEMOS_ALL
DEBIT MEMOS                                                        DEBIT MEMOS_ALL
RECONCILED PAYMENTS                                   RECONCILED PAYMENTS_ALL
MANUAL                                                      MANUAL
FUTURE DATED PAYMENTS                   FUTURE DATED PAYMENTS_ALL
INVOICES                                                     INVOICES_ALL
REFUNDS                                                     REFUNDS_ALL


XLA_ACCOUNTING_ERRORS

Links using entity_id or event_id
Stores the accounting errors from the Create Accounting process if error encountered for the event_id
Message_number stores the error number for the event.(Example:95340, 95353,0)

XLA_TRIAL_BALANCES

Links using source_entity_id to XLA_TRANSACTION_ENTITIES.entity_id
Stores the liability lines for trial balance reporting.

AP_PREPAY_APP_DISTS

Links using the invoice_distribution_id
Information on accounting events of prepayments that were applied to the invoice.
1)      invoice_distribution_id – the invoice_distribution_id to which the prepayment was applied.
2)      prepay_app_distribution_id – the invoice_distribution_id of the prepayment event distribution line
3)      accounting_event_id  - the accounting_event_id on the actual invoice distribution line
4)      prepay_dist_lookup_code: values are
PREPAY APPL
PREPAY APPL REC TAX
PREPAY APPL NONREC TAX

Thursday 13 March 2014

Sample Host Program To call the CTL file using Unix

#########################################################################
#
# Application   : AOL
# Module        : GL
# File          : XXXGLINTPRT.prog
# Version       : %R%.%L%
# Last Update   : %E%.%U%
# Description   : This host program will call sqlldr routine to load .csv file to  
#                 GL Interface Staging table for Japan.
#                 If the load is successful then the original file will be renamed  
#                 by suffixing the concurrent request id and date time stamp and put 
#                 to bkup directory, or else, the file will be renamed and copied 
#                 to bad directory alongwith the log generated by sqlldr builtin. 
# Parameters    : V_LOGIN
#                 V_USER_ID
#                 V_USER_NAME
#                 V_REQUEST_ID
#                 v_directory
#                 v_resp_name
#                 v_run_option
#                 v_appl_sname
#                 v_debug_flag
#                 v_file_name
#                 v_in_file_loc
#
# Exec Method   :  connect to sqlplus as < APPS> user and execute this script.
#                 
#
# Change History
# --------------
#
# Date        Author            Ver. Change Description
# ----------- ----------------- ---- ------------------------------------
# 23-Oct-2012 Ashok         1.0  Created
#########################################################################

# Accept the parameters
v_login=$1
v_user_id=$2
v_user_name=$3
v_request_id=$4
v_directory=$5
v_resp_name=$6
v_run_option=$7
v_appl_sname=$8
v_debug_flag=$9
shift
v_file_name=$9
shift
v_in_file_loc=$9
shift
v_in_utp_file_loc=$9

v_suffix=_$v_request_id`date +_%y%m%d_%H:%M`
v_control_file="$v_directory/bin/XXX_GLINT_CBS.ctl"

# Check the output file format 
v_user_name8=`echo $v_user_name | cut -c 1-8`
v_outputfile=`echo $v_directory`/bkup/$v_user_name8.$v_request_id

#-------------------------------------------------------------------------
proc_call_sql()
{
v_ret_code=$1
v_reqst_id=$2

echo ' RESP NAME   :' $v_resp_name
echo ' USER NAME   :' $v_user_name
echo ' REQUEST ID  :' $v_reqst_id
echo ' RETURN CODE :' $v_ret_code

if test $v_ret_code -eq 0
then
    echo '\n Load_Records() returned Success. Submitting Concurent Request to start the validation process.\n'
#   CONCSUB $v_login SQLAP "$v_resp_name" $v_user_name WAIT=N CONCURRENT "EXELAPAC" "EXELAPAC_ARINTV" "$v_reqst_id"
elif test $v_ret_code -eq 2
then
    echo '\n Load_Records() returned Warning. Please check the exception records (at section : between Stat of bad file and End of bad file).'
    echo '\n Load_Records() returned Warning. Submitting Concurent Request to start the validation process.\n'
else
    echo '\n Load_Records() returned Error. Submitting Concurent Request to start the validation process.\n'
fi

}

#-------------------------------------------------------------------------
load_records() 
{
   echo "\nProcedure load_records executes and starts loading..."
   echo "-+-Start of Loader Program -----------------------+-"
   echo " Control File :"$v_control_file
   echo " Log file :"$v_in_file_loc/$v_logfile
   echo " Data file :"$v_in_file_loc/$v_datafile

   #Call the sql loader to load the records into the temporary table.
   sqlldr userid=$v_login control=$v_control_file log=$v_in_file_loc/$v_logfile bad=$v_in_file_loc/$v_badfile data=$v_in_file_loc/$v_datafile errors=999999 silent=HEADER silent=FEEDBACK
   retcode=$?
   echo "\n Displaying log file  : "$v_logfile 
   echo "\n-+-Start of log file -----------------------------+-"
   cat $v_in_file_loc/$v_logfile
   echo "\n-+-End of log file -------------------------------+-\n"
  
   echo 'Return value from Sqlldr : ' $retcode

 # Check for loading with status 'Success'

   if test $retcode -eq 0
   then
       echo "\n Successfully Loaded All Record(s)"
       echo "\n The datafile $v_datafile is sent to the $v_in_file_loc/bkup"
       mv $v_in_file_loc/$v_datafile $v_in_file_loc/bkup/$v_datafile
       rm $v_in_file_loc/$v_logfile
       echo "\n-+-End of Loader Program -------------------------+-\n"
       
       echo $v_user_name
       echo $v_request_id
       
       #Added by Muthu 02-OCT-08
        $ORACLE_HOME/bin/sqlplus -s $v_login <<- EOF
UPDATE XXX_GLINT_TM300i
SET     user_name='$v_user_name'
WHERE request_id=$v_request_id;
SELECT 'Records Updated in Temporary Table for request'||$v_request_id from dual;
Commit;
exit
EOF
       
       return $retcode
   else
       echo "\n Bad Rows found while loading."
       echo "\n The datafile $v_datafile is sent to the $v_directory/bad"
       #mv $v_in_file_loc/$v_datafile $v_directory/bad/$v_datafile
       echo "\n The logfile $v_logfile is sent to the $v_directory/bad"
       mv $v_in_file_loc/$v_logfile $v_directory/bad/$v_logfile
       mv $v_in_file_loc/$v_datafile $v_in_file_loc/bkup/$v_datafile

       if test $retcode -eq 1
       then
           echo "\n Sqlldr has returnd Error code, no record(s) will be processed."
       elif test $retcode -eq 2
       then
echo $v_user_name
echo   $v_request_id

$ORACLE_HOME/bin/sqlplus -s $v_login <<- EOF
UPDATE XXX_GLINT_TM300i
SET     process_flag='B',
       error_description = 'This record is part of a file that had bad record(s) and hence will not be processed further. Please delete this record. Refer to log file of EXEL AU AP REGULAR INVOICE INTERFACE to identify the bad record(s).'
WHERE request_id=$v_request_id;
SELECT 'Records Updated in Temporary Table for request'||$v_request_id "MSG" from dual;
Commit;
exit
EOF
       
           echo "\n Sqlldr has return Warning code, only successfully loaded record(s) will be processed."
       fi

     # Check for the existence of bad file
       echo "\n Searching for Bad file (if Sqlldr has encountered Bad Rows while import)"
       if test -f $v_in_file_loc/$v_badfile
       then
           echo "\n Bad Rows found while loading."
           echo "\n Displaying bad file  : "$v_badfile
           echo "\n-+-Start of bad file -----------------------------+-\n"
           cat $v_in_file_loc/$v_badfile
           echo "\n-+-End of bad file -------------------------------+-\n"
           #rm $v_in_file_loc/$v_badfile
  mv $v_in_file_loc/$v_badfile $v_directory/bad/$v_badfile
           echo "\n-+-End of loader Program--------------------------+-\n"
       fi

       return $retcode
   fi
}

#-------------------------------------------------------------------------
# Main Program
#=========================================================================
echo 'Program Initiated'
echo '-+------------------------------------------------+-'

if test $v_run_option = "L" 
then
  echo 'Source file location  : '$v_in_utp_file_loc
  echo $v_in_utp_file_loc/$v_file_name.txt
  if test -f $v_in_utp_file_loc/$v_file_name.txt
  then
    echo 'Source file name      : '$v_file_name.txt
    cp $v_in_utp_file_loc/$v_file_name.txt $v_in_file_loc/$v_file_name$v_suffix.txt
    v_datafile=$v_file_name$v_suffix.txt
    v_logfile=$v_file_name$v_suffix.log
    v_badfile=$v_file_name$v_suffix.bad
    echo 'Source file renamed to: '$v_datafile
    echo 'Loading records from  : '$v_datafile
    rm $v_in_file_loc/$v_file_name.txt
   
    load_records 
    ret_load=$?

    if test $ret_load -eq 1
    then
      echo '\n Load_Records() returned Error.'  
      echo '\n Calling PL/SQL Script \n'
    else
      echo '\n Calling PL/SQL Script \n'
    fi
  else
    echo '\nFile not found at following path \n'
    echo $v_in_utp_file_loc/$v_file_name.txt
    echo '\n Calling PL/SQL Script \n'
    ret_load=1
  fi
  proc_call_sql "$ret_load" "$v_request_id"
elif test $v_run_option = "P" 
then
    ret_load=0
    v_request_id=-1
    echo ' '
    echo 'Calling PL/SQL Script \n'
    proc_call_sql "$ret_load" " "    
fi
echo ' '
echo "Program ended successfully. \n"
# End of Unix Program

Sample Control File To Load the Flat file data into Staging tables

--#########################################################################
--#
--# Application   : Oracle General Ledger
--# Module        : GL Journals Interface
--# File          : XXXGLINT_CBS.ctl
--# Version       : %R%.%L%
--# Last Update   : %E%.%U%
--# Description   : This scripts loads the data from csv to staging table 
--#                  
--# Parameters    : All the parameter are defined in ctl file
--#
--# Exec Method   : sqlldr userid/pwd 
--#                 
--#
--# Change History
--# --------------
--#
--# Date        Author            Ver. Change Description
--# ----------- ----------------- ---- ------------------------------------
--# 23-Oct-2012 Ashok             1.0  Created
--#
--#########################################################################
--

LOAD DATA
APPEND
INTO TABLE XXX_GLINT_TM300i
FIELDS TERMINATED BY "|"  OPTIONALLY ENCLOSED BY '"' TRAILING NULLCOLS    
(
SET_OF_BOOKS_ID ,
ORG_ID          ,
JOURNAL_SOURCE_NAME,
ACCOUNTING_DATE ,
AP_AR_IDENTIFIER,
CURRENCY_CODE   ,
SEGMENT1        ,
SEGMENT2        ,
SEGMENT3        ,
SEGMENT4        ,
SEGMENT5        ,
SEGMENT6        ,
SEGMENT7        ,
SEGMENT8        ,
SEGMENT9        ,
ENTERED_DR      ,
ENTERED_CR      ,
REFERENCE_10    ,
PERIOD_NAME     "substr(:PERIOD_NAME,1,5)",
RECORD_NUM      RECNUM , 
REQUEST_ID     "(SELECT MAX(b.request_id) FROM FND_CONCURRENT_PROGRAMS a, FND_CONCURRENT_REQUESTS b WHERE a.concurrent_program_id = b.concurrent_program_id AND a.concurrent_program_name = 'XXXGLINTCBS' AND b.phase_code = 'R' AND b.status_code = 'R')"
)

Wednesday 5 March 2014

Loading flat file using SQL Loader

SQL Loader is an Oracle tool used to load data from flat files into oracle tables. It allows the user to selectively load certain columns/rows. Although a programmer could duplicate the functionality of SQL Loader by writing his/her own load program, SQL Loader has the advantage of flexibility, ease of use, and performance. It is often used for migrating data from a non-Oracle source system to an Oracle database.
Two important components of SQL Loader are as follows
Data File: It is a flat file containing the data to be loaded in the oracle table. Each line contains one record to be loaded in the table. Each field in the datafile is separated by a separator character.
Below is a datafile containing 2 records, each row containing 3 columns separated by pipe (|).
1|Ninad|Kamerkar
2|Steven|Clau
Control File: It is a text file written in a language that SQL Loader understands. It tells SQL Loader where to find the data, how to parse and interpret the data, where to insert the data, and more. In other words, we can say that control file is a mapping file containing the mapping from the datafile to oracle table.

Below is the syntax for the control file.

LOAD DATA
INFILE <dataFile>
APPEND INTO TABLE <tableName>
FIELDS TERMINATED BY ‘<separator>’
(<list of all attribute names to load>)
Step-by-step approach for learning SQL Loader
Creating the Target Table
First we need to create a target oracle table in which SQL Loader will load the data from datafile. Execute below DDL on your oracle database schema.








CREATE TABLE Employee (
    Empno NUMBER(4),
    Name VARCHAR2(10),
    Age NUMBER(4),
    Dob DATE,
    Salary NUMBER(8,2),
    Emp_status VARCHAR2(5),
    Job_desgn VARCHAR2(20)
);

Creating the Data File

Below is the datafile that will be loaded into the Employee table. Save this file as Employee.LST. First line in the datafile is the header record. We need to ignore this first line while loading the file.


"NAME"|"AGE"|"SALARY"|"DOB"|"DEPARTMENT"|"JOB DESIGNATION"
"George"|"28"|"50000.30"|"02/02/1957"|"Soft. Dept."|"Soft. Eng."
"Steven"|"38"|"50000"|"08/11/1946"|"Accounts Dept"|""

Creating the Control File
Below is the control file for the above datafile. Save this file as Employee.ctl
















LOAD DATA
INFILE 'Employee.LST'
APPEND
INTO TABLE Employee
FIELDS TERMINATED BY '|'
ENCLOSED BY '"'
TRAILING NULLCOLS
(
    NAME CHAR
    ,AGE INTEGER EXTERNAL
    ,SALARY DECIMAL EXTERNAL
    ,DOB DATE "MM/DD/YYYY"
    ,EMP_STATUS CONSTANT 'A'
    ,EMPNO SEQUENCE( MAX, 1)
    ,SKIP_FIELD FILLER
    ,JOB_DESGN "NVL(:JOB_DESGN,'New Employee')"
)
Line 1: LOAD DATA specifies the beginning of the control file.
Line 2: INFILE specifies name of input files. An asterisk (*) following this keyword indicates data records are in the control file.
Line 3:This specifies the table operation to be performed. Below are all such optopns.
INSERT puts rows into an empty table.
APPEND adds rows to a table.
REPLACE and TRUNCATE delete all rows in a table and place the
current data records into the table.
Line 4: INTO TABLE specifies the target table to load data.
Line 5: FIELDS TERMINATED BY specifies the delimiter which is used in the datafile. In our example we are using pipe (|).
Line 6: ENCLOSED BY specify the character with which each field value is enclosed. In our example we are using double quotes (“). This is an optional parameter.
Line 7: TRAILING NULLCOLS tells SQL Loader how to handle short records with missing data. If the record ends before the field is found, then SQL Loader could either treat the field as null or generate an error. It uses the presence or absence of the TRAILING NULLCOLS clause to determine the action.
Line 9 – 16: These lines contain the actual mapping between the datafile fields and target table column. Here we say that the 1st field coming in the datafile should be put into Name column of employee table, 2nd field coming in the datafile should be put into Age column of employee table and so on.
Line 12: As we receive date in the 4th field, we specify the date format in which the date will come in the datafile.
Line 13: It may happen that you might not receive some data in the datafile and you may want to default some value into the target table column while inserting the record. CONSTANT keyword followed by constant value helps us to default column value.
Line 14: Here we use inbuilt function from SQL Loader called SEQUENCE(MAX, 1), which generates a sequence for the target table. It gets the max empno value from employee table and adds 1 to it to generate new sequence value.
Line 15: We might want to ignore some fields in the datafile. The keyword FILLER identifies the department fields (5th fields in datafile) that are not being loaded into the database. The name does not matter. You can change SKIP_FIELD to any desired value.
Line 16: Here we use oracle function NVL which check whether JOB_DESGN (6th field in datafile) is empty or not. If it is empty we use a default text ‘New Employee’.

Loading Your Data
Once the datafile and control file are ready, we are ready to do the loading. We use sqlldr command to load the datafile as below.


sqlldr <database_user>/<database_password>@<SID>
SKIP=1 ERRORS=1000 ROWS=100 CONTROL=Employee.ctl
LOG=Employee.log BAD=Employee.bad
Below is explanation for parameters passed to SQLLDR command.
Parameter Name
Description
SKIP
Number of logical records to skip (default: 0). SKIP=1 will skip the first record which is header.
ERRORS
Number of errors to terminate the load (default: 50)
ROWS
Number of rows in the conventional path bind array or between direct path data saves (conventional path: 64, direct path: all)
CONTROL
Control filename.
LOG
Log filename.
BAD
Bad filename. During the load process if sql loader finds records which could not be loaded, they are inserted in the file specified with this parameter. The record can then be reviewed to find out the problem. Conditions that may cause a record to be rejected include integrity constraint violation, datatype mismatches, and other errors in field processing.
DISCARD
Discard filename. SQL Loader provides options to selectively reject records based on special criteria. This criteria is defined in the control file as part of the when clause. If SQL Loader encounters a record that fails a specified when clause, the record is placed in a discard file.
DISCARDS
Number of discards to terminate the load (default: all).
When we execute the above command, data is loaded in the Employee table. If sql loader encounters any problem while loading records, a detailed log file containing summary of the load, any errors that occurred during the loading is also produced.
Below is the sample log file generated after successful load operation.
Advantages of using SQL Loader
1.       Has a powerful data parsing engine which puts little limitation on the format of the data in the datafile.
2.       Can load data from multiple data files during the same load session.
3.       Can load data into multiple tables during the same load session.
4.       Is character set aware (you can specify the character set of the data).
5.       Can selectively load data (you can load records based on the records’ values).
6.       Can manipulate the data before loading it, using SQL functions.
7.       Can generate unique sequential key values in specified columns.
8.       Can use the operating system’s file system to access the datafile(s).
9.       Does sophisticated error reporting which greatly aids troubleshooting.

Summary
Oracle SQL Loader processes the input data files according to the directions in a text file called the control file which specifies the names and locations of the source data files, the format of the data to be loaded and the data transformations to be performed when loading.
As the input files are processed, any records that do not pass format checks are written to the bad file and any records that do not meet the specified selection criteria are written to the discard file.
Records that pass both format and selection criteria are written to the specified target tables but they may still be rejected because of, for example, constraint violations in which case they are written to the bad file along with those records rejected for being invalid.

A log file containing a detailed summary of the load, including a description of any errors that occurred during the load is also produced.