Tuesday, May 30, 2023

RAC Database Startup, Stop and Status using srvctl

 RAC Database Startup, Stop and Status using srvctl


RAC Database Startup in NORMAL mode:


Run below command from primary node as a oracle owner:


srvctl stop database -d DB_NAME


srvctl start database -d DB_NAME


srvctl status database -d DB_NAME


RAC Database Startup in MOUNT /NOMOUNT / RESTRICT mode:


Startup in mount mode:


srvctl start database -d DB_NAME -o mount


Startup in nomount mode:


srvctl start database -d DB_NAME -o nomount


Startup in restrict mode:


srvctl start database -d DB_NAME -o restrict



START / STOP Specific RAC Instance:


To Stop specific RAC instance:


srvctl stop instance -d DB_NAME -i INSTANCE_NAME


To Start specific RAC instance:

srvctl start instance -d DB_NAME -i INSTANCE_NAME

Oracle-Cluster-commands

 Oracle Cluster commands


To stop all cluster services,ASMm, database and instance:


Login as root 


export GRID_HOME=/oradata/oracle/DEV/bin/

cd $GRID_HOME


[root@rac01 ]# ./crsctl stop cluster


To start all cluster services,ASM, database and instance:


Login as root 


export GRID_HOME=/oradata/oracle/DEV/bin/

cd $GRID_HOME


[root@rac01 ]# ./crsctl start cluster





Login as root 


export GRID_HOME=/oradata/oracle/DEV/bin/

cd $GRID_HOME


$ ./crsctl check cluster -all [verify cluster status on all nodes]

$ ./crsctl stop cluster -all [stop cluster on all nodes]

$ ./crsctl start cluster -all [start cluster on all nodes]

$ ./crsctl check cluster -n <nodename> [verify the cluster status on a particular remote node]





To check CRS (Cluster ready service) status:


Login as root


export GRID_HOME=/oradata/oracle/DEV/bin/

cd $GRID_HOME


[root@rac01 ]# ./crsctl check crs







To check status of node apps - where node is the name of the node where the applications are running:




Login as root


cd /oradata/oracle/db/tech_st/11.2.0/bin/


[root@rac01 ]# ./srvctl status nodeapps -n node





crs_stat: Displays CRS resource status:



Login as root


export GRID_HOME=/oradata/oracle/DEV/bin/

cd $GRID_HOME


[root@rac01 ]# ./crs_stat -t







To start particular resource:


Login as root


export GRID_HOME=/oradata/oracle/DEV/bin/

cd $GRID_HOME


[root@rac01 ]# ./crsctl start resource ora.oc4j


ORA-28040: No matching authentication protocol

Exception occurred: java.sql.SQLException: ORA-28040: No matching authentication protocol


Issue:


Unable to connect to database, getting below error:


Exception occurred: java.sql.SQLException: ORA-28040: No matching authentication protocol


Cause:


DB Connectivity Issue after upgrading to Oracle Database 12c.


The issue is caused by the default setting for allowed logon version in the Oracle 12c database.


Note that the SQLNET.ALLOWED_LOGON_VERSION parameter has been deprecated in 12c.


The parameter has been replaced with:


SQLNET.ALLOWED_LOGON_VERSION_SERVER



Solution:


To resolve the database connectivity issue, perform the following steps :


1. Review the sqlnet_ifile.ora or sqlnet.ora file and confirm the following entries are present:

  SQLNET.ALLOWED_LOGON_VERSION_SERVER


2. Update the sqlnet_ifile.ora or sqlnet.ora settings for the above parameters to the lowest version level 

that is required in your environment. 


For example:


a) If the initialization parameter SEC_CASE_SENSITIVE_LOGON is set to FALSE:


  SQLNET.ALLOWED_LOGON_VERSION_SERVER = 8


b) If SEC_CASE_SENSITIVE_LOGON is set to TRUE


  SQLNET.ALLOWED_LOGON_VERSION_SERVER = 10


After this Database connectivity issue should be resolved.

Query To Check Concurrent Program Run History

 

Query To Check Concurrent Program Run History



REM +======================================================================+
REM                    
REM File Name: reqhistory.sql
REM 
REM Description:
REM   Query To Check Concurrent Program Run History
REM   
REM Notes:
REM   Usage: sqlplus <apps_user/apps_passwd> @reqhistory.sql 
REM 
REM Input Required :
REM    Number of days and User Concurrent Program Name  
REM   
REM +======================================================================+

ACCEPT NO_DAYS PROMPT "Enter Number of Days for History: ";
ACCEPT USER_CONC_PROG_NAME PROMPT "Enter User Concurrent Program Name: ";

        clear columns

 set lines 180
 set pages 100
 col Parameters for a20 WORD_WRAPPED
 set pages 100
 col "Conc Program Name" for a30 WORD_WRAPPED
 col "Started at" for a20
 col "Completed at" for a20
 col "Username" for a10 WORD_WRAPPED
 SELECT distinct t.user_concurrent_program_name "Conc Program Name",
 r.REQUEST_ID "Request ID",
 to_char(r.ACTUAL_START_DATE,'dd-MON-yy hh24:mi:ss') "Started at",
 to_char(r.ACTUAL_COMPLETION_DATE,'dd-MON-yy hh24:mi:ss') "Completed at",
 decode(r.PHASE_CODE,'C','Completed','I','Inactive','P','Pending','R','Running','NA') "Phasecode",
 decode(r.STATUS_CODE, 'A','Waiting', 'B','Resuming', 'C','Normal', 'D','Cancelled', 'E','Error', 'F','Scheduled', 'G','Warning', 'H','On Hold', 'I','Normal', 'M',
 'No Manager', 'Q','Standby', 'R','Normal', 'S','Suspended', 'T','Terminating', 'U','Disabled', 'W','Paused', 'X','Terminated', 'Z','Waiting') "Status",r.argument_text "Parameters",
 u.user_name "Username",
 --ROUND ((v.actual_completion_date - v.actual_start_date) * 1440,
 --              2
  --            ) "Runtime (in Minutes)" 
 round(((nvl(v.actual_completion_date,sysdate)-v.actual_start_date)*24*60),2) "ElapsedTime(Mins)"
 FROM
 apps.fnd_concurrent_requests r ,
 apps.fnd_concurrent_programs p ,
 apps.fnd_concurrent_programs_tl t,
 apps.fnd_user u, apps.fnd_conc_req_summary_v v
 WHERE 
 r.CONCURRENT_PROGRAM_ID = p.CONCURRENT_PROGRAM_ID
 AND r.actual_start_date >= (sysdate - &NO_DAYS)
 --AND r.requested_by=22378
 AND   r.PROGRAM_APPLICATION_ID = p.APPLICATION_ID
 AND t.concurrent_program_id=r.concurrent_program_id
 AND r.REQUESTED_BY=u.user_id
 AND v.request_id=r.request_id
 --AND r.request_id ='2260046' in ('13829387','13850423')
 and t.user_concurrent_program_name like '&USER_CONC_PROG_NAME'
 order by to_char(r.ACTUAL_COMPLETION_DATE,'dd-MON-yy hh24:mi:ss') desc;

undef NO_DAYS
undef USER_CONC_PROG_NAME

Script to check workflow mailer service components status

Script to check workflow mailer service components status and send notification if DOWN

 

Pre-requisites to Run the script:

 

Make sure mailx and sendmail is installed on your OS, also check below environements/file is created prior to execute the scripts.

 

This script is tested on Linux Server.

 

This script needs to be deployed in database node/server.

 

#Base location for the DBA scripts

DBA_SCRIPTS_HOME=$HOME/DBA_MON

#OS User profile where database environment file is set

$HOME/.bash_profile

 

 

Add below lines in env

 

Vi .sysenv

 

export DBA_SCRIPTS_HOME=$HOME/DBA_MON

export PATH=${PATH}:$DBA_SCRIPTS_HOME

export DBA_EMAIL_LIST=user @domain.com,

#Below parameter is used in script, whenever there is planned downtime you can set it to Y so there will be no false alert.

export DOWNTIME_MODE=N

 

 

vi WorkflowStatusCheck.sh

 

 

#!/bin/bash

 

###################################################################################

# Script Name :  WorkflowStatusCheck.sh                                           #

#                                                                                 #

# Description:                                                                    #

# Script to check workflow mailer service components status and                   #

# to send alert notification if one or more workflow mailer components are down   #

#                                                                                 #

# Usage : sh <script_name> <ORACLE_SID>                                           #

# For example : sh WorkflowStatusCheck.sh ORCL                                    #

#                                                                                 #

#

###################################################################################

 

# Initialize variables

 

INSTANCE=$1

HOST_NAME=`hostname`

PROGRAM=`basename $0 | cut -d'.' -f1`

export DBA_SCRIPTS_HOME=$HOME/DBA_MON

APPS_ID=`echo $INSTANCE | tr '[:lower:]' '[:upper:]'`

LOG_DIR=$DBA_SCRIPTS_HOME/logs/$HOST_NAME

OUT_FILE=$LOG_DIR/`echo $PROGRAM`_$APPS_ID.html.out

LOG_FILE=$LOG_DIR/`echo $PROGRAM`_$APPS_ID.log

ERR_FILE=$LOG_DIR/`echo $PROGRAM`_$APPS_ID.err

LOG_DATE=`date`

 

 

# Source the env

. $HOME/.bash_profile

. $DBA_SCRIPTS_HOME/.sysenv

 

if [ $? -ne 0 ]; then

   echo "$LOG_DATE" > $LOG_FILE 

   echo "Please pass correct environment : exiting the script  \n" >> $LOG_FILE

   cat $LOG_FILE

   exit

fi

 

if [ -s $OUT_FILE ]; then

 echo "$LOG_DATE" > $LOG_FILE

 echo "Deleting existing output file $OUT_FILE" >> $LOG_FILE

 rm -f $OUT_FILE

 cat $LOG_FILE

fi

 

# If there is a planned downtime then create $ORACLE_SID.down file in $DBA_SCRIPTS_HOME to silent the alerts during maintenance window.

 

if [ -f $DBA_SCRIPTS_HOME/`echo $ORACLE_SID`.down ]; then

 echo "$LOG_DATE" >> $LOG_FILE

        echo "Host: $HOST_NAME | Instance: $ORACLE_SID is under maintenance: exiting the script" >> $LOG_FILE

        cat $LOG_FILE

 exit

fi

 

if [ $DOWNTIME_MODE = "Y" ]; then

 echo "$LOG_DATE" >> $LOG_FILE

 echo "Host: $HOST_NAME | Instance: $ORACLE_SID is under maintenance: exiting the script" >> $LOG_FILE

 cat $LOG_FILE

 exit

fi

 

usage()

{

 echo "$LOG_DATE" > $LOG_FILE

    echo "Script To Check Workflow Mailer Components Status"  >> $LOG_FILE

    echo "Usage   : sh <script_name> <ORACLE_SID> " >> $LOG_FILE

 echo "For example : sh $PROGRAM.sh $ORACLE_SID" >> $LOG_FILE

    echo

}

 

if [ $# -lt 1 ] || [ "$INSTANCE" != "$ORACLE_SID" ]; then

    usage

    echo "Error : Insufficient arguments." >> $LOG_FILE

    cat $LOG_FILE

    exit

fi

 

get_count()

{

 sqlplus -s '/as sysdba' <<!

 set heading off

 set feedback off

 select count(1) from

 (

  select component_id, startup_mode,component_status,component_type,component_name

  from applsys.fnd_svc_components

  where component_status <> 'RUNNING'

  order by component_id

 );

 

 exit;

!

}

 

count=`get_count`

#echo $count

 

echo "$LOG_DATE" > $ERR_FILE

get_count >> $ERR_FILE

ERR_COUNT=`grep "ORA-" $ERR_FILE |wc -l`

 

if [ $ERR_COUNT -gt 0 ]; then

 cat $ERR_FILE | mailx -s "<ERROR> Critical : $APPS_ID - Workflow Mailer Components are DOWN on $HOST_NAME " $DBA_EMAIL_LIST

 exit

fi

 

if [ $count -gt 0 ];

then

 sqlplus -s '/as sysdba' <<EOF

 

 SET ECHO OFF

 SET pagesize 1000

 set feedback off

 set lines 180

 SET MARKUP HTML ON SPOOL ON -

 HEAD '<title></title> -

 <style type="text/css"> -

 table { background: #eee; } -

 th { font:bold 10pt Arial,Helvetica,sans-serif; color:#b7ceec; background:#151b54; padding: 5px; align:center; } -

 td { font:10pt Arial,Helvetica,sans-serif; color:Black; background:#f7f7e7; padding: 5px; align:center; } -

 </style>' TABLE "border='1' align='left'" ENTMAP OFF

 

 spool $OUT_FILE

 

 

 PROMPT Hi Team,

 

 PROMPT

 PROMPT Logon on to OAM (Oracle Applications Manager) and make sure the following processes are running, if not restart them.

 PROMPT

 

 

 select component_id, startup_mode,component_status,component_type,component_name

 from applsys.fnd_svc_components

 where component_status <> 'RUNNING'

 order by component_id;

 

 SPOOL OFF

 SET MARKUP HTML OFF

 exit;

 

EOF

 

(

echo "To: $DBA_EMAIL_LIST"

echo "MIME-Version: 1.0"

echo "Content-Type: multipart/alternative; "

echo ' boundary="PAA08673.1018277622/server.xyz.com"'

echo "Subject: Critical : $APPS_ID - Workflow Mailer Components are DOWN on $HOST_NAME "

echo ""

echo "This is a MIME-encapsulated message"

echo ""

echo "--PAA08673.1018277622/server.xyz.com"

echo "Content-Type: text/html"

echo ""

cat $OUT_FILE

echo "--PAA08673.1018277622/server.xyz.com"

) | /usr/sbin/sendmail -t

 

echo "`date`" > $LOG_FILE

echo "Details sent through an email" >> $LOG_FILE

cat $LOG_FILE

 

else

    echo "`date`" > $OUT_FILE

 echo "Workflow Mailer Components are up and running fine" >> $OUT_FILE

fi

 

 

 

 

Logs and Out files will be generated under $DBA_SCRIPTS_HOME/logs/$HOST_NAME

So make sure to create logs/$HOST_NAME directory under $DBA_SCRIPTS_HOME before executing

the script.

Once the script is ready, then as per the requirement please schedule it in crontab/OEM.

 

Execute the Script as below:

Syntax :  sh <script_name> <ORACLE_SID>

 

 

$ cd $HOME/DBA_MON/bin

$ sh WorkflowStatusCheck.sh ORCL

 

This script will send alert notification if one or more workflow mailer components are down.

 

 

Sample Email Notification Output: