Jobacle
Oracle Enterprise Manager
Jobacle, New York, New York, United States,
In this cloud-era it’s almost weird to explore stuff on-premises, but I did…. In this case a slightly Higher Availibility configuration of Oracle Enterprise Manager (OEM) that doesn’t cost anything, is not disruptive to the existing configuration and gives a tiny bit more confidence during patching of OEM : Always On Monitoring (AOM). Maybe a bit underrated functionality – within Oracle .When Oracle Management Server (OMS) is down for patching, AOM is capable of mailing you when a critical target is down. It’s a limited functionality, but can be quite useful in some cases. Some details has changed since the start in 2016, when DBIservices
wrote about it. At the end of the blogpost my findings.What have I done in this blogpost1. Using an existing OEM 13.3 installation – Vagrant build, by Oracle-base .2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.3. Install / unzip the AOM-software, and create the AOM repository4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.6. Some findings at the end1. Using an existing OEM 13.3 installation – Vagrant build.
The following configuration is my base: OEM 13.3 with a repository on the same server.Connected to this database:Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – ProductionVersion 19.3.0.0.0SQL> show pdbsCON_ID CON_NAME OPEN MODE RESTRICTED———- —————————— ———- ———-2 PDB$SEED READ ONLY NO3 EMREP READ WRITE NO2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.
Create a seperate PDB OAMREP (sorry, should be AOMREP….) for the Always On Monitoring repository.SQL> Create pluggable database oamrep admin user oamsys identified by oracle;Pluggable database created.SQL> show pdbsCON_ID CON_NAME OPEN MODE RESTRICTED———- —————————— ———- ———-2 PDB$SEED READ ONLY NO3 EMREP READ WRITE NO4 OAMREP MOUNTEDSQL> alter session set container=oamrep;Session altered.Create a seperate PDB TESTDB for test-purposes later on.SQL> Create pluggable database testdb admin user testdba identified by oracle;Pluggable database created.SQL> show pdbsCON_ID CON_NAME OPEN MODE RESTRICTED———- —————————— ———- ———-2 PDB$SEED READ ONLY NO3 EMREP READ WRITE NO4 OAMREP READ WRITE NO5 TESTDB READ WRITE NOHere’s what it looks likePut the databases in OEM:Let them automatically start when CDB starts:alter pluggable database all save state;According to the documentation, trying to set the parameter:*ERROR at line 1:ORA-25138: OPTIMIZER_ADAPTIVE_FEATURES initialization parameter has been madeO.k. this has changed apparently, I’m fine.Characterset should be AL32UTF8SQL> SELECT * from nls_database_parameters where parameter=’NLS_CHARACTERSET’;NLS_CHARACTERSETAL32UTF8Put OAMREP for convenience in TNSNAMES.ORACreate user EMS in OAMREPcreate user ems identified by ems;grant CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER, CREATE TYPE, CREATE VIEW, UNLIMITED TABLESPACE, SELECT ANY DICTIONARY to ems;grant EXECUTE ON SYS.DBMS_CRYPTO to ems;grant EXECUTE ON SYS.DBMS_AQADM to ems;grant EXECUTE ON SYS.DBMS_AQ to ems;grant EXECUTE ON SYS.DBMS_AQIN to ems;grant EXECUTE on SYS.DBMS_LOCK to ems;grant EXECUTE ON SYS.DBMS_SCHEDULER to ems;grant create job to ems;3. Install / unzip the AOM-software, and create the AOM repository
Unzip the binaries:/u01/app/oracle/middleware/sysman/emsems_13.3.0.0.0.zipPut the emkey in the OEM-repository:Oracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.After the required operation has been completed, secure the EMKey by running “emctl config emkey -remove_from_repos”.Go to the script-directory:cd /u01/app/oracle/middleware/sysman/ems/ems/scriptsShould be Java to run emsca for creating the AOM repository:Either JAVA_HOME environment variable is not set or an incorrect Java version is used. To run emsca, use Java 1.7Trying seperate settingsJava(TM) SE Runtime Environment (build 1.7.0_171-b11)Nope, this won’t fly. Then a seperate install – yum install java. This is working!Creating OAM repository:Oracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.—————————————————————Always-On Monitoring Repository Username [ems] :Always-On Monitoring Repository Password [ems] :Always-On Monitoring Repository user “ems” has already been createdEnterprise Manager Repository Connection String : localhost:1521:emrepEnterprise Manager Repository Username : sysmanEnterprise Manager Repository Password :Agent Registration Password :Keystore for host ol7-emcc.localdomain created successfully.Connecting to Always-On Monitoring Repository.Creating repository storage for Targets data.Creating repository storage for Alerts and Availability data.Creating repository storage for Notification Metadata data.Creating repository storage for Target Metric Metadata data.Registering Always-On Monitoring instanceRemove emkey from OEM-repository:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emctl config emkey -remove_from_reposOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :The EMKey has been removed from the Management Repository.The status:4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.
Setting the mail property for OAMError: The command name “set_oms_property” is not a recognized command.Run the “help” command for a list of recognized commands.You may also need to run the “sync” command to synchronize with the current OMS.How could I forget…[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli login -username=sysmanLogin successful[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli syncSynchronized successfullyTrying to set the mail-property again:Property “oracle.sysman.core.events.ems.downtimeContact” has been successfully set to value “ ” on all Management Servers.Server restart is not required for the property change to get reflectedDefine downtime contact from the target property, tried it first with the examples I saw.Error: Invalid target in record: TESTDB:oracle_database:Downtime ContactAllright, it’s a pluggable database, need another name and type. What choices I have:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli get_targets -target=oracle_pdb;IDAnother attempt:Properties updated successfullySynchronizing for the first time:[oracle@ol7-emcc scripts]$ pwd/u01/app/oracle/middleware/sysman/ems/ems/scripts[oracle@ol7-emcc scripts]$ ./emsctl syncOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Connecting to Always-On Monitoring Repository.Starting synchronization with Enterprise Manager.Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrepSynchronizing Alerts and Availability data.Synchronization complete at : Wed Apr 22 11:43:01 UTC 2020Merge AOM with OEM by setting the AOM string in the agents:Oracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Property oracle.sysman.core.events.ems.emsURL has been set to value https://ol7-emcc.localdomain:8081/upload for all Management ServersOMS restart is not required to reflect the new property valueCheck URL in agent:[oracle@ol7-emcc config]$ pwd/u01/app/oracle/agent/agent_inst/sysman/config[oracle@ol7-emcc config]$ grep 8081 emd.propertiesYou can always start and stop OAM individual:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl startOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Always-On Monitoring is not running.Starting Always-On Monitoring.Waiting for process to startNotifications Enabled : trueTotal Downtime Contacts Configured : 1Always-On Monitoring is up.Enable notifications:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl enable_notificationOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Notifications have been enabled. There are downtime contacts configured.Connecting to Always-On Monitoring Repository.Starting synchronization with Enterprise Manager.Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrepSynchronizing Alerts and Availability data.Synchronization complete at : Wed Apr 29 09:55:56 UTC 2020Are the targets in de AOM-repository (user EMS):5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.
Let’s test it by stopping the oms.[oracle@ol7-emcc scripts]$ $OMS_HOME/bin/emctl stop oms –all. And we’ve got mail.Other test: stopping the TESTDB pluggable databaseResult: I’ve got (2) mails, from 2 different ‘event_types’.Let’s open the PDB testdb :And I’ve got 2 mails again:– 11:13 AM : emcdb_TESTDB – The pluggable database is in OPEN state.– 11:13 AM : emcdb_TESTDB – The pluggable database is up.Let’s start the OMS.$OMS_HOME/bin/emctl start omsLet’s see if OMS knows what happened in the time he was down.That seems to be the case (don’t mind the different time-lines. There’s a two hour difference between my Linux VM and my laptop, so 12:39 is for OEM 10:39) .Suppose we are closing the agent:[oracle@ol7-emcc scripts]$ $AGENT_HOME/bin/emctl stop agentOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Status OAM gives no clue of what is happening.[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl statusOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Always-On Monitoring Home : /u01/app/oracle/middleware/sysman/ems/emsLast Repository Sync : April 29, 2020 6:26:15 PM UTCAlways-On Monitoring Process ID : 26450Enterprise Manager Repository : localhost:1521:emrepNotifications Enabled : trueTotal Downtime Contacts Configured : 1Starting the agent gives us two mail: one to mention that the agent is down (!), and one that he’s up.6. Some findings
– Installation is not that hard, but requires a few steps in the right order.– Installation / configuration is not ‘disruptive’ of your current OEM-configuration.– It’s a limited functionality – in fact the down-time of a target – and only use of mail-communication is possible. But while it’s not disruptive, it could be worthy when you don’t want to be ‘blind’ while patching, and you’ve got a few very critical targets.– AOM is not very ‘hot’ within OEM and apparently not on the developer’s priorities-list.– Outage of a node / agent will not be noticed by AOM, although you can see if the agents are communicating with AOM through the command ‘emsctl list_agents’. Must be a way to use this.– It’s all commandline. To be hated, to be loved.– You need an extra hole in your firewall for the communication of the agents with the AOM software.– Configuring a lot of agents should be one command, haven’t tested this.– Made a mistake in configuring the URL in the agents. No mail, and no clue where I made the mistake. Always check if the agents are communicating with the AOM software with ‘emsctl list_agents’.
#J-18808-Ljbffr
wrote about it. At the end of the blogpost my findings.What have I done in this blogpost1. Using an existing OEM 13.3 installation – Vagrant build, by Oracle-base .2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.3. Install / unzip the AOM-software, and create the AOM repository4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.6. Some findings at the end1. Using an existing OEM 13.3 installation – Vagrant build.
The following configuration is my base: OEM 13.3 with a repository on the same server.Connected to this database:Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – ProductionVersion 19.3.0.0.0SQL> show pdbsCON_ID CON_NAME OPEN MODE RESTRICTED———- —————————— ———- ———-2 PDB$SEED READ ONLY NO3 EMREP READ WRITE NO2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.
Create a seperate PDB OAMREP (sorry, should be AOMREP….) for the Always On Monitoring repository.SQL> Create pluggable database oamrep admin user oamsys identified by oracle;Pluggable database created.SQL> show pdbsCON_ID CON_NAME OPEN MODE RESTRICTED———- —————————— ———- ———-2 PDB$SEED READ ONLY NO3 EMREP READ WRITE NO4 OAMREP MOUNTEDSQL> alter session set container=oamrep;Session altered.Create a seperate PDB TESTDB for test-purposes later on.SQL> Create pluggable database testdb admin user testdba identified by oracle;Pluggable database created.SQL> show pdbsCON_ID CON_NAME OPEN MODE RESTRICTED———- —————————— ———- ———-2 PDB$SEED READ ONLY NO3 EMREP READ WRITE NO4 OAMREP READ WRITE NO5 TESTDB READ WRITE NOHere’s what it looks likePut the databases in OEM:Let them automatically start when CDB starts:alter pluggable database all save state;According to the documentation, trying to set the parameter:*ERROR at line 1:ORA-25138: OPTIMIZER_ADAPTIVE_FEATURES initialization parameter has been madeO.k. this has changed apparently, I’m fine.Characterset should be AL32UTF8SQL> SELECT * from nls_database_parameters where parameter=’NLS_CHARACTERSET’;NLS_CHARACTERSETAL32UTF8Put OAMREP for convenience in TNSNAMES.ORACreate user EMS in OAMREPcreate user ems identified by ems;grant CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER, CREATE TYPE, CREATE VIEW, UNLIMITED TABLESPACE, SELECT ANY DICTIONARY to ems;grant EXECUTE ON SYS.DBMS_CRYPTO to ems;grant EXECUTE ON SYS.DBMS_AQADM to ems;grant EXECUTE ON SYS.DBMS_AQ to ems;grant EXECUTE ON SYS.DBMS_AQIN to ems;grant EXECUTE on SYS.DBMS_LOCK to ems;grant EXECUTE ON SYS.DBMS_SCHEDULER to ems;grant create job to ems;3. Install / unzip the AOM-software, and create the AOM repository
Unzip the binaries:/u01/app/oracle/middleware/sysman/emsems_13.3.0.0.0.zipPut the emkey in the OEM-repository:Oracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.After the required operation has been completed, secure the EMKey by running “emctl config emkey -remove_from_repos”.Go to the script-directory:cd /u01/app/oracle/middleware/sysman/ems/ems/scriptsShould be Java to run emsca for creating the AOM repository:Either JAVA_HOME environment variable is not set or an incorrect Java version is used. To run emsca, use Java 1.7Trying seperate settingsJava(TM) SE Runtime Environment (build 1.7.0_171-b11)Nope, this won’t fly. Then a seperate install – yum install java. This is working!Creating OAM repository:Oracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.—————————————————————Always-On Monitoring Repository Username [ems] :Always-On Monitoring Repository Password [ems] :Always-On Monitoring Repository user “ems” has already been createdEnterprise Manager Repository Connection String : localhost:1521:emrepEnterprise Manager Repository Username : sysmanEnterprise Manager Repository Password :Agent Registration Password :Keystore for host ol7-emcc.localdomain created successfully.Connecting to Always-On Monitoring Repository.Creating repository storage for Targets data.Creating repository storage for Alerts and Availability data.Creating repository storage for Notification Metadata data.Creating repository storage for Target Metric Metadata data.Registering Always-On Monitoring instanceRemove emkey from OEM-repository:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emctl config emkey -remove_from_reposOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :The EMKey has been removed from the Management Repository.The status:4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.
Setting the mail property for OAMError: The command name “set_oms_property” is not a recognized command.Run the “help” command for a list of recognized commands.You may also need to run the “sync” command to synchronize with the current OMS.How could I forget…[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli login -username=sysmanLogin successful[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli syncSynchronized successfullyTrying to set the mail-property again:Property “oracle.sysman.core.events.ems.downtimeContact” has been successfully set to value “ ” on all Management Servers.Server restart is not required for the property change to get reflectedDefine downtime contact from the target property, tried it first with the examples I saw.Error: Invalid target in record: TESTDB:oracle_database:Downtime ContactAllright, it’s a pluggable database, need another name and type. What choices I have:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli get_targets -target=oracle_pdb;IDAnother attempt:Properties updated successfullySynchronizing for the first time:[oracle@ol7-emcc scripts]$ pwd/u01/app/oracle/middleware/sysman/ems/ems/scripts[oracle@ol7-emcc scripts]$ ./emsctl syncOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Connecting to Always-On Monitoring Repository.Starting synchronization with Enterprise Manager.Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrepSynchronizing Alerts and Availability data.Synchronization complete at : Wed Apr 22 11:43:01 UTC 2020Merge AOM with OEM by setting the AOM string in the agents:Oracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Property oracle.sysman.core.events.ems.emsURL has been set to value https://ol7-emcc.localdomain:8081/upload for all Management ServersOMS restart is not required to reflect the new property valueCheck URL in agent:[oracle@ol7-emcc config]$ pwd/u01/app/oracle/agent/agent_inst/sysman/config[oracle@ol7-emcc config]$ grep 8081 emd.propertiesYou can always start and stop OAM individual:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl startOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Always-On Monitoring is not running.Starting Always-On Monitoring.Waiting for process to startNotifications Enabled : trueTotal Downtime Contacts Configured : 1Always-On Monitoring is up.Enable notifications:[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl enable_notificationOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Notifications have been enabled. There are downtime contacts configured.Connecting to Always-On Monitoring Repository.Starting synchronization with Enterprise Manager.Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrepSynchronizing Alerts and Availability data.Synchronization complete at : Wed Apr 29 09:55:56 UTC 2020Are the targets in de AOM-repository (user EMS):5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.
Let’s test it by stopping the oms.[oracle@ol7-emcc scripts]$ $OMS_HOME/bin/emctl stop oms –all. And we’ve got mail.Other test: stopping the TESTDB pluggable databaseResult: I’ve got (2) mails, from 2 different ‘event_types’.Let’s open the PDB testdb :And I’ve got 2 mails again:– 11:13 AM : emcdb_TESTDB – The pluggable database is in OPEN state.– 11:13 AM : emcdb_TESTDB – The pluggable database is up.Let’s start the OMS.$OMS_HOME/bin/emctl start omsLet’s see if OMS knows what happened in the time he was down.That seems to be the case (don’t mind the different time-lines. There’s a two hour difference between my Linux VM and my laptop, so 12:39 is for OEM 10:39) .Suppose we are closing the agent:[oracle@ol7-emcc scripts]$ $AGENT_HOME/bin/emctl stop agentOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.Status OAM gives no clue of what is happening.[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl statusOracle Enterprise Manager Cloud Control 13c Release 3Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.——————————————————————Always-On Monitoring Home : /u01/app/oracle/middleware/sysman/ems/emsLast Repository Sync : April 29, 2020 6:26:15 PM UTCAlways-On Monitoring Process ID : 26450Enterprise Manager Repository : localhost:1521:emrepNotifications Enabled : trueTotal Downtime Contacts Configured : 1Starting the agent gives us two mail: one to mention that the agent is down (!), and one that he’s up.6. Some findings
– Installation is not that hard, but requires a few steps in the right order.– Installation / configuration is not ‘disruptive’ of your current OEM-configuration.– It’s a limited functionality – in fact the down-time of a target – and only use of mail-communication is possible. But while it’s not disruptive, it could be worthy when you don’t want to be ‘blind’ while patching, and you’ve got a few very critical targets.– AOM is not very ‘hot’ within OEM and apparently not on the developer’s priorities-list.– Outage of a node / agent will not be noticed by AOM, although you can see if the agents are communicating with AOM through the command ‘emsctl list_agents’. Must be a way to use this.– It’s all commandline. To be hated, to be loved.– You need an extra hole in your firewall for the communication of the agents with the AOM software.– Configuring a lot of agents should be one command, haven’t tested this.– Made a mistake in configuring the URL in the agents. No mail, and no clue where I made the mistake. Always check if the agents are communicating with the AOM software with ‘emsctl list_agents’.
#J-18808-Ljbffr