ORA-04030: out of process memory when trying to allocate 81944 bytes

[oracle@agsdb ~]$ free -mh
total used free shared buffers cached
Mem: 251G 248G 3.6G 55G 843M 89G
-/+ buffers/cache: 157G 94G
Swap: 63G 9.5G 54G
[oracle@agsdb ~]$

--alter日志
Mon May 11 02:00:00 2020
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager CDB plan via parameter
Mon May 11 02:02:59 2020
Errors in file /u01/app/oracle/diag/rdbms/ora12c/ora12c/trace/ora12c_imco_12034.trc (incident=276331) (PDBNAME=CDB$ROOT):
ORA-04030: out of process memory when trying to allocate 81944 bytes (callheap,IMCO: Segment List)
Incident details in: /u01/app/oracle/diag/rdbms/ora12c/ora12c/incident/incdir_276331/ora12c_imco_12034_i276331.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Current system swap is below 1%
Memory (Avail / Total) = 3835.81M / 257663.78M
Swap (Avail / Total) = 55790.00M / 65536.00M
Errors in file /u01/app/oracle/diag/rdbms/ora12c/ora12c/trace/ora12c_imco_12034.trc (incident=276332) (PDBNAME=CDB$ROOT):
ORA-04030: out of process memory when trying to allocate 81944 bytes (callheap,IMCO: Segment List)
Incident details in: /u01/app/oracle/diag/rdbms/ora12c/ora12c/incident/incdir_276332/ora12c_imco_12034_i276332.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
........此处省略.......
Mon May 11 02:03:13 2020
Dumping diagnostic data in directory=[cdmp_20200511020313], requested by (instance=1, osid=12034 (IMCO)), summary=[incident=276334].
Current system swap is below 1%
Memory (Avail / Total) = 3745.87M / 257663.78M
Swap (Avail / Total) = 55793.23M / 65536.00M
Errors in file /u01/app/oracle/diag/rdbms/ora12c/ora12c/trace/ora12c_imco_12034.trc (incident=276336) (PDBNAME=SSPTRAD):
ORA-04030: out of process memory when trying to allocate 81944 bytes (callheap,IMCO: Segment List)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ora12c/ora12c/trace/ora12c_imco_12034.trc (incident=276337) (PDBNAME=SSPTRAD):
ORA-04030: out of process memory when trying to allocate 81944 bytes (callheap,IMCO: Segment List)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ora12c/ora12c/trace/ora12c_imco_12034.trc (incident=276338) (PDBNAME=SXP2):
ORA-04030: out of process memory when trying to allocate 81944 bytes (callheap,IMCO: Segment List)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Unable to allocate memory for new incident error in file /u01/app/oracle/diag/rdbms/ora12c/ora12c/trace/ora12c_imco_12034.trc:
ORA-04030: out of process memory when trying to allocate 81944 bytes (callheap,IMCO: Segment List)
........此处省略.......
Mon May 11 02:11:22 2020
Errors in file /u01/app/oracle/diag/rdbms/ora12c/ora12c/trace/ora12c_imco_12034.trc:
ORA-04030: out of process memory when trying to allocate 81944 bytes (callheap,IMCO: Segment List)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
DDE: Problem Key 'ORA 4030' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
Mon May 11 02:21:30 2020
DDE: Problem Key 'ORA 4030' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes

--trac 文件内容

--ora12c_imco_12034_i276331

sql_id=2y708xtd9vj74

sql_id=6mqd7fw1zmday

sql_id=dn1jfvz5mkxgy

---ora12c_imco_12034

*** CONTAINER ID:(3) 2019-02-08 14:00:13.675

*** TRACE FILE RECREATED AFTER BEING REMOVED ***

Process has gone over pga_aggregate_limit
Just allocated 131072 bytes
PGA LIMIT: pid 12034 has 0 MB tunable, 19614 MB untunable, and 0 MB freeable

PGA LIMIT: pid 12034 has 0 MB tunable, 32655 MB untunable, and 0 MB freeable
Dumping short stack
----- Abridged Call Stack Trace -----
ksedsts()+244<-ksm_pga_limit_short_stack()+1016<-ksm_check_over_limit()+805<-ksmarfg()+574<-kghgex()+1376<-kghfnd()+361<-kghalo()+4422<-kghgex()+414<-kghfnd()+361<-kghalo()+4422<-kghgex()+414<-kghfnd()+361<-kghalo()+4422<-kghgex()+414<-kghfnd()+361<-kghprmalo()+1617
<-kghalp()+1246<-stsAllocFromSubheap()+385<-smbalo()+155<-smboAloMem()+220<-smboAQSdoRS()+2215<-smboAQSdoQS()+813<-smboAQSdoRS()+3040<-smboAQSdoQS()+813<-smboAQSdoRS()+3040<-smboAQSdoQS()+660<-smboSort()+546<-smbeod()+100<-soreod()+72<-qersoProcessULS()+437
<-qersoFetchSimple()+1196<-qersoFetch()+210<-opifch2()+3115<-opiall0()+1584<-opikpr()+567<-opiodr()+1165<-rpidrus()+206<-skgmstack()+144<-rpiswu2()+723<-kprball()+1163<-kdmrRepopGetSegBatch()+824<-kdmrRepopGetNextSeg()+442<-kdmrRepopGetNextExtent()+790<-kdmrRepopExtentBatch()+43
<-kdmrTrickleRepopMain()+416<-kdmrTrickleRepopCB()+69<-ksbsdact_cbk()+19<-ksbcti()+5018<-ksbabs()+2357<-kdmrimcomain()+1180<-ksbrdp()+1068<-opirip()+1488<-opidrv()+616<-sou2o()+145<-opimai_real()+270<-ssthrdmain()+412<-main()+236<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----
=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
32 GB total:
20 GB commented, 277 KB permanent
12 GB free (0 KB in empty extents),
32 GB, 2 heaps: "callheap " 12 GB free held

*** 2020-05-11 02:33:42.994
------------------------------------------------------
Summary of subheaps at depth 1
20 GB total:
537 KB commented, 20 GB permanent
53 MB free (0 KB in empty extents),


--由此可见，PGA设置太小；需要进行修改；目前大小10G,需要设置成最小 12 GB

----解决办法

SQL> show parameter sga

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
lock_sga boolean FALSE
pre_page_sga boolean TRUE
sga_max_size big integer 232960M
sga_target big integer 232960M
unified_audit_sga_queue_size integer 1048576
SQL> show parameter pga

--SGA 227g

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_limit big integer 20G
pga_aggregate_target big integer 10G
SQL>

SQL> show parameter memory

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
hi_shared_memory_address integer 0
inmemory_clause_default string
inmemory_force string DEFAULT
inmemory_max_populate_servers integer 4
inmemory_query string ENABLE
inmemory_size big integer 200G
inmemory_trickle_repopulate_servers_ integer 1
percent
memory_max_target big integer 0
memory_target big integer 0
optimizer_inmemory_aware boolean TRUE

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
shared_memory_address integer 0
SQL>

SQL> show parameter spfile;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /u01/app/oracle/product/12.1.0
.2/db_1/dbs/spfileora12c.ora
SQL>

ALTER SYSTEM SET pga_aggregate_target='20g' SCOPE=MEMORY;

----ALTER SYSTEM SET sga_max_target=10000M SCOPE=SPFILE;

SQL> ALTER SYSTEM SET pga_aggregate_target='20g' SCOPE=MEMORY;

System altered.

SQL>
SQL> show parameter pga

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_limit big integer 20G
pga_aggregate_target big integer 20G
SQL>

SQL> ALTER SYSTEM SET pga_aggregate_target='20g' SCOPE=SPFILE;

System altered.

SQL> show parameter pga

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_limit big integer 20G
pga_aggregate_target big integer 20G
SQL>