ARRAYSIZE or ROWPREFETCH in sqlplus?
What is the difference between the well known sqlplus-setting arraysize and the new sqlplus-12.2.-feature rowprefetch? In Blog
https://blog.dbi-services.com/oracle-fasttrue-in-sqlplus-some-thoughts-about-rowprefetch/ I showed a case, which helps to reduce the logical IOs when using rowprefetch.
Here the definition of arraysize and rowprefetch according the documentation:
arraysize:
SET System Variable Summary: Sets the number of rows, called a batch, that SQL*Plus will fetch from the database at one time. Valid values are 1 to 5000. A large value increases the efficiency of queries and subqueries that fetch many rows, but requires more memory. Values over approximately 100 provide little added performance. ARRAYSIZE has no effect on the results of SQL*Plus operations other than increasing efficiency
About SQL*Plus Script Tuning: The effectiveness of setting ARRAYSIZE depends on how well Oracle Database fills network packets and your network latency and throughput. In recent versions of SQL*Plus and Oracle Database, ARRAYSIZE may have little effect. Overlarge sizes can easily take more SQL*Plus memory which may decrease overall performance.
REMARK: The arraysize setting also has an impact on the COPY-command with the COPYCOMMIT-setting (commits every n arraysize batches of records).
rowprefetch:
SET System Variable Summary: Sets the number of rows that SQL*Plus will prefetch from the database at one time. The default value is 1 (max is 32767).
Note: The amount of data contained in the prefetched rows should not exceed the maximum value of 2147483648 bytes (2 Gigabytes). The setting in the oraaccess.xml file can override the SET ROWPREFETCH setting in SQL*Plus.
Differences between ARRAYSIZE and ROWPREFETCH
When doing my tests one of the important differences between ARRAYSIZE and ROWPREFETCH is that ROWPREFETCH allows Oracle to transfer query results on return from its internal OCI execute call. I.e. in a 10046-trace the first FETCH is showing ROWPREFETCH rows fetched regardless of the ARRAYSIZE setting. E.g. with the default setting of ROWPREFETCH 1, ARRAYSIZE 15 I can see the following number of rows fetched (see the r= in the trace):
FETCH #139623638001936:c=448,e=1120,p=0,cr=8,cu=0,mis=0,r=1,dep=0,og=1,plh=3403427028,tim=110487525476
...
FETCH #139623638001936:c=66,e=66,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=3403427028,tim=110487525830
...
FETCH #139623638001936:c=15,e=15,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=3403427028,tim=110487526093
...I.e. 1, 15, 15,…
With ROWPREFETCH 3, ARRAYSIZE 15 the rows fetched are 3, 15, 15, …
The following table shows the number of rows fetched with different settings of ROWPREFETCH and ARRAYSIZE from a query, which returns 70 rows:
ROWPREFETCH ARRAYSIZE ROWS_FETCH1 ROWS_FETCH2 ROWS_FETCH3 ROWS_FETCH4 ROWS_FETCH5 ROWS_FETCH6 ROWS_FETCH7 ROWS_FETCH8
1 15 1 15 15 15 15 9
2 15 2 15 15 15 15 8
20 15 20 30 20
16 15 16 30 24
6 5 6 10 10 10 10 10 10 4
9 5 9 10 10 10 10 10 10 1
10 10 10 20 20 20 0
10 5 10 15 15 15 15 0
16 3 16 18 18 18 0We can see 3 things here:
- The first FETCH (from the internal OCI execute) contains always the number of rows as defined in the ROWPREFETCH setting
- The second FETCH (and all subsequent fetches) contains a multiple of the ARRAYSIZE setting rows. The following code fragment should show the logic:
2nd_Fetch_Rows = if ROWPREFETCH < ARRAYSIZE
then ARRAYSIZE
else (TRUNC(ROWPREFETCH/ARRAYSIZE)+1)*ARRAYSIZE
- If a fetch does not detect the end of the data in the cursor then an additional fetch is necessary. In 3 cases above a last fetch fetched 0 rows.
Memory required by the client
With the Linux pmap command I checked how much memory the client requires for different ROWPREFETCH and ARRAYSIZE settings.
Testcase:
SQL> connect cbleile/cbleile@orclpdb1
Connected.
SQL> create table big_type (a varchar2(2000), b varchar2(2000), c varchar2(2000), d varchar2(2000), e varchar2(2000));
Table created.
SQL> insert into big_type select
2 rpad('X',2000,'Y'),
3 rpad('X',2000,'Y'),
4 rpad('X',2000,'Y'),
5 rpad('X',2000,'Y'),
6 rpad('X',2000,'Y') from xmltable('1 to 4100');
4100 rows created.
SQL> commit;
Commit complete.
SQL> exec dbms_stats.gather_table_stats(user,'BIG_TYPE');
SQL> select avg_row_len from tabs where table_name='BIG_TYPE';
AVG_ROW_LEN
-----------
10005Before the test:
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] ps -ef | grep sqlplus
oracle 31537 31636 3 17:49 pts/2 00:01:20 sqlplus as sysdba
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 1580K rw--- [ anon ]
00007f7efcda6000 516K rw--- [ anon ]
...
SQL> show rowprefetch arraysize
rowprefetch 1
arraysize 15
SQL> set arraysize 1000 pages 2 pause on lines 20000
SQL> select * from big_type;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 1580K rw--- [ anon ]
00007f7efc40f000 10336K rw--- [ anon ]
...
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 1580K rw--- [ anon ]
00007f7efcda6000 516K rw--- [ anon ]
SQL> set arraysize 1
SQL> set rowprefetch 1000
SQL> select * from big_type;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 12664K rw--- [ anon ]
00007f7efcda6000 516K rw--- [ anon ]
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 12664K rw--- [ anon ]
00007f7efcda6000 516K rw--- [ anon ]
SQL> set rowprefetc 1
SQL> set arraysize 1000
SQL> select * from big_type;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 22472K rw--- [ anon ]
00007f7efcda6000 516K rw--- [ anon ]
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 12660K rw--- [ anon ]
00007f7efcda6000 516K rw--- [ anon ]
SQL> set rowprefetch 501 arraysize 500 pages 502
SQL> select * from big_type;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 31537 | grep anon
0000000000be6000 17568K rw--- [ anon ]
00007f7efcda6000 516K rw--- [ anon ]New table with just 1 Byte per column:
SQL> create table big_type_small_data as select * from big_type where 1=2;
Table created.
SQL> insert into big_type_small_data select 'X','X','X','X','X' from big_type;
4100 rows created.
SQL> commit;
Commit complete.
SQL> exec dbms_stats.gather_table_stats(user,'BIG_TYPE_SMALL_DATA');
PL/SQL procedure successfully completed.
SQL> select avg_row_len from tabs where table_name='BIG_TYPE_SMALL_DATA';
AVG_ROW_LEN
-----------
10
Client-Memory before the test:
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 14193 | grep anon
00000000014d1000 1580K rw--- [ anon ]
00007f3c3b8d2000 516K rw--- [ anon ]
SQL> show rowprefetch
rowprefetch 1
SQL> show array
arraysize 15
SQL> set arraysize 1000 rowprefetch 1 pages 2 pause on lines 20000
SQL> select * from big_type_small_table;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 14193 | grep anon
00000000014d1000 1580K rw--- [ anon ]
00007f3c3af3b000 10336K rw--- [ anon ]
--> 9.6MB allocated.
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 14193 | grep anon
00000000014d1000 1580K rw--- [ anon ]
00007f3c3b8d2000 516K rw--- [ anon ]
--> All memory released.
SQL> set arraysize 1 rowprefetch 1000
SQL> select * from big_type_snall_data;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 14193 | grep anon
00000000014d1000 1852K rw--- [ anon ]
00007f3c3b8d2000 516K rw--- [ anon ]
--> Only 272K allocated.
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 15227 | grep anon
0000000001fba000 1852K rw--- [ anon ]
00007f38c5ef9000 516K rw--- [ anon ]
--> Memory not released.
Back to previous setting:
SQL> set arraysize 1000 rowprefetch 1
SQL> select * from big_type_small_data;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 15227 | grep anon
0000000001fba000 11644K rw--- [ anon ]
00007f38c5ef9000 516K rw--- [ anon ]
--> 9.6MB addtl memory allocated.
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 15227 | grep anon
0000000001fba000 1832K rw--- [ anon ]
00007f38c5ef9000 516K rw--- [ anon ]
--> Memory released, but not to the initial value. I.e. it seems the memory for the rowprefetch is still allocated.
Back to the settings with rowprefetch:
SQL> set arraysize 1 rowprefetch 1000
SQL> select * from big_type_small_data;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 15227 | grep anon
0000000001fba000 1832K rw--- [ anon ]
00007f38c5ef9000 516K rw--- [ anon ]
--> It obviously reused the previous memory.
SQL> set arraysize 500 rowprefetch 501 pages 503
SQL> select * from big_type_small_data;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 15227 | grep anon
0000000001fba000 6752K rw--- [ anon ]
00007f38c5ef9000 516K rw--- [ anon ]
--> Relogin
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 16334 | grep anon
00000000010b0000 1580K rw--- [ anon ]
00007ff8cc272000 516K rw--- [ anon ]
SQL> set arraysize 500 rowprefetch 501 pages 503 pause on lines 20000
SQL> select * from big_type_small_data;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 16334 | grep anon
00000000010b0000 1720K rw--- [ anon ]
00007ff8cbda4000 5436K rw--- [ anon ]
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 16334 | grep anon
00000000010b0000 1720K rw--- [ anon ]
00007ff8cc272000 516K rw--- [ anon ]
SQL> select * from big_type_small_data;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 16334 | grep anon
00000000010b0000 6608K rw--- [ anon ]
00007ff8cc272000 516K rw--- [ anon ]
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 16334 | grep anon
00000000010b0000 6608K rw--- [ anon ]
00007ff8cc272000 516K rw--- [ anon ]
--> This time the memory for the arraysize has not been released.
--> Relogin
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 17005 | grep anon
0000000001c40000 1580K rw--- [ anon ]
00007f90ea747000 516K rw--- [ anon ]
SQL> set arraysize 1 rowprefetch 32767 pages 3 pause on lines 20000
SQL> select * from big_type_small_data;
--> 2 times <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 17005 | grep anon
0000000001c40000 8312K rw--- [ anon ]
00007f90ea6a2000 1176K rw--- [ anon ]
Ctrl-C <RETURN>
oracle@oracle-19c-vagrant:/home/oracle/Blogs/rowprefetch_test/ [ORCLCDB (CDB$ROOT)] pmap 17005 | grep anon
0000000001c40000 8308K rw--- [ anon ]
00007f90ea6a2000 1176K rw--- [ anon ]
--> almost nothing released.So the tests showed that ARRAYSIZE allocates more memory than ROWPREFETCH (i.e. it allocates according the data-type-size and not according the real data in the column), but in contrast to ROWPREFETCH memory is (often) released with ARRAYSIZE once the SQL finished fetching.
Summary
So when should ROWPREFETCH and ARRAYSIZE be used? As with all fetch-size-settings (e.g. for the JDBC-driver), both can be used to reduce the number of network roundtrips and logical IOs on the DB when lots of data has to be transferred between the server and the client. According my tests ROWPREFETCH requires less memory on the client, but does not release the memory after the query has finished. ARRAYSIZE requires more memory, but often releases memory when the query has finished. ROWPREFETCH = 2 is very useful in case only 1 row is returned by a query, because it returns the row with the internal OCI execute call (first fetch in the 10046 trace) and does not require a subsequent fetch to realize that all data has been fetched already. I.e. it saves 1 network roundtrip.
A good compromise is the use of
ROWPREFETCH = 2
ARRAYSIZE = 100
That setting is actually also used when starting sqlplus with -F(AST). If lots of data has to be transferred to the client then higher ROWPREFETCH or ARRAYSIZE settings can be used to reduce the number logical IOs and network roundtrips. But the best setting also depends on the data to transfer per row and client memory requirements may vary with higher ROWPREFETCH or ARRAYSIZE settings if sqlplus runs a batch-job with many queries or only a few queries. As usual, the best setting when transferring lots of data through sqlplus has to be found by testing the queries and scripts of your environment with different settings.