3.1.3 merge into master (#14)

* bke is now developing for 3.1.3

* Replace dbms_lock.sleep with dbms_session.sleep

* Save sql_id rather than a pointer to it

* $connecterror:accept directive

* Connection storm (plus some doc updates)

* Prevent extra logon for reconnect databases

* Fix storm graph

* Allow user to set everyuntil

* Document everyuntil can be set

* Connection storm fixes and documentation

* cstorm plots both by start/end of logon

* Add ora 12510 to dead database errors

* Add oltpping

* Distinguish thread/xtra storm

* Improve plot description

* $oraerror:showoci directive

* Few missing grants/synonyms

* New aw_cols skew calculation

* printf k and K specifiers

* Fix a few printf k/K format issues

* Yet another printf k fix

* Final fixes for k/K format

* Add end2end timing for mass insert

* Fix mtitplot e2e

* e2e is logscale

* -= operator, clflags bits, nostatistics statement

* Elaborate on ?: being right associative

* Better adplot check

* Use runseconds() as fucntion call in all documentation

* Gather sys/usr from gv$osstat

* Fix png files for osstat

* Updating changelong and 3.1 news

* oltpplot --fractiles-dropped option

* $ampersand:on in histdata.rwl

* Global osstat

* Add sys% usr% to global osstat graph

* Add drop%

* Fix < 1% drops

* Use exec in shell scripts

* Remove extra -= in rwldilex

* $musymbolfromstring directive

* Add xc_hardparse to aw_transaction and awindex_query

* Add keyring to doc, also gather PGA

* Memory plot is using logscale

* Add check for time difference

* Add checkepoch.rwl

* Split long lines

* Lob trim before write

* Fix full scan message

* Improve some comments

* Changes to rwlparser.y

* Add another local only file

* Enable compile in 23, disable 18

* Use loadfactor with mtit

* Add --fromminutes option to ashplot

* Comment

* $ora01013:break directive

* Running graphs in ashplot

* Add oltpconnect

* Better RWL-322 description

* Add -uno to git status to speed up rwlwatermark.sh

* Add (experimental) hcc table to mtit

* Correct synonym

* call exec rwloadsimNN via PATH rather than full pathname

* Print proccount in mtitxcset

* Allow different index partition count for mtit

* Variable partition count

* Make sure there is no ORA-00001 risk

* Modernize sqlmonitor, sqlreport; better tags in oltp

* no dropped in stead of 0% dropped

* Fixed rwlreadlob

* Changed end-to-end colors

* Add -2 option to oltpxcrun

* Release 3.1.3

---------

Co-authored-by: Omar Bakhir <omar.bakhir@oracle.com>

Author: bengsig

.gitignore

@@ -14,6 +14,7 @@ lib/*.o
 lib/*.obj
 lib/*.lib
 dist-rwl-bin.sh
+dist-rwldev.sh
 push_to_remotes.sh
 make.generated.sh
 *.tgz

BUILDING.md

@@ -3,14 +3,14 @@
 ## Prerequisites
 
  * An Oracle client environment (release 19.9 or higher) used for building the tool; Instant Client including basiclite, sdk and sqlplus is fine.
- * Potentially client environments for other releases than the primary build release, these can similarly use full client or Instant Client; releases 11, 12, 18, 19, 21, 23 are supported.
+ * Potentially client environments for other releases than the primary build release, these can similarly use full client or Instant Client; releases 11, 12, 19, 21, 23 are supported.
  * For client release 19, you must have at least 19.9.
- * A database (release 19 or higher, release 21.3 preferred) that can be used to run test scripts.
+ * A database (release 19 or higher, release 21.9 preferred) that can be used to run test scripts.
  * An environment with the gcc compiler and tools like make.
  * Both bison (3.0.4 or newer) and flex must be installed.
  * Installing ctags and cscope is recommended if you want to work on the source.
  * The Bourne Again Shell, /bin/bash.
- * Installing gnuplot is highly recommended, although not required for compiling or running test suite. It is required for running the oltp workload.
+ * Installing gnuplot is highly recommended, although not required for compiling or running test suite. It is required for running the oltp workload and several standard utilities
  * On Solaris, you must have GNU utilities installed and they _must_ be before ordinary utilities in your PATH. Typically, this means your PATH should have /usr/gnu/bin among the first parts.
 
 ## Building rwloadsim
@@ -63,6 +63,15 @@ It contains a set of defines that must be provided for the new platform.
 At present, the following platforms are in _Beta_ state, which means they are not yet ready for production use.
 If you complete a port, please contact the rwloadsim developer.
 
+### Microsoft Windows
+
+For building on Microsoft Windows, you need the following:
+
+ * Microsoft Visual Studio 2017 or later
+ * nmake, cl, link (command line C compiler and linker)
+ * Oracle Instant client for Windows release 19.21 or newer
+ * For certain utilities (ashplot, ociping in concurrency mode), an installation of gnuplot is required
+
 ### Apple OS X
 
 For building on OS X, please be aware of the following:

CHANGELOG.md

@@ -1,5 +1,17 @@
 # RWP\*Load Simulator Release Notes
 
+## 3.1.3
+
+* Bug fixes
+* Printf format for KiB, MiB, etc
+* Oltp workload can show connection storm effects
+* Oltp workload can optionally query gv$osstat to usr/sys on db server
+* The ashplot command can output a running graph
+* Added -= assignment
+* Various improvements to statistics gathering
+* Client version 18 no longer supported
+* Client version 23 now included
+
 ## 3.1.2
 
 * Bug fixes

admin/.vim/syntax/rwl.vim

@@ -4,7 +4,7 @@
 " Copyright (c) 2023 Oracle Corporation
 " Licensed under the Universal Permissive License v 1.0
 " as shown at https://oss.oracle.com/licenses/upl/
-" Latest Revision: 28 feb 2024
+" Latest Revision:  4 jun 2024
 
 if exists("b:current_syntax")
   finish
@@ -94,6 +94,7 @@ syn region rwlSQL	start='\<\ccreate[ \t\n]\+\(or[ \t\n]\+replace[ \t\n]\+\)\?pro
 syn region rwlSQL	start='\<\ccreate[ \t\n]\+\(or[ \t\n]\+replace[ \t\n]\+\)\?package[ \t\n]' end='\(^[ \t]*[/.]$\)'
 syn region rwlSQL	start='\<\ccreate[ \t\n]\+\(or[ \t\n]\+replace[ \t\n]\+\)\?library[ \t\n]' end='\(^[ \t]*[/.]$\)'
 
+syn match rwlDirective '\$connecterror:accept'
 syn match rwlDirective '\$statistics:basic'
 syn match rwlDirective '\$statistics:histograms'
 syn match rwlDirective '\$statistics:all'
@@ -140,6 +141,10 @@ syn match rwlDirective '\$maxmesg:\d\+'
 syn match rwlDirective '\$readbuffer:\d\+'
 syn match rwlDirective '\$ora01013:stop'
 syn match rwlDirective '\$ora01013:continue'
+syn match rwlDirective '\$ora01013:break'
+syn match rwlDirective '\$ora01013:reset'
+syn match rwlDirective '\$oraerror:showoci'
+syn match rwlDirective '\$oraerror:noshowoci'
 syn match rwlDirective '\$oraerror:stop'
 syn match rwlDirective '\$oraerror:continue'
 syn match rwlDirective '\$oraerror:nocount'
@@ -191,6 +196,7 @@ syn match rwlDirective '\$bindoutname:on'
 syn match rwlDirective '\$bindoutname:on:[a-zA-Z][0-9a-zA-Z_]*'
 syn match rwlDirective '\$bindoutname:off'
 syn match rwlDirective '\$musymbol:[^ ][^ ]*'
+syn match rwlDirective '\$musymbolfromstring:[a-zA-Z][a-zA-Z0-9_]*'
 syn match rwlDirective '\$hostname:[a-zA-Z0-9_][-a-zA-Z0-9_\.]*'
 syn match rwlDirective '\$slashconvert:on'
 syn match rwlDirective '\$slashconvert:off'

admin/rwl313.sql

@@ -0,0 +1,36 @@
+-- update the RWP*Load Simulator repository
+-- for version 3.1.3
+--
+-- Copyright (c) 2023 Oracle Corporation
+-- Licensed under the Universal Permissive License v 1.0
+-- as shown at https://oss.oracle.com/licenses/upl/
+--
+-- Changes
+-- 
+-- NAME     DATE         COMMENTS
+--
+-- bengsig  26-apr-2024  osstat
+-- bengsig  19-mar-2024  Created
+--
+create table cstorm
+( runnumber number not null
+, threadnumber number not null
+, begtime number
+, endtime number
+, offtime number
+, errnum number
+, errtxt varchar2(200)
+, primary key(runnumber, threadnumber)
+) ;
+
+create table osstat
+( runnumber number not null
+, second number not null
+, usr1 number(*,6), usr2 number(*,6), usr3 number(*,6), usr4 number(*,6)
+, usr5 number(*,6), usr6 number(*,6), usr7 number(*,6), usr8 number(*,6)
+, sys1 number(*,6), sys2 number(*,6), sys3 number(*,6), sys4 number(*,6)
+, sys5 number(*,6), sys6 number(*,6), sys7 number(*,6), sys8 number(*,6)
+, primary key(runnumber, second)
+)
+/
+

admin/rwlgrants.sql

@@ -5,6 +5,9 @@
 -- as shown at https://oss.oracle.com/licenses/upl/
 
 -- History
+-- bengsig  26-apr-2024 - osstat
+-- bengsig   5-apr-2023 - oltpxc
+-- bengsig  19-mar-2023 - cstorm
 -- bengsig   8-aug-2023 - Missing grant to PERSEC
 -- bengsig         2017 - Creation
 
@@ -37,5 +40,7 @@ grant insert,select on SYSRES to rwlpublic;
 grant insert,select on OERSTATS to rwlpublic;
 grant insert,select on ashdata to rwlpublic;
 grant insert,select on rwlash to rwlpublic;
+grant insert,select on cstorm to rwlpublic;
+grant insert,update,select on oltpxc to rwlpublic;
 grant select on runnumber_seq to rwlpublic;
-
+grant insert,select on osstat to rwlpublic;

admin/rwloadsim.sql

@@ -7,6 +7,8 @@
 -- Changes
 -- 
 -- NAME     DATE         COMMENTS
+-- bengsig  26-apr-2024  osstat table
+-- bengsig  19-mar-2024  cstorm table
 -- bengsig   1-mar-2024  atime,dtime column
 -- bengsig  25-apr-2022  MTIT
 -- bengsig   1-nov-2022  Make sequence nocache to prevent many lost numbers
@@ -192,3 +194,27 @@ create table oltpxc
 , constraint oltpxc_pk primary key(key, hostname)
 )
 /
+
+create table cstorm
+( runnumber number not null
+, threadnumber number not null
+, begtime number
+, endtime number
+, offtime number
+, errnum number
+, errtxt varchar2(200)
+, primary key(runnumber, threadnumber)
+) ;
+
+-- This table stores gv$sysstat values for USER_TIME and SYS_TIME
+-- from up to 8 instances
+create table osstat
+( runnumber number not null
+, second number not null
+, usr1 number(*,6), usr2 number(*,6), usr3 number(*,6), usr4 number(*,6)
+, usr5 number(*,6), usr6 number(*,6), usr7 number(*,6), usr8 number(*,6)
+, sys1 number(*,6), sys2 number(*,6), sys3 number(*,6), sys4 number(*,6)
+, sys5 number(*,6), sys6 number(*,6), sys7 number(*,6), sys8 number(*,6)
+, primary key(runnumber, second)
+)
+/

admin/rwlsynonyms.sql

@@ -5,6 +5,8 @@
 -- as shown at https://oss.oracle.com/licenses/upl/
 
 -- History
+-- bengsig  26-apr-2024 - osstat
+-- bengsig   5-apr-2024 - cstorm, oltpxc
 -- bengsig  03-dec-2020 - Add ash 
 -- bengsig         2017 - Creation
 
@@ -24,4 +26,6 @@ create or replace synonym rwlash for rwloadsim.rwlash;
 create or replace synonym ashdata for rwloadsim.ashdata;
 create or replace synonym oerstats for rwloadsim.oerstats;
 create or replace synonym runnumber_seq for rwloadsim.runnumber_seq;
-
+create or replace synonym cstorm for rwloadsim.cstorm;
+create or replace synonym oltpxc for rwloadsim.oltpxc;
+create or replace synonym osstat for rwloadsim.osstat;

bin/ashplot

@@ -2,12 +2,13 @@
 
 # RWP*Load Simulator
 #
-# Copyright (c) 2021 Oracle Corporation
+# Copyright (c) 2024 Oracle Corporation
 # Licensed under the Universal Permissive License v 1.0
 # as shown at https://oss.oracle.com/licenses/upl/
 #
 # call rwloadsim -u ashplot.rwl
 
+# bengsig   9-may-2024 - Use exec
 # bengsig  27-nov-2023 - Creation
 
-rwloadsim --pretend-gen-banner='RWP*ASH Plot' -u ashplot.rwl "$@"
+exec rwloadsim --pretend-gen-banner='RWP*ASH Plot' -u ashplot.rwl "$@"

bin/awrdump

@@ -2,12 +2,13 @@
 
 # RWP*Load Simulator
 #
-# Copyright (c) 2021 Oracle Corporation
+# Copyright (c) 2022,2024 Oracle Corporation
 # Licensed under the Universal Permissive License v 1.0
 # as shown at https://oss.oracle.com/licenses/upl/
 #
 # call rwloadsim -u awrdump.rwl
 
+# bengsig   9-may-2024 - Use exec
 # bengsig  23-nov-2022 - Creation
 
-rwloadsim --pretend-gen-banner='RWP*AWR Dump Repository' -u awrdump.rwl "$@"
+exec rwloadsim --pretend-gen-banner='RWP*AWR Dump Repository' -u awrdump.rwl "$@"