|
1 | 1 | # mq-smf-csv |
2 | | -Simple formatter for MQ's SMF records to assist with import to spreadsheets |
| 2 | +Basic formatter for IBM MQ's z/OS SMF records that can run on workstations |
| 3 | +to simplify import of statistics to spreadsheets and databases. |
3 | 4 |
|
4 | | -README for mqsmfcsv |
5 | | -=================== |
| 5 | +Description |
| 6 | +=========== |
| 7 | +On z/OS, IBM MQ generates SMF accounting and statistics records to track |
| 8 | +the usage of the product. These are often used for capacity planning, |
| 9 | +for performance tuning, or for chargeback. |
6 | 10 |
|
7 | | -This package has been created to simplify the job |
8 | | -of doing your own analysis of MQ SMF records from a z/OS system, |
9 | | -by making it easy to import formatted records into |
10 | | -a database or spreadsheet. |
| 11 | +This package has been created to simplify the task of doing your |
| 12 | +own analysis of MQ SMF records from a z/OS system, by making it |
| 13 | +easy to import formatted records into a database or spreadsheet. |
11 | 14 |
|
12 | | -The program takes the data from a z/OS file, and runs on |
13 | | -a Windows or Unix system, creating standard CSV |
14 | | -(comma-separated value) output files. |
| 15 | +The program takes the data from a z/OS dataset, downloaded to a |
| 16 | +local file, and then runs on a Windows or Unix system, creating |
| 17 | +standard CSV (comma-separated value) output files. |
15 | 18 |
|
16 | 19 | Unlike packages such as SupportPac MP1B, there is no interpretation |
17 | 20 | at all of the SMF data. That can be done by your own spreadsheet or |
18 | 21 | database programs and macros. |
19 | 22 |
|
20 | | -This makes the code for the program |
21 | | -very easy to understand; all the structures in the SMF records |
22 | | -are printed directly. |
23 | | - |
24 | | - |
25 | | -Usage |
26 | | -===== |
27 | | -Collect the MQ SMF data in the usual way. |
28 | | -Dump that data to a z/OS file using a tool such as ... |
29 | | -Then download that file to |
30 | | - |
31 | | -The output files are written to a directory that you name |
32 | | -on the command line. There is one file for each type of |
33 | | -element - QPST, WTID etc. The files contain a line of |
34 | | -column headers, followed by the actual data with each |
35 | | -field separated by a comma. Spreadsheets such as LibreOffice |
36 | | -or Excel can then import the data directly. |
| 23 | +This makes the code for the program very easy to understand; all the |
| 24 | +structures in the SMF records are printed directly. |
37 | 25 |
|
38 | | -No attempt is made to explain the column headers; these come |
39 | | -straight from the field names. More information can be found in |
40 | | -the csqdsmfc.h header file or the product documentation. |
| 26 | +See mqsmfcsv.doc for more information on using or building this code, and |
| 27 | +thoughts on possible enhancements to the code or this repository. |
41 | 28 |
|
42 | | -Date and time are split into separate columns; the time |
43 | | -includes a microsecond portion which is separated using ',' as using '.' |
44 | | -as the decimal point seems to confuse the automatic importers on |
45 | | -several spreadsheets. There is no information directly available about |
46 | | -timezone offsets, so it is possible that the timestamps could be either |
47 | | -GMT (CUT) or localtime, depending on system configurations. |
48 | 29 |
|
49 | | -Durations are split into separate columns for |
50 | | -seconds and microseconds. This is shown in the column headings. |
51 | | - |
52 | | -Options |
| 30 | +History |
53 | 31 | ======= |
| 32 | +March 2016 - Initial release |
54 | 33 |
|
55 | | - printf("Usage: mqsmfcsv [-o <output dir>] [-a] [ -d <level> ]\n"); |
56 | | - printf(" [ -i <input file> [-m <max records>] [-r ] [-t <ticker>]\n"); |
57 | | - printf(" -a Append to existing files if they exist. Default is overwrite\n"); |
58 | | - printf(" -d <level> Debug by dumping binary records (level = 1 or 2)\n"); |
59 | | - printf(" -i <Input file> Default is to read from stdin\n"); |
60 | | - printf(" -m <Max records> End after formatting M records. Default to process all\n"); |
61 | | - printf(" -o <Directory> Where to put output files\n"); |
62 | | - printf(" -r Do not print '=' on numeric-looking strings\n"); |
63 | | - printf(" -t <Ticker> Print progress message every T records\n"); |
64 | | - |
65 | | -Design and Development |
66 | | -====================== |
67 | | - |
68 | | -Binaries are provided for some platforms. For other |
69 | | -platforms, you will need to compile the code yourself. |
70 | | -Familiarity with Makefiles and compiling C programs is |
71 | | -assumed. |
72 | | - |
73 | | -First, you need to get a copy of the MQ product-provided |
74 | | -header file on z/OS that defines the structures for SMF |
75 | | -elements. This is called csqdsmfc.h and can be found on your |
76 | | -z/OS installation. Download a copy of that to your system. |
77 | | - |
78 | | -Using the provided Makefiles as a template, set the |
79 | | -appropriate compilation options for your platform. |
80 | | -For Unix platforms, the "M" script may be a convenient |
81 | | -way to override options in the Makefile, as most Unix |
82 | | -platforms are likely to be very similar in how to build |
83 | | -programs. |
84 | | - |
85 | | -The Makefiles take the z/OS header file and make some |
86 | | -minor transformations so that it is suitable for |
87 | | -compiling on other platforms. |
88 | | - |
89 | | -Note that the mqsmfcsv program needs to be compiled with |
90 | | -fixed datatype sizes - in particular, "int" is 32-bit, and |
91 | | -"long long" is used to represent a 64-bit value. Generally |
92 | | -this means that the program itself is compiled as 32-bit. |
93 | | - |
94 | | -The current version of this program was developed based on the |
95 | | -MQ V8 SMF structures; it will run against data |
96 | | -generated by older queue managers |
97 | | -but will not compile if an older version of csqdsmfc.h is used. |
98 | | - |
99 | | -Each data structure is formatted by its own function, contained in |
100 | | -the print*.c modules. Most of these are very straightforward to |
101 | | -understand. Some common macros defined in mqsmf.h make it |
102 | | -easy to add each field from the structure; they also simplify handling |
103 | | -common fields that need to be printed on every type of SMF element. |
104 | | - |
105 | | -A few special cases exist, such as handling bitfields or arrays. But |
106 | | -again, the code where these are used ought to be obvious, and can be |
107 | | -used as strategies for handling new types of SMF element. |
108 | | - |
109 | | -Sample data |
110 | | -=========== |
111 | | -The package includes a tiny amount of sample data that can be used to |
112 | | -demonstrate the formatting. |
113 | | - |
114 | | - |
115 | | - |
116 | | -downloading ... |
117 | | - |
118 | | - |
119 | | -Sequential, Variable Length Records |
120 | | - |
121 | | -The following is a sample FTP script to retrieve (or GET) a sequential file with variable-length records from a Mainframe system to a Windows system. This FTP script is for using at the client or Windows system. |
122 | | - |
123 | | -userid |
124 | | -password |
125 | | -CD .. |
126 | | -PWD |
127 | | -BINARY |
128 | | -QUOTE SITE RDW |
129 | | -GET mainframe.dataset.name drive:\directory\filename.ext |
130 | | -QUIT |
131 | | - |
132 | | -The file must be transferred in BINARY. This will maintain the EBCDIC-encoding and the mainframe |
133 | | -numeric formats for packed-decimal, binary and zoned-decimal data. |
134 | | - |
135 | | -One of the following FTP statements must be used prior to a GET or PUT statement in order to |
136 | | -download the Record-Descriptor-Word (RDW) and the possible Block-Descriptor-Word (BDW). The BDW and |
137 | | -RDW are in binary format so it is critical to download in binary even if the records in the file |
138 | | -are all text strings. Prior to a GET use one of the following statements. |
139 | | -QUOTE SITE RDW |
140 | | - or |
141 | | -LITERAL SITE RDW |
142 | | - |
143 | | -If the FTP process is to be executed on the mainframe the GET statement will need to be replaced |
144 | | -with a PUT and the QUOTE or LITERAL statement will need to be replaced with the following statement. |
145 | 34 |
|
146 | | -LOCSITE RDW |
| 35 | +Pull requests |
| 36 | +============= |
| 37 | +Contributions to this package can be accepted under the terms of the |
| 38 | +IBM Contributor License Agreement, found in the file CLA.md of this repository. |
147 | 39 |
|
148 | | -Once the file is transferred it is a mirror of the mainframe format with EBCDIC-encoding. Each |
149 | | -record is preceded by the Record Descriptor Word (RDW) and possible Block Descriptor Word (BDW) and |
150 | | -must be converted to a Micro Focus formatted sequential file with variable length records. This |
151 | | -process is described in the following sections of this document. |
| 40 | +When submitting a pull request, you must include a statement stating you accept the terms in CLA.md. |
0 commit comments