root / plugins / emc / emc_vnx_file_ @ 17f78427
Historique | Voir | Annoter | Télécharger (27,6 ko)
| 1 |
#!/bin/bash |
|---|---|
| 2 |
|
| 3 |
: <<=cut |
| 4 |
|
| 5 |
=head1 NAME |
| 6 |
|
| 7 |
emc_vnx_file_stats - Plugin to monitor Basic, NFSv3 and NFSv4 statistics of |
| 8 |
EMC VNX 5300 Unified Storage system's Datamovers |
| 9 |
|
| 10 |
=head1 AUTHOR |
| 11 |
|
| 12 |
Evgeny Beysembaev <megabotva@gmail.com> |
| 13 |
|
| 14 |
=head1 LICENSE |
| 15 |
|
| 16 |
GPLv2 |
| 17 |
|
| 18 |
=head1 MAGIC MARKERS |
| 19 |
|
| 20 |
#%# family=auto |
| 21 |
#%# capabilities=autoconf suggest |
| 22 |
|
| 23 |
=head1 DESCRIPTION |
| 24 |
|
| 25 |
The plugin monitors basic statistics of EMC Unified Storage system Datamovers |
| 26 |
and NFS statistics of EMC VNX5300 Unified Storage system. Probably it can |
| 27 |
also be compatible with other Isilon or Celerra systems. It uses SSH to connect |
| 28 |
to Control Stations, then remotely executes '/nas/sbin/server_stats' and |
| 29 |
fetches and parses data from it. It supports gathering data both from |
| 30 |
active/active and active/passive Datamover configurations, ignoring offline or |
| 31 |
standby Datamovers. |
| 32 |
If all Datamovers are offline or absent, the plugin returns error. |
| 33 |
This plugin also automatically chooses Primary Control Station from the list by |
| 34 |
calling '/nasmcd/sbin/getreason' and '/nasmcd/sbin/t2slot'. |
| 35 |
|
| 36 |
At the moment data is gathered from the following statistics sources: |
| 37 |
* nfs.v3.op - Tons of timings about NFSv3 RPC calls |
| 38 |
* nfs.v4.op - Tons of timings about NFSv4 RPC calls |
| 39 |
* nfs.client - Here new Client addresses are rescanned and added automatically. |
| 40 |
* basic-std Statistics Group - Basic Statistics of Datamovers (eg. CPU, Memory |
| 41 |
etc.) |
| 42 |
|
| 43 |
It's quite easy to comment out unneeded data to make graphs less overloaded or |
| 44 |
to add new statistics sources. |
| 45 |
|
| 46 |
The plugin has been tested in the following Operating Environment (OE): |
| 47 |
File Version T7.1.76.4 |
| 48 |
Block Revision 05.32.000.5.215 |
| 49 |
|
| 50 |
=head1 LIST OF GRAPHS |
| 51 |
|
| 52 |
These are Basic Datamover Graphs. |
| 53 |
Graph category CPU: |
| 54 |
EMC VNX 5300 Datamover CPU Util % |
| 55 |
Graph category Network: |
| 56 |
EMC VNX 5300 Datamover Network bytes over all interfaces |
| 57 |
EMC VNX 5300 Datamover Storage bytes over all interfaces |
| 58 |
Graph category Memory: |
| 59 |
EMC VNX 5300 Datamover Memory |
| 60 |
EMC VNX 5300 File Buffer Cache |
| 61 |
EMC VNX 5300 FileResolve |
| 62 |
|
| 63 |
These are NFS (v3,v4) Graphs. |
| 64 |
Graph category FS: |
| 65 |
EMC VNX 5300 NFSv3 Calls per second |
| 66 |
EMC VNX 5300 NFSv3 uSeconds per call |
| 67 |
EMC VNX 5300 NFSv3 Op % |
| 68 |
EMC VNX 5300 NFSv4 Calls per second |
| 69 |
EMC VNX 5300 NFSv4 uSeconds per call |
| 70 |
EMC VNX 5300 NFSv4 Op % |
| 71 |
EMC VNX 5300 NFS Client Ops/s |
| 72 |
EMC VNX 5300 NFS Client B/s |
| 73 |
EMC VNX 5300 NFS Client Avg uSec/call |
| 74 |
EMC VNX 5300 Std NFS Ops/s |
| 75 |
EMC VNX 5300 Std NFS B/s |
| 76 |
EMC VNX 5300 Std NFS Average Size Bytes |
| 77 |
EMC VNX 5300 Std NFS Active Threads |
| 78 |
|
| 79 |
=head1 COMPATIBILITY |
| 80 |
|
| 81 |
The plugin has been written for being compatible with EMC VNX5300 Storage |
| 82 |
system, as this is the only EMC storage which i have. |
| 83 |
By the way, i am pretty sure it can also work with other VNX1 storages, like |
| 84 |
VNX5100 and VNX5500. |
| 85 |
About VNX2 series, i don't know whether the plugin will be able to work with |
| 86 |
them. Maybe it would need some corrections in command-line backend. The same |
| 87 |
situation is with other EMC systems, so i encourage you to try and fix the |
| 88 |
plugin. |
| 89 |
|
| 90 |
=head1 CONFIGURATION |
| 91 |
|
| 92 |
The plugin uses SSH to connect to Control Stations. It's possible to use |
| 93 |
'nasadmin' user, but it would be better if you create read-only global user by |
| 94 |
Unisphere Client. The user should have only Operator role. |
| 95 |
I created "operator" user but due to the fact that Control Stations already |
| 96 |
had one internal "operator" user, the new one was called "operator1". So be |
| 97 |
careful. After that, copy .bash_profile from /home/nasadmin to a newly created |
| 98 |
/home/operator1 |
| 99 |
|
| 100 |
On munin-node side choose a user which will be used to connect through SSH. |
| 101 |
Generally user "munin" is ok. Then, execute "sudo su munin -s /bin/bash", |
| 102 |
"ssh-keygen" and "ssh-copy-id" to both Control Stations with newly created |
| 103 |
user. |
| 104 |
|
| 105 |
Make a link from /usr/share/munin/plugins/emc_vnx_file_stats to |
| 106 |
/etc/munin/plugins/. If you want to get NFS statistics, name the link as |
| 107 |
"emc_vnx_file_nfs_stats_<NAME>", otherwise to get Basic Datamover statistics |
| 108 |
you have to name it "emc_vnx_file_basicdm_stats_<NAME>", where <NAME> is any |
| 109 |
arbitrary name of your storage system. The plugin will return <NAME> in its |
| 110 |
answer as "host_name" field. |
| 111 |
|
| 112 |
For example, assume your storage system is called "VNX5300". |
| 113 |
Make a configuration file at |
| 114 |
/etc/munin/plugin-conf.d/emc_vnx_file_stats_VNX5300 |
| 115 |
|
| 116 |
[emc_vnx_file_*] |
| 117 |
user munin |
| 118 |
env.username operator1 |
| 119 |
env.cs_addr 192.168.1.1 192.168.1.2 |
| 120 |
env.nas_servers server_2 server_3 |
| 121 |
|
| 122 |
Where: |
| 123 |
user - SSH Client local user |
| 124 |
env.username - Remote user with Operator role |
| 125 |
env.cs_addr - Control Stations addresses |
| 126 |
env.nas_servers - This is the default value and can be omitted |
| 127 |
|
| 128 |
=head1 HISTORY |
| 129 |
|
| 130 |
08.11.2016 - First Release |
| 131 |
17.11.2016 - NFSv4 support, Memory section |
| 132 |
16.12.2016 - Merged "NFS" and "Datamover Stats" plugins |
| 133 |
26.12.2016 - Compatibility with Munin coding style |
| 134 |
|
| 135 |
=cut |
| 136 |
|
| 137 |
export LANG=C |
| 138 |
|
| 139 |
. "$MUNIN_LIBDIR/plugins/plugin.sh" |
| 140 |
|
| 141 |
nas_server_ok="" |
| 142 |
cs_addr=${cs_addr:=""}
|
| 143 |
username=${username:=""}
|
| 144 |
nas_servers=${nas_servers:="server_2 server_3"}
|
| 145 |
|
| 146 |
# Prints "10" on stdout if found Primary Online control station. "11" - for Secondary Online control station. |
| 147 |
ssh_check_cmd() {
|
| 148 |
ssh -q "$username@$1" "/nasmcd/sbin/getreason | grep -w \"slot_\$(/nasmcd/sbin/t2slot)\" | cut -d- -f1 | awk '{print \$1}' "
|
| 149 |
|
| 150 |
} |
| 151 |
|
| 152 |
check_conf () {
|
| 153 |
if [ -z "$username" ]; then |
| 154 |
echo "No username ('username' environment variable)!"
|
| 155 |
return 1 |
| 156 |
fi |
| 157 |
|
| 158 |
if [ -z "$cs_addr" ]; then |
| 159 |
echo "No control station addresses ('cs_addr' environment variable)!"
|
| 160 |
return 1 |
| 161 |
fi |
| 162 |
|
| 163 |
#Choosing Cotrol Station. Code have to be "10" |
| 164 |
for CS in $cs_addr; do |
| 165 |
if [[ "10" = "$(ssh_check_cmd "$CS")" ]]; then |
| 166 |
PRIMARY_CS=$CS |
| 167 |
break |
| 168 |
fi |
| 169 |
done |
| 170 |
|
| 171 |
if [ -z "$PRIMARY_CS" ]; then |
| 172 |
echo "No alive primary Control Station from list \"$cs_addr\""; |
| 173 |
return 1 |
| 174 |
fi |
| 175 |
return 0 |
| 176 |
} |
| 177 |
|
| 178 |
if [ "$1" = "autoconf" ]; then |
| 179 |
check_conf_ans=$(check_conf) |
| 180 |
if [ $? -eq 0 ]; then |
| 181 |
echo "yes" |
| 182 |
else |
| 183 |
echo "no ($check_conf_ans)" |
| 184 |
fi |
| 185 |
exit 0 |
| 186 |
fi |
| 187 |
|
| 188 |
if [ "$1" = "suggest" ]; then |
| 189 |
echo "nfs_stats" |
| 190 |
echo "basicdm_stats" |
| 191 |
exit 0; |
| 192 |
fi |
| 193 |
|
| 194 |
STATSTYPE=$(echo "${0##*/}" | cut -d _ -f 1-5)
|
| 195 |
if [ "$STATSTYPE" = "emc_vnx_file_nfs_stats" ]; then STATSTYPE=NFS; |
| 196 |
elif [ "$STATSTYPE" = "emc_vnx_file_basicdm_stats" ]; then STATSTYPE=BASICDM; |
| 197 |
else echo "Do not know what to do. Name the plugin as 'emc_vnx_file_nfs_stats_<HOSTNAME>' or 'emc_vnx_file_basicdm_stats_<HOSTNAME>'" >&2; exit 1; fi |
| 198 |
|
| 199 |
TARGET=$(echo "${0##*/}" | cut -d _ -f 6)
|
| 200 |
|
| 201 |
check_conf 1>&2 || exit 1 |
| 202 |
|
| 203 |
run_remote () {
|
| 204 |
# shellcheck disable=SC2029 |
| 205 |
ssh -q "$username@$PRIMARY_CS" ". /home/$username/.bash_profile; $*" |
| 206 |
} |
| 207 |
|
| 208 |
echo "host_name ${TARGET}"
|
| 209 |
|
| 210 |
if [ "$1" = "config" ] ; then |
| 211 |
# TODO: active/active |
| 212 |
for server in $nas_servers; do |
| 213 |
run_remote nas_server -i "$server" | grep -q 'type *= nas' || continue |
| 214 |
nas_server_ok=TRUE |
| 215 |
filtered_server="$(clean_fieldname "$server")" |
| 216 |
|
| 217 |
if [ "$STATSTYPE" = "BASICDM" ] ; then |
| 218 |
cat <<-EOF |
| 219 |
multigraph emc_vnx_cpu_percent |
| 220 |
graph_title EMC VNX 5300 Datamover CPU Util % |
| 221 |
graph_vlabel % |
| 222 |
graph_category cpu |
| 223 |
graph_scale no |
| 224 |
graph_args --upper-limit 100 -l 0 |
| 225 |
${server}_cpuutil.min 0
|
| 226 |
${server}_cpuutil.label $server CPU util. in %.
|
| 227 |
|
| 228 |
multigraph emc_vnx_network_b |
| 229 |
graph_title EMC VNX 5300 Datamover Network bytes over all interfaces |
| 230 |
graph_vlabel B/s recv. (-) / sent (+) |
| 231 |
graph_category network |
| 232 |
graph_args --base 1000 |
| 233 |
${server}_net_in.graph no
|
| 234 |
${server}_net_in.label none
|
| 235 |
${server}_net_out.label $server B/s
|
| 236 |
${server}_net_out.negative ${server}_net_in
|
| 237 |
${server}_net_out.draw AREA
|
| 238 |
|
| 239 |
multigraph emc_vnx_storage_b |
| 240 |
graph_title EMC VNX 5300 Datamover Storage bytes over all interfaces |
| 241 |
graph_vlabel B/s recv. (-) / sent (+) |
| 242 |
graph_category network |
| 243 |
graph_args --base 1000 |
| 244 |
${server}_stor_read.graph no
|
| 245 |
${server}_stor_read.label none
|
| 246 |
${server}_stor_write.label $server B/s
|
| 247 |
${server}_stor_write.negative ${server}_stor_read
|
| 248 |
${server}_stor_write.draw AREA
|
| 249 |
|
| 250 |
multigraph emc_vnx_memory |
| 251 |
graph_title EMC VNX 5300 Datamover Memory |
| 252 |
graph_vlabel KiB |
| 253 |
graph_category memory |
| 254 |
graph_args --base 1024 |
| 255 |
graph_order ${server}_used ${server}_free ${server}_total ${server}_freebuffer ${server}_encumbered
|
| 256 |
${server}_used.label ${server} Used
|
| 257 |
${server}_free.label ${server} Free
|
| 258 |
${server}_free.draw STACK
|
| 259 |
${server}_total.label ${server} Total
|
| 260 |
${server}_freebuffer.label ${server} Free Buffer
|
| 261 |
${server}_encumbered.label ${server} Encumbered
|
| 262 |
|
| 263 |
multigraph emc_vnx_filecache |
| 264 |
graph_title EMC VNX 5300 File Buffer Cache |
| 265 |
graph_vlabel per second |
| 266 |
graph_category memory |
| 267 |
graph_args --base 1000 |
| 268 |
graph_order ${server}_highw_hits ${server}_loww_hits ${server}_w_hits ${server}_hits ${server}_lookups
|
| 269 |
${server}_highw_hits.label High Watermark Hits
|
| 270 |
${server}_loww_hits.label Low Watermark Hits
|
| 271 |
${server}_loww_hits.draw STACK
|
| 272 |
${server}_w_hits.label Watermark Hits
|
| 273 |
${server}_hits.label Hits
|
| 274 |
${server}_lookups.label Lookups
|
| 275 |
|
| 276 |
multigraph emc_vnx_fileresolve |
| 277 |
graph_title EMC VNX 5300 FileResolve |
| 278 |
graph_vlabel Entries |
| 279 |
graph_category memory |
| 280 |
graph_args --base 1000 |
| 281 |
${server}_dropped.label Dropped Entries
|
| 282 |
${server}_max.label Max Limit
|
| 283 |
${server}_used.label Used Entries
|
| 284 |
EOF |
| 285 |
fi |
| 286 |
if [ "$STATSTYPE" = "NFS" ] ; then |
| 287 |
#nfs.v3.op data |
| 288 |
# [nasadmin@mnemonic0 ~]$ server_stats server_2 -info nfs.v3.op |
| 289 |
# server_2 : |
| 290 |
# |
| 291 |
# name = nfs.v3.op |
| 292 |
# description = NFS V3 per operation statistics |
| 293 |
# type = Set |
| 294 |
# member_stats = nfs.v3.op.ALL-ELEMENTS.calls,nfs.v3.op.ALL-ELEMENTS.failures,nfs.v3.op.ALL-ELEMENTS.avgTime,nfs.v3.op.ALL-ELEMENTS.opPct |
| 295 |
# member_elements = nfs.v3.op.v3Null,nfs.v3.op.v3GetAttr,nfs.v3.op.v3SetAttr,nfs.v3.op.v3Lookup,nfs.v3.op.v3Access,nfs.v3.op.v3ReadLink,nfs.v3.op.v3Read,nfs.v3.op.v3Write,nfs.v3.op.v3Create,nfs.v3.op.v3Mkdir,nfs.v3.op.v3Symlink,nfs.v3.op.v3Mknod,nfs.v3.op.v3Remove,nfs.v3.op.v3Rmdir,nfs.v3.op.v3Rename,nfs.v3.op.v3Link,nfs.v3.op.v3ReadDir,nfs.v3.op.v3ReadDirPlus,nfs.v3.op.v3FsStat,nfs.v3.op.v3FsInfo,nfs.v3.op.v3PathConf,nfs.v3.op.v3Commit,nfs.v3.op.VAAI |
| 296 |
# member_of = nfs.v3 |
| 297 |
member_elements_by_line=$(run_remote server_stats "$server" -info nfs.v3.op | grep member_elements | sed -ne 's/^.*= //p') |
| 298 |
IFS=',' read -ra graphs <<< "$member_elements_by_line" |
| 299 |
cat <<-EOF |
| 300 |
multigraph vnx_emc_v3_calls_s |
| 301 |
graph_title EMC VNX 5300 NFSv3 Calls per second |
| 302 |
graph_vlabel Calls |
| 303 |
graph_category fs |
| 304 |
graph_args --base 1000 |
| 305 |
EOF |
| 306 |
for graph in "${graphs[@]}"; do
|
| 307 |
field=$(echo "$graph" | cut -d '.' -f4 ) |
| 308 |
echo "${server}_$field.label $server $field"
|
| 309 |
done |
| 310 |
|
| 311 |
cat <<-EOF |
| 312 |
|
| 313 |
multigraph vnx_emc_v3_usec_call |
| 314 |
graph_title EMC VNX 5300 NFSv3 uSeconds per call |
| 315 |
graph_vlabel uSec / call |
| 316 |
graph_category fs |
| 317 |
graph_args --base 1000 |
| 318 |
EOF |
| 319 |
for graph in "${graphs[@]}"; do
|
| 320 |
field=$(echo "$graph" | cut -d '.' -f4 ) |
| 321 |
echo "${server}_$field.label $server $field"
|
| 322 |
done |
| 323 |
cat <<-EOF |
| 324 |
|
| 325 |
multigraph vnx_emc_v3_op_percent |
| 326 |
graph_title EMC VNX 5300 NFSv3 Op % |
| 327 |
graph_vlabel % |
| 328 |
graph_scale no |
| 329 |
graph_category fs |
| 330 |
EOF |
| 331 |
for graph in "${graphs[@]}"; do
|
| 332 |
field=$(echo "$graph" | cut -d '.' -f4 ) |
| 333 |
echo "${server}_$field.label $server $field"
|
| 334 |
echo "${server}_$field.min 0"
|
| 335 |
done |
| 336 |
graphs=() |
| 337 |
#nfs.v4.op data |
| 338 |
member_elements_by_line=$(run_remote server_stats "$server" -info nfs.v4.op | grep member_elements | sed -ne 's/^.*= //p') |
| 339 |
IFS=',' read -ra graphs <<< "$member_elements_by_line" |
| 340 |
cat <<-EOF |
| 341 |
multigraph vnx_emc_v4_calls_s |
| 342 |
graph_title EMC VNX 5300 NFSv4 Calls per second |
| 343 |
graph_vlabel Calls |
| 344 |
graph_category fs |
| 345 |
graph_args --base 1000 |
| 346 |
EOF |
| 347 |
for graph in "${graphs[@]}"; do
|
| 348 |
field=$(echo "$graph" | cut -d '.' -f4 ) |
| 349 |
echo "${server}_$field.label $server $field"
|
| 350 |
done |
| 351 |
|
| 352 |
cat <<-EOF |
| 353 |
|
| 354 |
multigraph vnx_emc_v4_usec_call |
| 355 |
graph_title EMC VNX 5300 NFSv4 uSeconds per call |
| 356 |
graph_vlabel uSec / call |
| 357 |
graph_category fs |
| 358 |
graph_args --base 1000 |
| 359 |
EOF |
| 360 |
for graph in "${graphs[@]}"; do
|
| 361 |
field=$(echo "$graph" | cut -d '.' -f4 ) |
| 362 |
echo "${server}_$field.label $server $field"
|
| 363 |
done |
| 364 |
cat <<-EOF |
| 365 |
|
| 366 |
multigraph vnx_emc_v4_op_percent |
| 367 |
graph_title EMC VNX 5300 NFSv4 Op % |
| 368 |
graph_vlabel % |
| 369 |
graph_scale no |
| 370 |
graph_category fs |
| 371 |
EOF |
| 372 |
for graph in "${graphs[@]}"; do
|
| 373 |
field=$(echo "$graph" | cut -d '.' -f4 ) |
| 374 |
echo "${server}_$field.label $server $field"
|
| 375 |
echo "${server}_$field.min 0"
|
| 376 |
done |
| 377 |
|
| 378 |
#nfs.client data |
| 379 |
# Total Read Write Suspicious Total Read Write Avg |
| 380 |
# Ops/s Ops/s Ops/s Ops diff KiB/s KiB/s KiB/s uSec/call |
| 381 |
member_elements_by_line=$(run_remote server_stats server_2 -monitor nfs.client -count 1 -terminationsummary no -titles never | sed -ne 's/^.*id=//p' | cut -d' ' -f1) |
| 382 |
#Somewhy readarray adds extra \n in the end of each variable. So, we use read() with a workaround |
| 383 |
IFS=$'\n' read -rd '' -a graphs_array <<< "$member_elements_by_line" |
| 384 |
cat <<-EOF |
| 385 |
|
| 386 |
multigraph vnx_emc_nfs_client_ops_s |
| 387 |
graph_title EMC VNX 5300 NFS Client Ops/s |
| 388 |
graph_vlabel Ops/s |
| 389 |
graph_category fs |
| 390 |
EOF |
| 391 |
echo -n "graph_order " |
| 392 |
for graph in "${graphs_array[@]}"; do
|
| 393 |
field="$(clean_fieldname "_$graph")" |
| 394 |
echo -n "${server}${field}_r ${server}${field}_w ${server}${field}_t ${server}${field}_s "
|
| 395 |
done |
| 396 |
echo " " |
| 397 |
for graph in "${graphs_array[@]}"; do
|
| 398 |
field="$(clean_fieldname "_$graph")" |
| 399 |
echo "${server}${field}_r.label $server $graph Read Ops/s"
|
| 400 |
echo "${server}${field}_w.label $server $graph Write Ops/s"
|
| 401 |
echo "${server}${field}_w.draw STACK"
|
| 402 |
echo "${server}${field}_t.label $server $graph Total Ops/s"
|
| 403 |
echo "${server}${field}_s.label $server $graph Suspicious Ops diff"
|
| 404 |
done |
| 405 |
|
| 406 |
cat <<-EOF |
| 407 |
|
| 408 |
multigraph vnx_emc_nfs_client_b_s |
| 409 |
graph_title EMC VNX 5300 NFS Client B/s |
| 410 |
graph_vlabel B/s |
| 411 |
graph_category fs |
| 412 |
EOF |
| 413 |
echo -n "graph_order " |
| 414 |
for graph in "${graphs_array[@]}"; do
|
| 415 |
field="$(clean_fieldname "_$graph")" |
| 416 |
echo -n "${server}${field}_r ${server}${field}_w ${server}${field}_t "
|
| 417 |
done |
| 418 |
echo " " |
| 419 |
for graph in "${graphs_array[@]}"; do
|
| 420 |
field="$(clean_fieldname "_$graph")" |
| 421 |
echo "${server}${field}_r.label $server $graph Read B/s"
|
| 422 |
echo "${server}${field}_w.label $server $graph Write B/s"
|
| 423 |
echo "${server}${field}_w.draw STACK"
|
| 424 |
echo "${server}${field}_t.label $server $graph Total B/s"
|
| 425 |
done |
| 426 |
|
| 427 |
cat <<-EOF |
| 428 |
|
| 429 |
multigraph vnx_emc_nfs_client_avg_usec |
| 430 |
graph_title EMC VNX 5300 NFS Client Avg uSec/call |
| 431 |
graph_vlabel uSec/call |
| 432 |
graph_category fs |
| 433 |
EOF |
| 434 |
for graph in "${graphs_array[@]}"; do
|
| 435 |
field="$(clean_fieldname "_$graph")" |
| 436 |
echo "${server}${field}.label $server $graph Avg uSec/call"
|
| 437 |
done |
| 438 |
|
| 439 |
#nfs-std |
| 440 |
# Timestamp NFS Read Read Read Size Write Write Write Size Active |
| 441 |
# Ops/s Ops/s KiB/s Bytes Ops/s KiB/s Bytes Threads |
| 442 |
cat <<-EOF |
| 443 |
|
| 444 |
multigraph vnx_emc_nfs_std_nfs_ops |
| 445 |
graph_title EMC VNX 5300 Std NFS Ops/s |
| 446 |
graph_vlabel Ops/s |
| 447 |
graph_category fs |
| 448 |
EOF |
| 449 |
echo "graph_order ${filtered_server}_rops ${filtered_server}_wops ${filtered_server}_tops"
|
| 450 |
echo "${filtered_server}_rops.label $server Read Ops/s"
|
| 451 |
echo "${filtered_server}_wops.label $server Write Ops/s"
|
| 452 |
echo "${filtered_server}_wops.draw STACK"
|
| 453 |
echo "${filtered_server}_tops.label $server Total Ops/s"
|
| 454 |
|
| 455 |
cat <<-EOF |
| 456 |
|
| 457 |
multigraph vnx_emc_nfs_std_nfs_b_s |
| 458 |
graph_title EMC VNX 5300 Std NFS B/s |
| 459 |
graph_vlabel B/s |
| 460 |
graph_category fs |
| 461 |
EOF |
| 462 |
echo "graph_order ${filtered_server}_rbs ${filtered_server}_wbs ${filtered_server}_tbs"
|
| 463 |
echo "${filtered_server}_rbs.label $server Read B/s"
|
| 464 |
echo "${filtered_server}_wbs.label $server Write B/s"
|
| 465 |
echo "${filtered_server}_wbs.draw STACK"
|
| 466 |
echo "${filtered_server}_tbs.label $server Total B/s"
|
| 467 |
echo "${filtered_server}_tbs.cdef ${filtered_server}_rbs,${filtered_server}_wbs,+"
|
| 468 |
|
| 469 |
cat <<-EOF |
| 470 |
|
| 471 |
multigraph vnx_emc_nfs_std_nfs_avg |
| 472 |
graph_title EMC VNX 5300 Std NFS Average Size Bytes |
| 473 |
graph_vlabel Bytes |
| 474 |
graph_category fs |
| 475 |
EOF |
| 476 |
echo "${filtered_server}_avg_readsize.label $server Average Read Size Bytes"
|
| 477 |
echo "${filtered_server}_avg_writesize.label $server Average Write Size Bytes"
|
| 478 |
|
| 479 |
cat <<-EOF |
| 480 |
|
| 481 |
multigraph vnx_emc_nfs_std_nfs_threads |
| 482 |
graph_title EMC VNX 5300 Std NFS Active Threads |
| 483 |
graph_vlabel Threads |
| 484 |
graph_category fs |
| 485 |
EOF |
| 486 |
echo "${filtered_server}_threads.label $server Active Threads"
|
| 487 |
fi |
| 488 |
done |
| 489 |
if [ -z "$nas_server_ok" ]; then |
| 490 |
echo "No active data movers!" 1>&2 |
| 491 |
fi |
| 492 |
exit 0 |
| 493 |
fi |
| 494 |
|
| 495 |
for server in $nas_servers; do |
| 496 |
run_remote nas_server -i "$server" | grep -q 'type *= nas' || continue |
| 497 |
nas_server_ok=TRUE |
| 498 |
filtered_server="$(clean_fieldname "$server")" |
| 499 |
|
| 500 |
if [ "$STATSTYPE" = "BASICDM" ] ; then |
| 501 |
#basicdm data |
| 502 |
# [nasadmin@mnemonic0 ~]$ server_stats server_2 -count 1 -terminationsummary no |
| 503 |
# server_2 CPU Network Network dVol dVol |
| 504 |
# Timestamp Util In Out Read Write |
| 505 |
# % KiB/s KiB/s KiB/s KiB/s |
| 506 |
# 20:42:26 9 16432 3404 1967 24889 |
| 507 |
|
| 508 |
member_elements_by_line=$(run_remote server_stats "$server" -count 1 -terminationsummary no -titles never | grep '^[^[:space:]]') |
| 509 |
IFS=$' ' read -ra graphs <<< "$member_elements_by_line" |
| 510 |
|
| 511 |
echo "multigraph emc_vnx_cpu_percent" |
| 512 |
echo "${server}_cpuutil.value ${graphs[1]}"
|
| 513 |
|
| 514 |
echo -e "\nmultigraph emc_vnx_network_b" |
| 515 |
echo "${server}_net_in.value $((graphs[2] * 1024))"
|
| 516 |
echo "${server}_net_out.value $((graphs[3] * 1024))"
|
| 517 |
|
| 518 |
echo -e "\nmultigraph emc_vnx_storage_b" |
| 519 |
echo "${server}_stor_read.value $((graphs[4] * 1024))"
|
| 520 |
echo "${server}_stor_write.value $((graphs[5] * 1024))"
|
| 521 |
|
| 522 |
# [nasadmin@mnemonic0 ~]$ server_stats server_2 -monitor kernel.memory -count 1 -terminationsummary no |
| 523 |
# server_2 Free Buffer Buffer Buffer Buffer Buffer Buffer Cache Encumbered FileResolve FileResolve FileResolve Free KiB Page Total Used KiB Memory |
| 524 |
# Timestamp Buffer Cache High Cache Cache Cache Cache Low Watermark Memory Dropped Max Used Size Memory Util |
| 525 |
# KiB Watermark Hits/s Hit % Hits/s Lookups/s Watermark Hits/s Hits/s KiB Entries Limit Entries KiB KiB % |
| 526 |
# 20:44:14 3522944 0 96 11562 12010 0 0 3579268 0 0 0 3525848 8 6291456 2765608 44 |
| 527 |
|
| 528 |
member_elements_by_line=$(run_remote server_stats "$server" -monitor kernel.memory -count 1 -terminationsummary no -titles never | grep '^[^[:space:]]') |
| 529 |
IFS=$' ' read -ra graphs <<< "$member_elements_by_line" |
| 530 |
|
| 531 |
echo -e "\nmultigraph emc_vnx_memory" |
| 532 |
#Reserved for math |
| 533 |
echo "${server}_total.value $((graphs[14] / 1))"
|
| 534 |
echo "${server}_used.value $((graphs[15] / 1))"
|
| 535 |
echo "${server}_free.value $((graphs[12] / 1))"
|
| 536 |
echo "${server}_freebuffer.value $((graphs[1] / 1))"
|
| 537 |
echo "${server}_encumbered.value $((graphs[8] / 1))"
|
| 538 |
|
| 539 |
echo -e "\nmultigraph emc_vnx_filecache" |
| 540 |
echo "${server}_highw_hits.value ${graphs[2]}"
|
| 541 |
echo "${server}_loww_hits.value ${graphs[6]}"
|
| 542 |
echo "${server}_w_hits.value ${graphs[7]}"
|
| 543 |
echo "${server}_hits.value ${graphs[4]}"
|
| 544 |
echo "${server}_lookups.value ${graphs[5]}"
|
| 545 |
|
| 546 |
echo -e "\nmultigraph emc_vnx_fileresolve" |
| 547 |
echo "${server}_dropped.value ${graphs[9]}"
|
| 548 |
echo "${server}_max.value ${graphs[10]}"
|
| 549 |
echo "${server}_used.value ${graphs[11]}"
|
| 550 |
|
| 551 |
|
| 552 |
fi |
| 553 |
if [ "$STATSTYPE" = "NFS" ] ; then |
| 554 |
#nfs.v3.op data |
| 555 |
# [nasadmin@mnemonic0 ~]$ server_stats server_2 -monitor nfs.v3.op -count 1 -terminationsummary no |
| 556 |
# server_2 NFS Op NFS NFS Op NFS NFS Op % |
| 557 |
# Timestamp Op Errors Op |
| 558 |
# Calls/s diff uSec/Call |
| 559 |
# 22:14:41 v3GetAttr 30 0 23 21 |
| 560 |
# v3Lookup 40 0 98070 27 |
| 561 |
# v3Access 50 0 20 34 |
| 562 |
# v3Read 4 0 11180 3 |
| 563 |
# v3Write 2 0 2334 1 |
| 564 |
# v3Create 1 0 1743 1 |
| 565 |
# v3Mkdir 13 0 953 9 |
| 566 |
# v3Link 6 0 1064 4 |
| 567 |
|
| 568 |
member_elements_by_line=$(run_remote server_stats "$server" -monitor nfs.v3.op -count 1 -terminationsummary no -titles never | sed -ne 's/^.*v3/v3/p') |
| 569 |
NUMCOL=5 |
| 570 |
LINES=$(wc -l <<< "$member_elements_by_line") |
| 571 |
while IFS=$'\n' read -ra graphs ; do |
| 572 |
elements_array+=( $graphs ) |
| 573 |
done <<< "$member_elements_by_line" |
| 574 |
|
| 575 |
if [ "${#elements_array[@]}" -eq "0" ]; then LINES=0; fi
|
| 576 |
|
| 577 |
echo "multigraph vnx_emc_v3_calls_s" |
| 578 |
for ((i=0; i<$((LINES)); i++ )); do |
| 579 |
echo "${server}_${elements_array[i*$NUMCOL]}".value "${elements_array[i*$NUMCOL+1]}"
|
| 580 |
done |
| 581 |
|
| 582 |
echo -e "\nmultigraph vnx_emc_v3_usec_call" |
| 583 |
for ((i=0; i<$((LINES)); i++ )); do |
| 584 |
echo "${server}_${elements_array[i*$NUMCOL]}".value "${elements_array[i*$NUMCOL+3]}"
|
| 585 |
done |
| 586 |
|
| 587 |
echo -e "\nmultigraph vnx_emc_v3_op_percent" |
| 588 |
for ((i=0; i<$((LINES)); i++ )); do |
| 589 |
echo "${server}_${elements_array[i*$NUMCOL]}".value "${elements_array[i*$NUMCOL+4]}"
|
| 590 |
done |
| 591 |
|
| 592 |
elements_array=() |
| 593 |
|
| 594 |
#nfs.v4.op data |
| 595 |
# [nasadmin@mnemonic0 ~]$ server_stats server_2 -monitor nfs.v4.op -count 1 -terminationsummary no |
| 596 |
# server_2 NFS Op NFS NFS Op NFS NFS Op % |
| 597 |
# Timestamp Op Errors Op |
| 598 |
# Calls/s diff uSec/Call |
| 599 |
# 22:13:14 v4Compound 2315 0 7913 30 |
| 600 |
# v4Access 246 0 5 3 |
| 601 |
# v4Close 133 0 11 2 |
| 602 |
# v4Commit 2 0 6928 0 |
| 603 |
# v4Create 1 0 881 0 |
| 604 |
# v4DelegRet 84 0 19 1 |
| 605 |
# v4GetAttr 1330 0 7 17 |
| 606 |
# v4GetFh 164 0 3 2 |
| 607 |
# v4Lookup 68 0 43 1 |
| 608 |
# v4Open 132 0 1061 2 |
| 609 |
# v4PutFh 2314 0 11 30 |
| 610 |
# v4Read 359 0 15561 5 |
| 611 |
# v4ReadDir 1 0 37 0 |
| 612 |
# v4Remove 62 0 1096 1 |
| 613 |
# v4Rename 1 0 947 0 |
| 614 |
# v4Renew 2 0 3 0 |
| 615 |
# v4SaveFh 1 0 3 0 |
| 616 |
# v4SetAttr 9 0 889 0 |
| 617 |
# v4Write 525 0 16508 7 |
| 618 |
|
| 619 |
member_elements_by_line=$(run_remote server_stats "$server" -monitor nfs.v4.op -count 1 -terminationsummary no -titles never | sed -ne 's/^.*v4/v4/p') |
| 620 |
NUMCOL=5 |
| 621 |
LINES=$(wc -l <<< "$member_elements_by_line") |
| 622 |
while IFS=$'\n' read -ra graphs ; do |
| 623 |
elements_array+=( $graphs ) |
| 624 |
done <<< "$member_elements_by_line" |
| 625 |
|
| 626 |
if [ "${#elements_array[@]}" -eq "0" ]; then LINES=0; fi
|
| 627 |
|
| 628 |
echo -e "\nmultigraph vnx_emc_v4_calls_s" |
| 629 |
for ((i=0; i<$((LINES)); i++ )); do |
| 630 |
echo "${server}_${elements_array[i*$NUMCOL]}".value "${elements_array[i*$NUMCOL+1]}"
|
| 631 |
done |
| 632 |
|
| 633 |
echo -e "\nmultigraph vnx_emc_v4_usec_call" |
| 634 |
for ((i=0; i<$((LINES)); i++ )); do |
| 635 |
echo "${server}_${elements_array[i*$NUMCOL]}".value "${elements_array[i*$NUMCOL+3]}"
|
| 636 |
done |
| 637 |
|
| 638 |
echo -e "\nmultigraph vnx_emc_v4_op_percent" |
| 639 |
for ((i=0; i<$((LINES)); i++ )); do |
| 640 |
echo "${server}_${elements_array[i*$NUMCOL]}".value "${elements_array[i*$NUMCOL+4]}"
|
| 641 |
done |
| 642 |
|
| 643 |
elements_array=() |
| 644 |
|
| 645 |
#nfs.client data |
| 646 |
# [nasadmin@mnemonic0 ~]$ server_stats server_2 -monitor nfs.client -count 1 -terminationsummary no |
| 647 |
# server_2 Client NFS NFS NFS NFS NFS NFS NFS NFS |
| 648 |
# Timestamp Total Read Write Suspicious Total Read Write Avg |
| 649 |
# Ops/s Ops/s Ops/s Ops diff KiB/s KiB/s KiB/s uSec/call |
| 650 |
# 20:26:38 id=192.168.1.223 2550 20 2196 13 4673 159 4514 1964 |
| 651 |
# id=192.168.1.2 691 4 5 1 1113 425 688 2404 |
| 652 |
# id=192.168.1.1 159 0 0 51 0 0 0 6017 |
| 653 |
# id=192.168.1.6 37 4 2 0 586 295 291 5980 |
| 654 |
# id=192.168.1.235 21 1 0 0 0 0 0 155839 |
| 655 |
# id=192.168.1.224 5 0 5 0 20 0 20 704620 |
| 656 |
|
| 657 |
member_elements_by_line=$(run_remote server_stats server_2 -monitor nfs.client -count 1 -terminationsummary no -titles never | sed -ne 's/^.*id=//p') |
| 658 |
echo -e "\nmultigraph vnx_emc_nfs_client_ops_s" |
| 659 |
NUMCOL=9 |
| 660 |
LINES=$(wc -l <<< "$member_elements_by_line") |
| 661 |
while IFS=$'\n' read -ra graphs; do |
| 662 |
elements_array+=($graphs) |
| 663 |
done <<< "$member_elements_by_line" |
| 664 |
|
| 665 |
#Not drawing elements in case of empty set |
| 666 |
if [ "${#elements_array[@]}" -eq "0" ]; then LINES=0; fi
|
| 667 |
|
| 668 |
for (( i=0; i<$((LINES)); i++ )); do |
| 669 |
client="$(clean_fieldname "_${elements_array[i*$NUMCOL]}")"
|
| 670 |
echo "${server}${client}_r".value "${elements_array[$i*$NUMCOL+2]}"
|
| 671 |
echo "${server}${client}_w".value "${elements_array[$i*$NUMCOL+3]}"
|
| 672 |
echo "${server}${client}_t".value "${elements_array[$i*$NUMCOL+1]}"
|
| 673 |
echo "${server}${client}_s".value "${elements_array[$i*$NUMCOL+4]}"
|
| 674 |
done |
| 675 |
echo -e "\nmultigraph vnx_emc_nfs_client_b_s" |
| 676 |
for (( i=0; i<$((LINES)); i++ )); do |
| 677 |
client="$(clean_fieldname "_${elements_array[i*$NUMCOL]}")"
|
| 678 |
echo "${server}${client}_r".value "$((${elements_array[$i*$NUMCOL+6]} * 1024))"
|
| 679 |
echo "${server}${client}_w".value "$((${elements_array[$i*$NUMCOL+7]} * 1024))"
|
| 680 |
echo "${server}${client}_t".value "$((${elements_array[$i*$NUMCOL+5]} * 1024))"
|
| 681 |
done |
| 682 |
echo -e "\nmultigraph vnx_emc_nfs_client_avg_usec" |
| 683 |
for (( i=0; i<$((LINES)); i++ )); do |
| 684 |
client="$(clean_fieldname "_${elements_array[i*$NUMCOL]}")"
|
| 685 |
echo "${server}${client}".value "${elements_array[$i*$NUMCOL+8]}"
|
| 686 |
done |
| 687 |
|
| 688 |
#nfs-std |
| 689 |
# bash-3.2$ server_stats server_2 -monitor nfs-std |
| 690 |
# server_2 Total NFS NFS NFS Avg NFS NFS NFS Avg NFS |
| 691 |
# Timestamp NFS Read Read Read Size Write Write Write Size Active |
| 692 |
# Ops/s Ops/s KiB/s Bytes Ops/s KiB/s Bytes Threads |
| 693 |
# 18:14:52 688 105 6396 62652 1 137 174763 3 |
| 694 |
member_elements_by_line=$(run_remote server_stats "$server" -monitor nfs-std -count 1 -terminationsummary no -titles never | grep '^[^[:space:]]') |
| 695 |
IFS=$' ' read -ra graphs <<< "$member_elements_by_line" |
| 696 |
# echo "$member_elements_by_line" |
| 697 |
# echo "${graphs[@]}"
|
| 698 |
|
| 699 |
echo -e "\nmultigraph vnx_emc_nfs_std_nfs_ops" |
| 700 |
echo "${filtered_server}_rops.value ${graphs[2]}"
|
| 701 |
echo "${filtered_server}_wops.value ${graphs[5]}"
|
| 702 |
echo "${filtered_server}_tops.value ${graphs[1]}"
|
| 703 |
|
| 704 |
echo -e "\nmultigraph vnx_emc_nfs_std_nfs_b_s" |
| 705 |
echo "${filtered_server}_rbs.value $((graphs[3] * 1024))"
|
| 706 |
echo "${filtered_server}_wbs.value $((graphs[6] * 1024))"
|
| 707 |
echo "${filtered_server}_tbs.value 0"
|
| 708 |
|
| 709 |
|
| 710 |
echo -e "\nmultigraph vnx_emc_nfs_std_nfs_avg" |
| 711 |
echo "${filtered_server}_avg_readsize.value ${graphs[4]}"
|
| 712 |
echo "${filtered_server}_avg_writesize.value ${graphs[7]}"
|
| 713 |
|
| 714 |
|
| 715 |
echo -e "\nmultigraph vnx_emc_nfs_std_nfs_threads" |
| 716 |
echo "${filtered_server}_threads.value ${graphs[8]}"
|
| 717 |
fi |
| 718 |
done |
| 719 |
if [ -z "$nas_server_ok" ]; then |
| 720 |
echo "No active data movers!" 1>&2 |
| 721 |
fi |
| 722 |
exit 0 |
| 723 |
|