Igor's Blog

We use partitioning in at least one of the databases at work, changing to partitioned tables has allowed us to keep the system running within our imposed SLAs. Recently I've started to wonder however, what happens when we start hammering individual partitions more than others. The data distribution would be skewed and the benefits of partitioning lost. I decided to put together a bit of SQL and then use gnuplot to show me how well our data is distributed.

These results are from a development environment where I was testing out my scripts, but it clearly shows how some partitions can get filled at a much quicker rate than others. Each of the bars is an individual partition, the height of the bar represents the number of rows of data in that partition.
dbpartplot.png


The SQL behind this graph is very simple...adjust the like statement to suit. In my case I get data for multiple tables and then filter it later in a shell script.
 SQL
select up.table_name, up.partition_name, up.num_rows
from user_tab_partitions up
where table_name like 'MY_TABLES_%'
order by up.table_name, up.partition_name asc;


I export the result of this query to a file called export.d. This has to be a tab-delimited file that doesn't use quotes around each of the data values. The data looks something like this...
 export.d
...
MY_TABLES_TBL1 SYS_P42665 822568
MY_TABLES_TBL1 SYS_P42666 394797
...


This is then processed by a shell script.



To make the graph, I used gnuplot with the following shell script to generate the image...
 Bash Script
#!/bin/bash
function plot {
gp_table=$1
gp_file=$2.$1
grep $1 $2 > $gp_file
gp_95pct=`cat $gp_file|awk '{print $3}' |sort -n|awk 'BEGIN{i=0} {s[i]=$1; i++;} END{print s[int(NR*0.95-0.5)]}'`
gp_99pct=`cat $gp_file|awk '{print $3}' |sort -n|awk 'BEGIN{i=0} {s[i]=$1; i++;} END{print s[int(NR*0.99-0.5)]}'`
$GNU_PLOT << EOF
set terminal svg size 800,400 noenhanced font "Verdana,10"
set output "export_$gp_table.svg"
set title "Table Partitions Row Sizing: $gp_table"
set ylabel "Data Rows"
set grid y
set format x ''
set style fill solid 1.0
set palette defined (0 "red", 1 "#FFA500", 2 "#555577")
unset colorbox
plot "$gp_file" using 0:3:(\$3 > $gp_99pct ? 0 : (\$3 > $gp_95pct ? 1 : 2)) with boxes palette notitle
EOF
rm $gp_file
}
GNU_PLOT=gnuplot
plot MY_TABLES_TBL1 export.d



What this bash script does is define a function called plot, then it defines the gnuplot executable and calls the plot function passing in the name of the table to filter by and the name of the data file.

Inside the plot function, I use grep to get the data for the table that's been passed in. Then awk is used to calculate the 95th and 99th percentile values for the data points in the filtered file. These are used for colouring the bars. 95th percentile bars are orange and 99th are red, others are blue-gray.

This is all followed by calling gnuplot to generate the bar graph, and then the filtered file is cleaned up.

It's all quite simple, it did take me a long time to get the syntax right for gnuplot however. In my actual script I also have a loop to process all of the tables in the exported file all in one go.

-i

, , , , ,
About — I'm an enthusiastic software engineer and consultant interested in many fields including J2EE, programming, electronics, 3D printing, video games, wood working and gardening.
See my Resume for more information.
The views expressed in this blog are my own and not those of my employer.
comments powered by Disqus
My other posts you may like...
Programming, DIY, Games, Hacks, Tech and more.
Follow me on...
Current and Past Projects
See my Resume

Subscribe


RSS Feed

My Other Web Sites

Igor and Elise's Travels
Riverside Expressway Cam
StrFunc() Online
300 George St Blogumentary
Guru JSON-RPC Tester
Extrudifier Object Designer

Recent Blog Posts

Binding extra variables with PHP PDO returns no results

New Nintendo 3DS XL, R4i SDHC RTS Lite and DSDoom

Older versions of JD-GUI crashing on macOS Sierra

How does Seeed Fusion stack up as a low budget PCB fab

Where to find the last item (Robbie's lost lyrics) in GravityFalls Lake level

Detecting print requests to print contents of a DIV with JavaScript

Using Google DFP with AdSense on responsive pages

How to change the email subject line in Gmail

Force PHP GD library to load JPEG images that have invalid data

How to uninstall Samsung Smart Switch from your Mac

Recent Galleries

Space Food - Chocolate Ice Cream with Chocolate Chips

Legeod Star Wars AT-DP kit

DIY spare parts computer build with a RAIDMAX Anura case

Fake 'Lepin' brand Lego packaging

Hardwood garden bench with clear resin void filler

Fixing a 3D printer extruder that stopped heating up

Easily increase disk space in a Lenovo Ideapad 100S 14" laptop with an M.2 SSD

Making a multi-piece 3D printed solder spool holder stand

DIY indoor apartment grow light wiring

Good Friday Electronics fun Easter Bunny LED PCB Kit IBEABU-01.0

Top Categories

Blogs I follow

Matt Moores Blog
Georgi's FlatPress Guide
Perplexing Permutations
The Security Sleuth

Friends

RAWS Parts Online
Alpha Dimensions Hosting
Kristensen Photography
Ilia Rogatchevski
Travelling Fairy

Blog Activity

Blog Activity
Follow me on... 
     
...or subscribe for updates!

Don't show this again