dataMine Overview

dataMine is a data capture and visualisation plugin for Vera. It includes a Vera plugin to record and
process the data, and a web GUI for data visualisation and configuration. The web GUI is shown in
Figure 1.
The plugin that runs in Lua records all changes for variables that are being logged. The raw data is
then written into log files on Vera (preferabl on a U!" stic#$, and the GUI then re%uests this data
fro& the plugin. "oth sides are therefore re%uired to operate ' the Vera, providing the data capture
and server, and the GUI, providing the data &anipulation and visualisation.
This version of the guide was updated on () *a +(1,.
dataMine wor#s b &onitoring all changes in device variables in Vera. -henever a variable
changes, dataMine gets called, and it logs this data into a file for later use. The GUI then interacts
with this data to provide the user a visualisation interface.
Installation
dataMine is installed through the UI. &ar#etplace. This installs all re%uired files onto our Vera,
however so&e configuration is re%uired before dataMine will run.
/ U!" stic# is highl reco&&ended to store dataMine data. 0epending on the nu&ber of variables
ou are logging, dataMine can generate a reasonable a&ount of data, additionall, as this data is
written regularl, this &a cause the internal Vera &e&or proble&s if it is used for an e1tended
duration.
2age 1
The U!" stic# can be for&atted on -indows and plugged into a free U!" slot on the Vera. 3ou
should avoid for&atting the drive with the F/T14 file sste& as this has li&itations. 5n an &odern
version of -indows, it should co&pile as F/T,+ be default so this shouldn6t be a proble&. Linu1
et1+7e1t, file sste&s are fine. /fter for&atting in -indows, ou should set the dis# label in the
dis# properties ' this allows ou to identif it once it is plugged into Vera.
3ou should then run the GUI and ou will be presented with the dialogue shown in Figure 1. This
dialogue as#s ou to select the UUI0 (the Universall Uni%ue Identifier$ of the U!" to be used for
dataMine. If ou have &ore than 1 U!" stic# installed in our Vera, and ou6re unsure which one is
the one for dataMine, then it is suggested to add a label to the drive ou wish to use for dataMine,
and refresh the GUI page so that onl one UUI0 is shown.
5nce ou6ve selected the UUI0, press !ave, and dataMine should be configured. 8estart Luup
(press 8eload in UI.$, and dataMine should be up and running.
The configuration dialogue will set the variables shown in Table 1. It is also possible to set these
values &anuall if ou please, however the dialogue will be shown anti&e that the U!" stic# fails
to &ount correctl.
Configuration Variable Suggested Value
!et0ata0irector 7data*ine
!et*ount2oint "lan#
!et*anual*ount (
!et*ountUUI0 U!" UUI0
Table 1: Suggested configuration for use with Vera USB logging.
2age +
Figure 2: dataMine initialisation dialogue.
-hen dataMine starts, it will create two further variables 9 :&ountLocation: and :&ountTpe:. It
should be noted that these ;ust report the current &ount status, and are not &eant to be used as data
entr points b the user. If ou have &anuall set a &ount point with the <!et*ount2oint= setting,
then if the &ounting wor#s then :&ountLocation: will be the sa&e as :!et*ount2oint:.
:&ountTpe: will reflect the filesste& used b the U!" stic# ' tpicall :vfat: when purchased
straight fro& the shop. This should loo# si&ilar to what is shown in Figure ,.
Using dataMine with the Vera USB logging
For use with the Vera U!" logging, additional configuration &a be re%uired. This is because Vera
will &ount the U!" stic#, and if dataMine is also allowed to &ount the sa&e stic# to a different
&ount point, corruption can occur. 3ou &ust either use a separate U!" stic# for dataMine and use
the configuration described above, or point the dataMine director to a location below the log
director. >ote that the preferred option is to use a separate stic# if possible to avoid conflicts.
!uggested configuration for use with U!" logging is shown in Table +.
Configuration Variable Suggested Value
!et0ata0irector 7t&p7log7c&h7data*ine
!et*ount2oint >ot re%uired to be set
!et*anual*ount 1
!et*ountUUI0 >/
Table 2: Suggested configuration for use with Vera USB logging.
2age ,
Figure 3: dataMine configuration variables.
Use
Plugin Properties
The data*ine plugin presents a si&ple device in the UI. dashboard ' as shown in Figure ?. This
;ust shows how &an channels are being &onitored, and how &an channels have been configured
in all.
Figure . shows the &ain properties tab. /gain, this present the sa&e infor&ation as the device
overview, however there is also a lin# to the GUI webpage which is outside of the UI. interface.
2age ?
Figure : dataMine !ro!erties !age.
Figure ": #evice overview.
Graphical User Interface
The GUI is a web interface that allows ou to visualise the data fro& Vera. It uses a tabbed interface
to select different data views or tpes as shown in Figure 4. /t the right side of the tab bar is an
<5nline !tatus= indicator ' this is green if the web interface is co&&unicating with Vera, and
gre if Vera is not co&&unicating with the browser. It displas red if co&&unication is o#,
however there are errors within dataMine (eg. Initialisation error, or error writing data to the U!"$.
Graph Tab
The graph tab provides an interface to graph the data being recorded b the dataMine plugin.
2age .
Figure $: dataMine User %nterface.
The graph displa is split into two &ain areas ' the configuration area to the left, and the graph
itself. In between the two is a slider bar that allows ad;ust&ent of the position between the two.
>ote that there6s also a button in the &iddle of this slider with an arrow pointing to the left. @lic#ing
on this will hide the configuration area and ou6ll have a screen that loo#s li#e Figure ). @lic#ing on
the bar on the left will te&poraril displa the configuration area, while clic#ing on the arrows at
the top of the bar will restore the configuration area. This allows the full window area to be &ade
available for graphs.
If running data*ine on a &obile browser, the bar will default to the left to provide the &ost space
possible for s&all screens.
2age 4
Figure &: dataMine gra!h window.
The configuration area includes areas for selecting the channels to graph, and the graph options. In
the Graph @hannels area, clic#ing on a variable na&e, the variable will select or deselect. -hen
selected, the variables can be graphed b clic#ing on the U!date 'ra!h button (the graph will
use the sa&e ti&e period as is on the current graph$. @lic#ing on the (eset 'ra!h button will
clear the current selection and reset the ti&e period bac# to default ' it will not affect the current
graph, however. @lic#ing on the Save 'ra!h button will allow ou to save the graph for later
%uic# recall. !aved graphs will be listed in the Saved 'ra!hs list.
2age A
Figure ): 'ra!h tab with configuration area hidden.
Bach channel has an icon on the left side 9 this indicates the tpe of data. The *ast U!dated colu&n
also shows if data has been received recentl. / tic# is shown if the variable data has changed
recentl, a ellow e1cla&ation is shown if the data is %uite old, and a red e1cla&ation is
shown if the data is ver old.
The definition of these flags is configurable per channel since different channel tpes e1pect
different change rates.
@lic#ing on the 'ra!h +!tions navigation bar brings up a nu&ber of para&eters that can be
changed in the graph. !ee Figure 1( and Table , for details of the options.
2age )
Figure ,: 'ra!h channel selection
Figure 1-: 'ra!h o!tion window.
Option Description
0ispla data &ar#ers 0isplas a point on the graph where each <real= data point e1ists.
Lines are then drawn between the data points to &a#e up the
graph.
0ispla shadows 0isplas a border and shadow around the chart,
0ispla legend 0isplas the graph legend showing what data is displaed with
each colour.
0ispla crosshairs Bnables crosshairs.
0ispla tooltip !hows the tooltips where the cursor or crosshair is near a data
point. The tooltip shows the actual data displaed on the graph.
0ispla local ti&es !witches between local ti&e and Universal ti&e.
Line width @hanges the line width of graphs.
/ni&ation Bnable or disable the chart ani&ation when data is drawn.
Table 3: 'ra!h o!tions.
The toolbar across the top of the graph screen provides a nu&ber of buttons for interacting with the
graph. Bach of the buttons has a tooltip, so if ou put the &ouse over the button, ou will get a hint
as to what the button does.
2age C
Icon Description
Doo& in
Doo& out
0ispla last das data
0ispla last wee#s data
0ispla last &onths data
0ispla last ears data
!croll left
!croll right
0ispla real9ti&e graph
Turn off displaing Vera >otifications
0ispla notifications relevant to the channels being graphed (fro& the sa&e device, or its
parent device$
0ispla all notifications ' note that this can be overwhel&ing when long ti&e9scales are
displaed.
0isplas gre bands on the chart during night9ti&e.
0ispla graph infor&ation
Table ": 'ra!h toolbar buttons.
It is possible to Eoo& using the &ouse as shown in Figure 11. !i&pl hold the &ouse button down
at the start of the Eoo& period, &ove to the end, and release the &ouse. >ew graph data will be
loaded.
2age 1(
The real9ti&e graph displas data for the past 1. &inutes, and updates ever second. 0ue to the
nature of the displa, it displas data in <absolute for&= 9 ie. it does not interpolate between data
points. This gives the displa a &ore stepped nature than the standard graph.
(>oteF 8eal9ti&e graphing is not i&ple&ented in the current version ' it will return soon$.
Notifications
dataMine can displa Vera notifications on the graph as shown in Figure 1+. The notifications are
updated ever &inute, so ver new notifications &a not be displaed. If there are &ore than .((
notifications in the graph, dataMine will onl show a selection of .(( b tri&&ing out ever n
th

notification. In realit, this is far too &an to be able to view anwaG
2utting our &ouse over the notification will displa &ore infor&ation on the notification.
2age 11
Figure 11: 'ra!h window .oo/
Day-Night banding
dataMine records the sunrise7sunset ti&es calculated b Vera, and then allows the nights to be
displaed as gre bands in the graphs. This is shown in Figure 1,. To activate7deactivate this option,
clic# on the &oon icon in the toolbar.
>ote that as dataMine does not calculate the sunrise7sunset ti&es, this feature will onl wor# after
the date that this was added to dataMine (/pril +(1,$. /lso, if our Vera is turned off for a period,
this will also stop the ti&es being logged.
2age 1+
Figure 12: 0otification dis!la1
Saing Graphs
Fro& the graph selection window, ou can clic# the !ave button to save a graph configuration for
later recall. Figure 1? shows the window that is then displaed. This allows selection of which a1is
each channel will be displaed on, as well as an icon that will be displaed alongside the graph in
the selection list.
The a1is selection &ust be a nu&ber between 1 and ?.
The Huic#View reference is a reference for displaing the graph in an e&bedded webpage. For
&ore infor&ation, see the section on e&bedding below. >ote that this should >5T contain spaces.
2age 1,
Figure 13: #a120ight banding.
Figure 1. !hows the saved graph selection list. The icon allows ou to delete the saved graph,
and the icon allows ou to edit the graph data.
>ote that when editing a graph, if ou change the na&e, dataMine will save a new cop of the
graph and ou will need to delete the one ou don6t want.
Dashboard Tab
Figure 14 shows the dashboard. @urrentl this is (ver &uchG$ wor# in progress and it shows the
energ currentl being used. There is a gauge showing the current whole house energ, and a donut
chart showing the split b device and categor. / trend chart shows the whole9house use over the
past +? hours.
2age 1?
Figure 1": 'ra!h Save 3 4dit window.
Figure 1: Saved gra!h selection list.
(This will change$.
The whole9house gauge provides the current whole house
energ use. It is a su& of all the whole house energ
categories, less the energ generator categories.
The energ consu&ption donut shows energ consu&ers b
categor and device. The Unaccounted categor is the
difference between the total of all consu&ers, and the current
whole9house consu&ers (note that this doesn6t include the
energ generators$.
2age 1.
Figure 1$: #ashboard tab
Configuration Tab
The configuration tab shown in Figure 1A allows configuration of dataMine. @urrentl, this
pri&aril includes the configuration of variable logging, but further configuration will be added in
future.
Variable !onfiguration
Two lists are presented for variable configuration. The active devices are listed under #evice
Variables, and inactive devices are listed under 'host Variables. Ghost variables are variables that
still e1ist in the dataMine configuration, but the devices have been deleted fro& Vera. 3ou can
delete the ghost variable, or #eep it for reference. 0eleting the ghost variable will not delete the
historical data fro& the archive.
5nce configuration changes have been &ade, ou &ust press the Save button for the& to ta#e
effect.
Option Description
0evice The Vera device na&e and nu&ber (can not be changed$.
!ervice The Vera !ervice na&e (can not be changed$.
Variable The Vera Variable na&e (can not be changed$.
Vera @ategor The Vera 0evice @ategor (can not be changed$.
2age 14
Figure 1&: 5onfiguration Tab
Option Description
0ispla >a&e The displa na&e that will be used in dataMine graphs.
Units / te1t string to set the units.
Logging Bnabled Indicates if dataMine is logging data for this variable. !et to true to
enable logging.
Bnerg @ategor The dataMine energ categor. This is #ept separate fro& the Vera
device categor. The following categories can be used 9F
1 -hole9house
+ Lighting
, Ieating
? Bntertain&ent
. /ppliances
4 @o&puting
A Utilities
+1 !olar 2V generation
++ -ind generation
(>ote that this will be &ade &ore user friendl in future. It is currentl li&ited b the libraries being used$
Graph Tpe !ets the graph tpe to be used with this channel
( 0efault (spline$
1 Line
+ !catter
, /rea
? /rea7Line co&bined (for status graphs$
(>ote that this will be &ade &ore user friendl in future. It is currentl li&ited b the libraries being used$
0ata 5ffset /dds a data offset when graphing out the channel. The value will be
added to each value to allow calibration of a channel. This does not
affect the logged data which will alwas be the value.
0rows -arning /llows &odification of the drows warning ti&e. This is set in
&inutes. The drows warning is the ti&e after which the icon is
shown. !etting this to ( will use the sste& default.
0rows Brror /llows &odification of the drows error ti&e. This is set in &inutes.
The drows error is the ti&e after which the icon is shown. !etting
this to ( will use the sste& default.
Bnable out9of9li&its filter Bnables or disables filtering of values. Filtered data outside of the
specified li&its will not be displaed in graphs. This can be used to
filter out large spi#es.
Filter &a1i&u& *a1i&u& filter value. 0ata above this nu&ber will not be displaed.
Filter &ini&u& *ini&u& filter value. 0ata below this nu&ber will not be displaed.
Table : Variable configuration o!tions.
2age 1A
Figure 1) shows the 9a1is loo#up. This is pri&aril useful for alphanu&erical data (wind direction,
weather condition etc$. dataMine actuall generates this loo#up auto&aticall for such data,
however this configuration window allows ou to change the wa the data is graphed. " changing
the nu&bers against the labels, ou will change the order the data is graphed in the vertical a1is.
It is also possible to use this configuration for nu&erical data that ou would prefer to be displaed
using labels. >ote however that this &a produce strange results for data with continuall varing
data, and is best used for variables with a li&ited nu&ber of possible points (as per the data
&entioned above$.
Embedding graphs in a web page
-hen saving a graph, there is the option to specif a <Huic#View reference=. This is a short na&e
for the graph ' it &ust not contain spaces, and it should be #ept short. 3ou can then use this
reference to e&bed a full window page into a web browser to displa the saved graph.
The U8L should be Jvera ipK7d&7graph.ht&lLrefMJ%uic#refKNperiodMJdasKNupdateMJ&insK
If period is specified, it will override the nu&ber of das to displa the graph, otherwise it will
default to the nu&ber of das specified when the graph was saved.
If update is specified, the graph will auto&aticall be updated periodicall. The update ti&e is in
&inutes.
B&bedded graphs are &eant to be %uic# and full screen. The do not offer the sa&e interaction as
2age 1)
Figure 1): 62a7is loo8u! configuration
the full dataMine GUI. 0a7night boundaries can be displaed, but notifications are not supported.
Advanced Topics
Under the hood
dataMine wor#s b loo#ing for changes in device variables ' it is not periodicall logging data
based on ti&e, it si&pl logs changes. This is i&portant to note when thin#ing about how a graph is
rendered ' for status tpe data (eg. on7off$, the graph needs to account for this to avoid diagonal
lines.
dataMine records data in @!V files (@o&&a !eparated Value$. (@urrentl$ three different files are
generated for each variable being recorded 9 a raw, hourl and dail file. Files are then split to avoid
the& getting too large (which is a potential proble& for constantl changing sensor data such as
te&perature$. Bach ti&e Vera observes a change in the data for a &onitored variable, dataMine will
log the current value into a file. Bach da, the long ter& histor files are generated fro& the raw
data.
Bach line of data in the @!V logfiles includes two nu&bers 9 the ti&e of the data9point (as a uni1
ti&esta&p$, and the data value. For the raw file, the actual changes are logged. For the hourl and
dail files, the &ini&u& and &a1i&u& value and ti&e are logged for each ti&e period.
USB and "inu#
Linu1 has a different philosoph to -indows when it co&es to attaching U!" &e&or stic#s, or
dis# drives. !uch devices are <&ounted= into a director in the root file9sste&, rather than being
seen as a separate drive. / director is first created in the root file9sste&, and the U!" stic# is then
&ounted into this director. This has the following i&plications 9F
"efore the U!" stic# is &ounted, if ou were to loo# in the director, ou would see files
that are resident in the &ain file9sste&.
/fter the U!" stic# is &ounted, ou will see files that are in the root director of the U!"
stic#. If there were an files in the &ount director on the &ain file9sste&, these will no
longer be visible as this director is effectivel replaced b the U!" stic#.
Setting syste$ defaults
/ UI will be added to set the sste& defaults, but for now, this can be done in the following wa...
Tpe the following U8L into our browser
<<Vera Address>>/port_3480/data_request?id=lr_dmCtrl&control=saveConi!&
then add the following para&eter to the end of the above line to set variables (note ' configuration
na&es are case sensitiveG$.
2age 1C
energ-holeIouse*inMnu& !ets the &ini&u& value of the whole house gauge
@an not be set above (.
energ-holeIouse*a1Mnu& !ets the &a1i&u& value of the whole house gauge.
@an not be set less than (.
general0rows-arningMnu& !ets the drows warning sste& default.
Values in &illiseconds ' defaults to 1 hour (,4((((($
general0rowsBrrorMnu& !ets the drows error sste& default.
Values in &illiseconds ' defaults to 1 da ()4?((((($
graph0efault2eriodMnu& !ets the default graph period.
Values in das ' defaults to 1 da. *ust be between 1 to ,(.
3ou should get the <5O= response if everthing went o#. >ote that no error chec#ing is perfor&ed
when ou &a#e changes ' these settings are used in the GUI, and when the GUI reads the& bac#
fro& Vera, it does perfor& a sanit chec# and &a reset to a default if the are dee&ed to be outside
of nor&al bounds.
Bac%ing up dataMine data
The following is a &ethod fro& Pvespa&an on how to auto&aticall #eep a bac#up of our
data*ine data director. This runs on linu1. >ote that ou need to change the ip address and
directories.
Fro& Pvespa&an original post Qanuar (4, +(1,
(httpF77foru&.&icasaverde.co&7inde1.php7topic,1,(A).&sgCA+4,.ht&lR&sgCA+4,$
5O, so I have now whipped so&ething up, which Ssea&sS to do about what I need for now.
I thin#, that if the logs get ver big over ti&e, &abe this isn6t the correct &ethod. "ut at least it can
run at ti&es where house is at sleep.
"asicall three stepsF
1. @reate public ssh #es for unattended login to our Vera fro& our linu1 server
+. Using scp script, cop the whole data*ine log direcor to our linu1 server.
,. 2ut a line in ou crontab so it is done once ever night.
It would have been nice to use rsnc instead of scp, but it sea&s that it is not included in the vera
dist, so scp will have to do. >ot sure how &uch data we will see in the end, but ti&e will tell.
!o in detail for anone that wants to use this tpe of bac#upF
1. @reate our public #es according to e.g. this guideF httpF77wi#i.openwrt.org7do#u.phpL
idMoldwi#iFdropbearpublic#eauthenticationhowto
2age +(
+. *odif the script for our path6s and vera ip (if ou don6t have 6vera6 in our dns7hosts file$.
/long with our veras I2, ou will want to change both destdir and the source path on our vera.
#!/bin/bash
date="`date +%Y-%m-%d`"
olddate="`date --date '2 months ago' +%Y-%m`"
destdir=~/backu/datamine
# get the !iles" #era does not ha$e rs%nc& so 'e use regular sc
sc -(r root)$era*/mnt/datamine+logs ,destdir/
# create the backu tar ball
tar -c-! ,destdir/datamine+,date"tar"g- ,destdir/datamine+logs
# !i. rights""
chmod /00 ,destdir/datamine+,date"tar"g-
# remo$e tem!iles
rm -r! ,destdir/datamine+logs
# 1emo$e all backus created 2 months ago
rm -! ,destdir/datamine+,olddate-2"tar"g-
@reate the destdir ( in this case T7bac#up7data&ine $
,. 5nce ou have tested the above configuration, and ou are sure it wor#s ;ust add line to our own
crontab (crontab 9e in & case$ and add a line li#e this (running . o6cloc# in the &orning$.
0 3 2 2 2 ~/bin/backu+$era+datamine
If you hae a proble$
If ou have a proble&, I6& happ to help. Iowever, please send the Vera logfiles, provide
infor&ation on what Vera &odel ou are using, if ou have other U!" stic#s fitted.
There is now a %uic# wa to get so&e of this with the following U8L
44#516 7899/ort+:;<0/data+re(uest=id=lr+dm>trl?control=debug
This will du&p our configuration, U!" &ount data, and an recent log entries. Iowever, this will
onl wor# once the basic plugin is loaded and running. 2rior to that, please provide as &uch
infor&ation as ou can.
U!"F 8ead9onl file sste&
@ogging error logging $ariable /A**urn*un-org*ser$ice7d*BemeratureCensorD**>urrentBemerature
Enable to oen !ile !or 'rite /dataFine/@o!t Bemerature - >urrentBemerature G122;HI"t.t
5rror* '/dataFine/@o!t Bemerature - >urrentBemerature G122;HI"t.t* 1ead-onl% !ile s%stem'
@aused b errors in the file sste&. @op all files off and refor&at.
2age +1