Jeg har et ønske om å sette opp en database og et Grafana dashboard for Futurehome installasjonen min. For å følge med på ting som temeperatur, strømforbruk osv.
Jeg forstår at dette skal være mulig ved hjelp ecollector i Thingsplex og at man der skal kunne opprette en kobling mot DBInflux for å feede data.
Men siden jeg ikke kan denne typen konfigurasjon fra før så sliter jeg litt med hvordan jeg setter opp koblingen.
Har noen gjort dette og har lyst til å liste opp en kort “how to” på hvordan man går frem?
Jeg har gjort dette tidligere med både Influx og Grafana hosted hos Aiven.io. Husker det som ganske rett frem ift. konfigurasjon. Først opprette en InfluxDB hos Aiven, deretter legge inn riktig config for Ecollector i Thingsplex. Til sist opprette Grafana hos Aiven. Forholdsvis raskt å sette opp, men synes Aiven ble litt kostbart og ga liten fleks for å installere plugins til Grafana så avsluttet det.
Tenkte å finne en rimeligere/bedre løsning, men har ikke rukket det enda. Del gjerne om du kommer over noe.
Aha, kult å høre.
Det jeg slet med (ikke helt forstår) er hvordan konfigen i Ecollector skal se ut. Mulig det er rett frem, men forsto ikke helt hvilke verdier jeg skulle konfigurerer opp.
Jeg testet ut https://www.influxdata.com/, men vet ikke hvordan det ser ut vs feks Aiven
Når du lager en InfluxDB hos Aiven får du host url, username, password etc. som deretter legges i Thingsplex under Analytics → Settings → Data Collection Settings → Add new → Configure.
Hvis jeg husker rett så leverer influxdata.com Influx “as a service” så der må du først konfigurere data input via Telegram eller en annen plugin først - for å få et “endpoint”. Men har ikke sjekket det på en stund, så ikke sikker.
Aiven kostet fort 150 USD pr. mnd så det droppet jeg.
Aha, ok. Det høres litt for avansert ut for min bok og som du sier blir disse betal-servicene ganske dyre.
Leser at en del kjører en lokal instans av Influx og Grafana på server/maskin hjemme, mulig det er det beste å gå for. Ihvertfall som en test!
Jeg er også takknemlig for alle tips😊 Har en Synology NAS hjemme, og tenkte å teste med å kjøre influxdb og Grafana i Docker på den👍🏻
Noen som har testet det oppsettet?
Jeg har det kjørende på en IntelNUC hjemme, men raspberryPI burde også gå fint.
Influx som er konfigurert på smarthubene nå bruker v1 API og ikke v2 som er tilgjengelig fra influx sine sider. @alivinco ser på muligheter for å støtte begge, som gjør oppsettet enklere
Yes, soon ecollector will support InfluxDB V2 as an external export target in addition to InfluxDB V1 and CSV export. Also, ecollector will be available as docker container that can be installed on any NAS server or Rpi on the same host as InfluxDB itself. That will allow installing the latest updates before my contribution will be officially accepted by FH.
Would you be able to elaborate on how those CSV files are planned to be delivered @alivinco? I would really like an alternative where I could get those written to eg. a GCP bucket or similar.
CSV export is already implemented (the documentation will follow) but not very useful if the service (ecollector) is running on FH hub. But once it’s installed on a different host, a user can configure the destination folder for csv files either on a local or mounted file system. For instance, mounted GCP bucket using Cloud Storage FUSE or configure something like rsync and then load data into BigQuery.
Thanks. It seems like there is a need to run ecollector on a local device anyhow then, eg. on a pi. Or do you have any experience running ecollector from a cloud hosted container? My end goal would be to load aggregated logs into BigQuery with as little effort as possible
I was able to set up a docker instance on my Mac and run influxdb(v1.8.9-alpine) and Grafana inside it. I also set up the connection in Ecollecter in TP and i get a lot of measurements into my db.
They are listed as
electricity_meter_energy
electricity_meter_power
sensor_lumin.evt.sensor.report
sensor_temp.evt.sensor.report
etc
My issue now is that I dont get any values inside the measurements.
Grafana does not display any data points when I add a measurement in the “FROM” field
When I try to access the data inside terminal it tells me:
"error query: too many segments in “sensor temp”.“evt”.“sensor”.report at line 1, char 1
Can anyone her help me with that I have done wrong and what needs to be adjusted?
It’s been some months since I was playing around with Ecollector data and I don’t have much experience with Grafana/Influx. So bear with me; If I remember correctly you need to keep in mind that all data sets (except gen_raw) are aggregated. Ie. you would need to apply aggregation to your values when writing queries.
If you can share your full query, what you are trying to get and/or some screenshots I might be able to guide you.
Also, did you create your own container or which did you use?
Hmm, then I need to look into that I guess
Is that something I would do inside if Grafana or in the InfluxDB instance?
See screenshot from the query/filtering I can do for setting up a graph in the dashboard. (This example is a graph for energy usage, but obviously I don’t have any values since that is my issue )
In Grafana. I would go to the “explore” tab of Grafana to find the correct data from the source you would like to use. If I remember right data “columns” have a slightly different namespace in gen_raw and the other “tables”. Can’t really see which one you are querying. Should be 5 sources (aggregations) of high freq data and one (default_w20) holding low freq data, according to the docs. Unfortunately my instance is deactivated so can’t go back and find how I did it.
Awesome , I see that everything has been solved. As you managed to catch, it’s important to select correct retention policy in Grafana. Ecollector creates multiple buckets/retention policies with different retention time , and data flows from one bucket to another as it gets older and does downsampling (reducing precision). It’s done for storage optimisations . But if you are running DB on external host you might want to write data without any optimisations, it can be done by setting profile configuration to raw. Default profile is optimized . Raw profile writes data directly into DB without doing any processing , so you can get higher precision.
Just so I understand it correctly, the different buckets is just how long the data is stored there and are accessible for collecting?
Ex: if I create a new dashboard and try to look 3-4 weeks back in time, the gen_raw will only show for the last 2 weeks. E.g. if I want to have dashboards with a long perspective back in time I either need to adjust the buckets I use? Or I can store the gen_raw data in my db?
All measurements are divided into 2 groups - normal measurements and high-frequency measurements.
Retentions policies for high-frequency measurements :
Name
Expected query range
Aggregation
Retention
gen_year
1-5 years for last 5 years
1 d avg
5 years
gen_month
1-12 month for last year
1 hour avg
1 year
gen_week
1-4 weeks for last 3 month
10 min avg
3 month
gen_day
2-14 days for last 2 weeks
1 min avg
2 weeks
gen_raw
1-2 days for last 2 weeks
none
2 weeks
Normal measurements stored in gen_default which has 1 year retention policy , which means that data is stored only for one year .
High frequency measurements are : electricity_meter_power, electricity_meter_energy , electricity_meter_ext , electricity_meter_energy_sampled and all sensors except sensor_presence and sensor_contact .
PS : retention defines for how long data point is stored before it’s get deleted
)
