Inacceptable huge default "sensor unavailable time" of 24 hours

[bcutter]

Problem:
Once sensors do not provide any updates, it takes deCONZ full 24 hours (!!!) to categorize them as “unavailable”.

I experienced this multiple times and it’s always the same behaviour: 24 hours flat line in sensor once they switch to “unavailable” (and an additional, self-made monitoring based on HA sends me an alert).

Those 24 hours are not acceptable.

I believe this is something hard coded directly in deCONZ, Phoscon or whatever.
If not, it must be something from the official deCONZ addon for Home Assistant.

Please shed some light on this:
1) Where do those 24 hours come from? Firmware, deCONZ, deCONZ integration, …?
2) Why 24 hours.
3) Why hard coded. Or did I miss a user custom setting for this?

In case it matters:

• I use a ConBee 2 on a Pi 4 running the official deCONZ addon for Home Assistant.
• In the end I integrate all devices/entities using the official deCONZ integration.
• Devices are paired using Phoscon.
• My ZigBee mesh is very strong and I no all the ZigBee/WiFi channel separation best practices.
• It even doesn’t matter what sensors are affected and for which reason (worst case: dead battery).
• The point is: deCONZ seems to accept A WHOLE FREAKING DAY of missing sensor updates.

I am not willing to add a second “because of strange deCONZ behaviour needed monitoring” (which would look for the entities dynamic within a shorter time range - e. g. “if no value change of at least one entity of every device happens within 3 hours, send an alert”). That would at best be a workaround, only help me and would not improve deCONZ.

Please help to make deCONZ even better. First lets have a look at those 3 questions to sort out components and reasons.

[Mimiix]

Hi,

I am more intrested in why the sensors stop reporting.

Can you share some logs? Please see #deconz to see what log levels are required.

Secondly, are you on latest firmware?

[Smanar]

I think it’s hard coded yes, in deconz API.

    if (sensor->node() && !sensor->node()->nodeDescriptor().receiverOnWhenIdle() &&
        sensor->lastRx().isValid() &&
        sensor->lastRx().secsTo(now) < (60 * 60 * 24)) // if end device was active in last 24 hours
    {
        reachable = true;
    }

But, why not 24 h ? how many time you want to use ? remember some device can stay somes hours without notification, or can rejoin the network after a “break”.
I know some Legrands switchs can be marked as “unavailable” because they can stay 48h without notification)

Why not hardcoded ? You imagine the result if all users can use all possibles values ? with all possibles hardwares ?

You can use your third app to monitor your sensor, and trigger an alarm if a device don’t send a notification every hours, for exemple if you use it as alarm sensor. But the API is not for specific use, but generic. You can use “lastseen” or “lastupdated” value in the JSON.

[bcutter]

It’s so freaking annoying as it happened once again.

A temperature sensor died (out of battery) and I only noticed it now, 24 hours later when FINALLY the entities states switched to unavailable. INACCEPTABLE!

Have a look please:

I think I will need to continue trying to create a monitoring based on latest update of critical sensors in Home Assistant („if not updated within 3 hours, send an alarm“) because relying on deCONZ is simply not sufficient.

[Smanar]

Nothing to see but for information a battery under 90% is to monitor and under 80% to replace.
80% don’t mean 80% of his life, but 80% of its “capacity”.

Xiaomi battery stay above 90% during years.

[bcutter]

You might be right for Aqara devices. But it makes no sense to monitor the battery entities because they might work for MONTHS after going below 80 %.
It also strongly depends on what battery is used; the OEM ones behave like you wrote: some are still at 98 % after more than one year.

I want deCONZ to poll non-powered/battery-powered devices more often than once a day. That would be a solution, not one of many workarounds.

For powered devices it’s working just fine - pull the smart plug, after few minutes it’s reported as unavailable.

[bcutter]

And the next sensor died. Again it took me 24 + 2 hours to get notice of this.

Battery level was between 98 % and 100 % before that. Battery itself had 2.81 V (CR2032).

So obviously relying on the battery sensor of some devices it not reliable. That’s why a closer monitoring from deCONZ is necessary. deCONZ currently fails in that relation.

Unfortunately I don’t see any progress here. I don’t really remember WHY I argued FOR ConBee 2 and deCONZ and AGAINST ZHA and ZigBee2MQTT in several communities in the past. I guess I will need to rethink that position cause this really brakes a lot of trust in terms of reliability.

So once again:

@bcutter

I want deCONZ to poll non-powered/battery-powered devices more often than once a day. That would be a solution, not one of many workarounds.

Is this going to happen or not. Yes or no, it’s a binary thing. Thank you.

[Smanar]

On Xiaomi, it’s not a setting but it’s the device itslef that choose the report.
You have at least one by hour, I can be wrong but I think the battery level is updated too at this moment.

For me it’s strange the battery level move from 100% to 98% so often …

If you want to make something securised, check for values “lastseen” if this value > 1H it mean you have problem with the device.

[bcutter]

@Smanar

If you want to make something securised, check for values “lastseen” if this value > 1H it mean you have problem with the device.

I have no idea what “lastseen” is related to (deCONZ? HA?). But my plan is indeed to implement a monitoring in HA which checks the last sensor updates. So I’m going to implement what deCONZ is not able to too… thank you for nothing.

Meanwhile I had three more Aqara sensors found dead, of course always being reported by deCONZ 24 hours after the incident.

1. 2.81 V / reported as 98 %
2. 2.86 V / reported as 68 %
3. 2.87 V / reported as 95 %

So relying on the battery percentage entities is completely useless. If I must see 98 % already as “low or dead battery”, I can skip that completely.

That´s the reason why I need accurate reporting of unavailable devices by deCONZ. The gateway (ConBee 2) is the only/single point of knowledge/truth!

Unfortunately my question is still not answered here (@Mimiix):

@bcutter

So once again:

@bcutter

I want deCONZ to poll non-powered/battery-powered devices more often than once a day. That would be a solution, not one of many workarounds.

Is this going to happen or not. Yes or no, it’s a binary thing. Thank you.

[Smanar]

Here you have a complete json for a random device

  "4": {
    "config": {
      "battery": 68,
      "offset": 0,
      "on": true,
      "reachable": true
    },
    "ep": 1,
    "etag": "4dd42af9ba780e295e5d0db21e5d69d2",
    "lastannounced": null,
    "lastseen": "2022-04-23T17:40Z",
    "manufacturername": "LUMI",
    "modelid": "lumi.weather",
    "name": "Pressure 4",
    "state": {
      "lastupdated": "2022-04-23T15:43:26.147",
      "pressure": 968
    },
    "swversion": "3000-0001",
    "type": "ZHAPressure",
    "uniqueid": "00:15:8d:00:72:3f:38:41-01-0403"
  },

There is a value "lastseen": "2022-04-23T17:40Z" usefull to check dead devices, if this value is >1h for xiaomi device, it mean connexion issue (as Xiaomi have hourly report)

For your battery sample, IDK what happen because deconz use the same formula to compute the value and use value from the device itself too.
Remember some devices have voltage and battery percent, and other only voltage (and it’s deconz that compute the percent)

[Mimiix]

I am not able to answer this

@de_employees

[manup]

The config.reachable is a device specific matter, the rather large 24h timeout is due the fact that the legacy code to integrate devices was limited in knowledge about device details.

For example: there are switches which sleep for ever until a button is pressed, some sensors ping the gateway every x minutes, some once per hour. Without knowing this the plugin can’t determine when a (sleeping) device is actually not reachable.

With DDF we have now a way to express such information if needed, so this could be a good feature to add in future versions.

Beside the lastseen and state.lastupdated there is also a more detailed API output available via /devices/<device-mac-address> endpoint:

For example:

192.168.2.34:80/api/12345/devices/00:15:8d:00:01:0f:fd:66

{
    "lastannounced": null,
    "lastseen": "2022-04-24T18:47Z",
    "manufacturername": "LUMI",
    "modelid": "lumi.sensor_ht",
    "name": "Xiaomi round humi",
    "productid": "Aqara temp./hum. sensor WSDCGQ01LM",
    "subdevices": [
        {
            "config": {
                "battery": {
                    "lastupdated": "2022-04-24T10:27:37Z",
                    "value": 38
                },
                "offset": {
                    "lastupdated": "2022-04-21T23:39:03Z",
                    "value": 0
                },
                "on": {
                    "lastupdated": "2022-03-27T13:32:30Z",
                    "value": true
                },
                "reachable": {
                    "lastupdated": "2022-03-27T13:58:16Z",
                    "value": true
                }
            },
            "state": {
                "temperature": {
                    "lastupdated": "2022-04-24T20:47:04Z",
                    "value": -2094
                }
            },
            "type": "ZHATemperature",
            "uniqueid": "00:15:8d:00:01:0f:fd:66-01-0402"
        },
        {
            "config": {
                "battery": {
                    "lastupdated": "2022-04-24T10:27:37Z",
                    "value": 38
                },
                "offset": {
                    "lastupdated": "2022-04-21T23:39:03Z",
                    "value": 0
                },
                "on": {
                    "lastupdated": "2022-03-27T13:32:30Z",
                    "value": true
                },
                "reachable": {
                    "lastupdated": "2022-03-27T13:58:16Z",
                    "value": true
                }
            },
            "state": {
                "humidity": {
                    "lastupdated": "2022-04-24T20:47:04Z",
                    "value": 7920
                }
            },
            "type": "ZHAHumidity",
            "uniqueid": "00:15:8d:00:01:0f:fd:66-01-0405"
        }
    ],
    "swversion": "20160516",
    "uniqueid": "00:15:8d:00:01:0f:fd:66"
}

For most Xiaomi sensors the config.battery field is updated roughly once per hour, so the timestamp of this field could be used to determine if the sensor still has it’s connection.

A few thoughts about your sensors going silent:

In most cases where Xiaomi sensors stop sending it’s not the battery or a bad coordinator, but the fact that (old) Xiaomi sensors stubbornly stick to their parent, and don’t try to reconnect. They only work great if their routers are always on and don’t remove the sensor from the child table.

I think the T1 Xiaomi Aqara line has improved this, but I have only tested the switches so far.

I want deCONZ to poll non-powered/battery-powered devices more often than once a day. That would be a solution, not one of many workarounds.

The coordinator can’t poll sleeping end devices, while they are sleeping, they don’t answer after the initial pairing phase. Xiaomi sensors usually only wake up once per hour for a report if no other measurements are send in between, we call them “deep sleepers”. The sensor reports are one way “send only” so even if the sensor reports a temperature value, it ignores commands from the coordinator right after it. Good for battery, bad for communication.

There are other sensors which work way more robust and can even be reached by the coordinator since they wake up more often, e.g. Philips Hue motion sensor checks if it looses connection to the parent and reconnects automatically, it’s also a “light sleeper” waking up every few seconds and can be reconfigured.

But as mentioned earlier the biggest problem is that some devices don’t handle re-connection properly.

[bcutter]

@manup

But as mentioned earlier the biggest problem is that some devices don’t handle re-connection properly.

I don’t think that’s relevant here. Never seen that issue. I see hard coded 24 hours polling time. Let’s focus on that.

For now I have a monitoring (all Aqara devices not updated for 61 minutes) as a workaround in place in Home Assistant, thanks to @amelchio. Surprising a few lines of code achieve what deCONZ can’t. Now I hopefully will be notified pretty quickly (after 61 minutes) once e. g. a battery dies - which means winning 23 hours, almost one day compared to the deCONZ default.

[BabaIsYou]

As the saying goes, “There are none so deaf as those who will not listen.”
BTW, the fact that you were able to make this script is the best demonstration that Deconz is doing his job well!
ConBee II It is only a Zigbee coordinator, and Deconz a kind of coordinator, certainly not a monitoring system. These sterile discussions remind me of the same endless discussions about the frequency of temperature ratio or humidity reporting frequence of a battery-powered sensor.

The answer is always a compromise between what you want this sensor serves to, battery life and what the sensor is capable of doing. But no configuration will satisfy everyone…

[Smanar]

You are just using a script that make a notification if the device don’t send the hourly periodic report.
Some device like Legrand can stay somes days without report.

You are just using a device specification, the 24h delay for marking a sensor as unreachable is a choice, can use script to make a hourly monitoring, but it can’t be in standard for all sensors.

[bcutter]

@Smanar

You are just using a script that make a notification if the device don’t send the hourly periodic report.

Exactly. So why „just“: that’s pretty much what I initially wanted to achieve: know about connection issues / dead batteries quickly, not after loosing at least 24 hours of sensor data. Using the „lastseen“ attribute as a basis for this.

We can now discuss this workaround (which I won’t any further as I think it’s a pretty simple and great improvement on top of the default behavior) - or ideally find a solution for e. g. making this user configurable. Probably most users just don’t care, fine. Otherwise why not allowing a node specific „status update warning limit“ or sth. like that. Seeing how much points are presented on why things are not possible I think this will lead to nothing and I‘m tired of wasting even more time in fixing kind of strange deCONZ/Phoscon behavior. Maybe next time/instance of ConBee will be run with ZB2MQTT or even ZHA just to see what (other ) difficulties they are facing. For now I will probably live with the workaround for a few weeks seeing how it goes, better than loosing sensor data from time to time.

[bcutter]

Just as a note to myself: the workaround works perfectly as tested yesterday.

• Triggered 2 hours (my current sweet spot) after last update of one sensor (Aqara multi sensor with battery reported 78 % so already on the “watch list”)
• While all entities of the multi sensor device (as reported by deCONZ) took 22 more hours to finally report the “dead battery situation” - the default behavior of deCONZ

So happy this little hack to get a “pre-alert” works so smoothly. Best optimization I ever implemented to make Phoscon/deCONZ/ConBee II even better.

Anyway, I tend to not mark this post as solution, as my workaround is just a workaround - only working for me.

[edwin2023]

@bcutter.
Also willing to share the complete solution/workaround including the script?

[amelchio]

This is my command-line Home Assistant sensor that I shared with bcutter, obviously the IP address and the API key must be modified:

sensor:
  - platform: command_line
    name: Xiaomi missing
    command: curl -s http://192.168.0.80/api/3456789ABC/sensors | jq -r "[ .[] | select(.manufacturername==\"LUMI\") | select(.lastseen < \"$(date -u -d@$(($(date +%s)-3660)) +%Y-%m-%dT%H:%M:%S)\")] | map(.name) | sort | unique | join(\", \") "
    scan_interval: 600

[edwin2023]

Thnx!