Phoscon app not responding through service - Lost devices

Hi,

After some discussions on the Discord channel, I expose here my weird problem…

  • all was up and running. To be honest, can’t remember the versions of the Conbee 2 stick firmware, deconz and domoticz before
  • made an update of the Conbee 2 firmware to 26780700 (I did a backup through the app before, but the restoration ends with an error)
  • made an update of the deconz package to 2.19.01, now 2.19.03
  • noticed (I think few days after) that the Phoscon app wasn’t accessible anymore and that my Zigbee devices weren’t updating in domoticz. Seems to have worked few days after upgrades
  • when I (re)launch deconz via the command “sudo systemctl start deconz”, it’s not working: Phoscon app not accessible (website not responding on myIP:80), and the deconz process takes 80% of CPU
  • when I launch deconz via the command “/usr/bin/deCONZ -platform minimal --dbg-info=2 --dbg-error=2”, Phoscon app is accessible (website responding on myIP:80) but with no more devices, and the deconz process is not overloading the CPU
  • no more devices present in the Phoscon app: not sure where it stocks the information (deconz DB or the stick itself?). It is not impossible I reinitialized the stick when trying to debug, don’t remember…
  • checked that I do not have both deconz and deconz-gui running at the same time
  • checked that the Conbee 2 stick is visible (/dev/ttyACM0)
  • checked that deconz is listening on port 80 : tcp 3 0 0.0.0.0:80 0.0.0.0:* LISTEN 3423/deCONZ

@de_employees can you help out?

Juste noticed a difference between lauching via the service “sudo systemctl start deconz” and via the command “/usr/bin/deCONZ -platform minimal --dbg-info=2 --dbg-error=2”: there is 1 port listening in the first case (80) and 2 in the second case (80 and 8081).
In the first case, does it mean it can’t manage to launch the websocket?

First case : "sudo systemctl start deconz"
domoticz@raspberrypi:~ $ sudo netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 627/vncserver-x11-c
tcp 0 0 127.0.0.1:22222 0.0.0.0:* LISTEN 486/python3
tcp 4 0 0.0.0.0:80 0.0.0.0:* LISTEN 2323/deCONZ
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 10311/cupsd
tcp 0 0 0.0.0.0:55555 0.0.0.0:* LISTEN 630/sshd
tcp6 0 0 :::5900 :::* LISTEN 627/vncserver-x11-c
tcp6 0 0 :::8080 :::* LISTEN 709/domoticz
tcp6 0 0 ::1:631 :::* LISTEN 10311/cupsd
tcp6 0 0 :::443 :::* LISTEN 709/domoticz
tcp6 0 0 :::6144 :::* LISTEN 709/domoticz
tcp6 0 0 :::55555 :::* LISTEN 630/sshd

Second case : "/usr/bin/deCONZ -platform minimal --dbg-info=2 --dbg-error=2"
domoticz@raspberrypi:~ $ sudo netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 627/vncserver-x11-c
tcp 0 0 127.0.0.1:22222 0.0.0.0:* LISTEN 486/python3
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 17003/deCONZ
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 17003/deCONZ
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 10311/cupsd
tcp 0 0 0.0.0.0:55555 0.0.0.0:* LISTEN 630/sshd
tcp6 0 0 :::5900 :::* LISTEN 627/vncserver-x11-c
tcp6 0 0 :::8080 :::* LISTEN 709/domoticz
tcp6 0 0 ::1:631 :::* LISTEN 10311/cupsd
tcp6 0 0 :::443 :::* LISTEN 709/domoticz
tcp6 0 0 :::6144 :::* LISTEN 709/domoticz
tcp6 0 0 :::55555 :::* LISTEN 630/sshd

Here are my parameters for the first case:

/lib/systemd/system/deconz.service

[Unit]
Description=deCONZ: ZigBee gateway – REST API
Wants=deconz-init.service deconz-update.service
StartLimitIntervalSec=0

[Service]
User=1000
ExecStart=/usr/bin/deCONZ -platform minimal --http-port=80
Restart=on-failure
RestartSec=30
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_KILL CAP_SYS_BOOT CAP_SYS_TIME

[Install]
WantedBy=multi-user.target

/etc/systemd/system/deconz.service.d/override.conf

[Service]
ExecStart=
ExecStart=/usr/bin/deCONZ -platform minimal --http-port=80 --ws-port=8081 --auto-connect=1 --dbg-error=1

Strange, and if you remove the “override.conf” (just to test) ?

Same effect as “first case”…

Phoscon not responding

And for the ports, only 80 listening:

domoticz@raspberrypi:~ $ sudo netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 627/vncserver-x11-c
tcp 0 0 127.0.0.1:22222 0.0.0.0:* LISTEN 486/python3
tcp 10 0 0.0.0.0:80 0.0.0.0:* LISTEN 8250/deCONZ
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 10311/cupsd
tcp 0 0 0.0.0.0:55555 0.0.0.0:* LISTEN 630/sshd
tcp6 0 0 :::5900 :::* LISTEN 627/vncserver-x11-c
tcp6 0 0 :::8080 :::* LISTEN 709/domoticz
tcp6 0 0 ::1:631 :::* LISTEN 10311/cupsd
tcp6 0 0 :::443 :::* LISTEN 709/domoticz
tcp6 0 0 :::6144 :::* LISTEN 709/domoticz
tcp6 0 0 :::55555 :::* LISTEN 630/sshd

domoticz@raspberrypi:~ $ sudo systemctl status deconz
● deconz.service - deCONZ: ZigBee gateway – REST API
Loaded: loaded (/lib/systemd/system/deconz.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-12-05 17:16:04 CET; 4s ago
Main PID: 8250 (deCONZ)
Tasks: 3 (limit: 2059)
CGroup: /system.slice/deconz.service
└─8250 /usr/bin/deCONZ -platform minimal --http-port=80

déc. 05 17:16:04 raspberrypi systemd[1]: Started deCONZ: ZigBee gateway – REST API.
déc. 05 17:16:05 raspberrypi deCONZ[8250]: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to ‘/tmp/runtime-pi’
déc. 05 17:16:05 raspberrypi deCONZ[8250]: libpng warning: iCCP: known incorrect sRGB profile

I think I’m good for a fresh reinstall of deconz :frowning:
Already tried an apt-get reinstall, it didn’t resolve anything…

  • Do I still have a chance to find my devices back? seems to have my MAC adresses of my devices somewhere as in the logs of the second method they appear, but not in Phoscon
  • What is the best method to reinstall / reinitialize deconz?

Juste found something else weird, don’t know if existed before, but the timestamp let us know that it has been accessed these days (surely the difference between the two launch methods):
image

Will have to test which method links to which directory…
And by the way, more than 800MB seems way to big no?

800MB looks like a bug. I bet this is the file where your devices are stored? Thats possibly also the reason the backup won’t work. The devices should return to deCONZ and the app though when you still have the same network settings of your zigbee network and when you open your network (start light search).

Do you updated deCONZ from an very old version to 2.19.1 ? There was a bug in old versions that could possibly make many double entries.

I think I had the 2.17.1 version previously.

I have to check if the devices are present and maybe have multiple entries.

For the small DB in domoticz user, I think it came when I tried to change the user from 1000 to 1001 to test if it worked better, but it didn’t :frowning:

Could you also check if the zcl_values table has an enourmous number of entries and
in the config2 table the zclvaluemaxage value should be 0

Well, no entries at all in the zcl_values table.
In the config2 table, the zclvaluemaxage value is 0.
My biggest table is sensors, which make sense as I have entries since multiple years for multiple devices.

In the same directory, I have a deconz.tar.gz created on 29th november, same date of my Conbee 2 backup made by the Phoscon app that I cannot manage to restore.

My bad, seems corrupted, I can’t access the data in it :frowning:

@Mimiix / @Smanar / @ChrisHae
Thank you for trying, appreciate it.
What do you think, given the problems I have, it may be better that I reset everything?
If so, do you have a procedure to do this properly (reinstalling deCONZ from scratch)?

Yes the deconz.tar.gz is the backup file and contains the DB and the network configuration.
You are the 2nd user i know with this problem. Would you mind to send your zll.db and the .gz to support@dresden-elektronik.de for further examination?

Have you access to the deCONZ GUI or the Phoscon App? The you should copy your network settings. With the same settings you should be able to restore most of your network.

Or perhaps you should try a fresh install of our phoscon image to test if the ConBee2 still works. You should be able to restore your network as the network settings are saved on the stick and don’t get lost even if you upgrade the FW. Groups or Rules are lost though.

I will try to answer each point, sorry if I misunderstand some :frowning:

Would you mind to send your zll.db and the .gz : it is possible for the .gz (but corrupted: I cannot un tar it), but as mentionned, the zll.db is way to big (800MB)

Have you access to the deCONZ GUI or the Phoscon App : I can manage to access the Phoscon app when I launch deCONZ manually, but not by the service (see above). I cannot manage to launch the GUI, even manually (maybe because of the DB size?)

You should copy your network settings. : could you be more precise? I don’t know where I have to do that? Moreover, this won’t make the deCONZ service work?

Try a fresh install of our phoscon image to test if the ConBee2 still works. : I install Phoscon via the deCONZ package (using the apt-get install deconz command). I do not know how to install an image? Moreover, when I access the Phoscon app when launching it manually, I have no more devices BUT when I press on the detection button, it manage to pair very few devices without any action on the devices itself (I didn’t press the pairing button). It concerns some plugs. This makes me think that I do not have any problem with the Conbee 2 stick.

Restore your network as the network settings are saved on the stick and don’t get lost even if you upgrade the FW : where do I found those settings? Still won’t bring the deCONZ service back :frowning:

Groups or Rules are lost : I do only pairing in Phoscon, I had only groups automaticaly created by Phoscon. All my scripts are in Domoticz, using a plugin to fetch deCONZ data. It should not be a problem for me.

Thank you again for your help.

  1. Right the DB file is far too big to send it
  2. I also think the ConBee2 is OK. I think it is the overall system that make the problems here. I just tested on a RPI pi with an very old install base. I don’t know the exact confgiruation but deCONZ was versio 2.10.4. I had also problems to launch deCONZ in GUI mode with the command or the service. So I installed the newest Phoscon Image from our homepage (wrote the image to the SD card with a card writer - belanaEtcher or win32diskImager - And I can confirm that everything works.

So I recommend a fresh install or update your whole system, uninstall deCONZ, check if there are any running processes (ps -aux | grep deCONZ) and reinstall it.

  1. The network settings are stored on the Conbee2. If you get your system running the devices should rejoin automatically when you open your network. A power cycle of bulbs or pressing a button on a sensor or switch would help to wake them up. If you have access to the GUI you can also have a look at those settings.

Thank you very much for the explaination.
I can’t recall, but it is possible I cliked on reinitialize in the Phoscon app, hope it didn’t erase everything.

To uninstall deCONZ, is a “sudo apt-install uninstall deCONZ” enough or do I have to erase manually the zll.db (or anything else as my override.conf file)?