I keep seeing these 2 (orphaned?) devices popping up from time to time:
When I restart deconz they are usually gone. Just deleting them also solves the issue for a while, but they occasionally return when I pair other devices days/weeks later.
Are they part of an existing device, or the remains of a deleted device?
Could they cause any issues or can I safely ignore them?
They are devices that were in your network but aren’t anymore. They try to join your network. You probably have them in your drawer somewhere.
So I guess they are safe to ignore?
Is there a way to make deconz forget/delete them forever? 
Remove their battery i suppose, if their your devices. I think @ebaauw has a similar issue with a neighbour having Tradfri lights they don’t use. Those lights join his network automatically without him wanting that.
Speaking of cleaning up: are there any methods of removing stale/duplicate/junk/orphaned records from the database? Or is DB maintenance performed automatically by deconz?
Afaik the database cleans its own when you remove stuff.
Clearing in the database was fixed a while ago, however I guess they show up again since these are reported by some router in the neighbor table. Hue lights for example are known to keep these entries for a very long time.