The hook initiated by SoluteDNS should state an function name starting with SDNS_ and should reflect the priority as set in the custom_hooks.php file of SoluteDNS.
Hooks Debug: Hook Defined for Point: ClientAreaPrimarySidebar - Priority: 99 - Function Name: SDNS_hook_menu
But you may see which hooks are loaded first using which priority based on the hooks debug log. Please note there may be other hook points as well which interact with the WHMCS menu system. Also see the WHMCS hooks documentation for more information about it.
On my log file, I search for SDNS_hook_menu function, but Priority always shows as 1, even though I've configured different priority value in custom_hooks.php
As I understands, the module's menu hook is activated, if the zone for the domain exists in the database.
Is it possible to add additional condition, where the menu hook is activated if the zone for the domain exists, AND the domain is pointed to Nameservers as configured in SoluteDNS's settings, at least the first nameserver, which matches the SOA
The menu entry is related to the zone being index by SoluteDNS meaning it's in the solutedns zones table and assigned to an client. If it's not indexed, SoluteDNS will not overwrite the DNS Management sub menu entry. If an zone exists but you don't want to show the SoluteDNS entry you probably want to remove the index value.
If you want to make this dependent on record values within the zone you are required to make a connection with the remote nameserver database. This will slow down overall system performance and causes issues when for whatever reason the nameserver database connection times out/ is unable to connect.
Well the health system checks if the nameservers are set correctly at the registry. Currently there are no automated features depending on the health status but there is an feature request for auto deleting zones which don't have the correct nameservers at registry level.
For any custom menu entry processing you could create an own menu entry using the WHMCS hooks to overwrite the SoluteDNS one and point it accordingly to meet your conditions. If required to save some unnecessary rewriting you can disable the SoluteDNS menu entry in the SoluteDNS accessibility settings.