I'm not sure you will be able to get much help on this issue as you are talking about custom code you added to the Agentry Java to do the LDAP authentication. It will be very hard to provide suggestions on possible solutions since we are not familiar with your custom code base.
I can say that my custom LDAP authentication code works the same under SMP3 as it did under Agentry 6. I only had to adjust for finding the configuration files in the new SMP3 locations. I did not need any additional certificates but your environment may require additional configuration.
One option would be to talk to your CP about engaging the services team to investigate and help troubleshoot your code.
Sorry,
--Bill