I would cease doing any business with the business partner immediately. Your organization must act quickly and decisively to demonstrate what practices it considers unacceptable for the sake of your other business partners. Also, be sure to check with your legal counsel to make sure you are not in a situation where you need to disclose the privacy breach to your customers.
You should revisit your agreements with the business partner and ensure that the legal documents reflect the acceptable practices of how you work with trading partners. But to be clear, you need to make an example of this business partner; not doing a simple background check is unacceptable.
As an information security professional, many business deals create significant risk to your own organization. You connect systems to partners that have insufficient controls and protections. But, ultimately, business will win out, and if you make too much noise, you run the risk of being perceived
As part of your overarching security program, I recommend communicating with the legal team and discussing the things you think are important to look at when doing diligence on an acquisition or other business deal. It's critical to do this before the deal is underway. If you do this early, then you are proactive. If you do this later, then you are in the way of a deal getting done. Which do you think will be better perceived in the executive suite?
Ultimately, the role of the security staff is to present the risks. Business people need to make the decisions as to whether the risks are justified when weighed against the reward of doing the deal.