Interacting proteins can contact with each other at three different levels: by a domain binding to another domain, by a domain binding to a short protein motif, or by a motif binding to another motif. In our previous work, we proposed an approach to predict motif-motif binding sites for the yeast interactome by contrasting high-quality positive interactions and high-quality non-interactions using a simple statistical analysis. Here, we extend this idea to more comprehensively infer binding sites, including domain-domain, domain-motif, and motif-motif interactions. In this study, we integrated 2854 yeast proteins that undergo 13 531 high-quality interactions and 3491 yeast proteins undergoing 578 459 high-quality non-interactions. Overall, we found 6315 significant binding site pairs involving 2371 domains and 637 motifs. Benchmarked using the iPfam, DIP CORE, and MIPS, our inferred results are reliable. Interestingly, some of our predicted binding site pairs may, at least in the yeast genome, guide researchers to assay novel protein-protein interactions by mutagenesis or other experiments. Our work demonstrates that by inferring significant protein-protein binding sites at an aggregate level combining domain-domain, domain-motif and motif-motif levels based on high-quality positive and negative datasets, this method may be capable of identifying the binding site pairs that mediate protein-protein interactions.