Counterfeit agricultural products pose significant challenges for conventional classification models like CLIP, particularly under few-shot learning scenarios due to subtle visual ambiguities. This study introduces RAC (Reducing Ambiguity in CLIP for Counterfeit Detection), a novel framework designed to identify and mitigate these ambiguities through an active refinement process. Specifically, RAC employs a Multilayer Perceptron Plus (MLP-P) module to enhance discriminative power by effectively synthesizing multi-level visual representations derived from CLIP’s image encoder. Subsequently, the Inter-class Ambiguity Reduction (IRA) module suppresses specific visual patterns responsible for confusion between authentic and counterfeit products. Finally, a Network Fusion (NF) stage dynamically combines outputs from these enhanced pathways to ensure robust classification outcomes. Experimental validation demonstrates that RAC markedly improves detection accuracy, achieving an average of 76.95% in 16-shot learning across four benchmarks, surpassing the state-of-the-art by 2.50%. On the specialized TFS-Fruit dataset, the framework reaches 80.0% accuracy, confirming its efficacy in critical agricultural applications.