This is a bit confusing or misleading since the terms “Semite” as used in linguistics, ancient history, and bible studies has very little to do with the term “Semite” in relation to anti-Semitism.
In the former, it is about people speaking a language with certain characteristics, e.g. Babylonians, Nabataens, Amharer, and Phoenicians. Because of the specific linguistic characters of their languages, Hebrew and Arabic are also counted among semitic languages. Within this usage today, neither Arabs nor Jews are “Semites” but rather Hebrew and Arabic are Semitic languages.
While the term has its origins in linguistics, during the 19th century it underwent a distinct transformation in usage as well as content to, in the sense of “anti-Semitism”, mean a political agenda against the imagined dangers of the imagined Jewish race, where the latter emerged in the racist discourse as the main antagonist of the “Aryans”, another term which originated in linguistics and became to take on a completely different meaning, from speakers of indoiranian languages to the the pinnacle of the “European race”.
To understand this transformation, it is important to understand what the context is. The 19th century is marked by a huge shift in terms of paradigms on how to explain the world, especially in regards to such factors as nationalism, race, and science. To break it down to the essentials: The French Revolution and its aftermath accelerated a trend that had been forming for some time: God, in the broadest sense, became obsolete as an explanation/justification for why the world was it was. Neither political rule nor the fact that people e.g. in Micronesia were distinctly different in how they organized their society, lived their lives etc. etc. from people in Europe could be explained in a theological sense anymore.
And while there had been previous attempts to theorize about these things without the use of theological arguments / God as a strict fact – such as Kant and Voltaire debating whether humans have two different origins or just one –, the 19th century sees these ideas combined and investigated with methods and theories we today would classify as scientific, e.g. Mendel or Darwin.
Out of this endeavor to explain why people were different, soon emerged what we today understand as modern racism, meaning not just theories on why people are different but constructing a dichotomy of worth out of these differences. Jews as a group got also swept up in this trend and discourse.
While the exact process of how this happened and when it started is a long and complicated one, what is important for the context of the term “anti-Semitic” is that both in order to distinguish themselves from a religiously motivated othering of the Jews as well as to encompass their new world view, which out of factors such as nationalism and new philosophies of history (Hegel, Whigs, Marx), saw history as a constant conflict between races, adherents chose to utilize the terms “Semitism” as well as “Aryan”.
The use of these terms to mean that behind nations lay races and that the Jews not only constituted their own “race” but also that they were dangerous and on contrarian terms with the Aryan race, was intended to show that not only was this a new way to understand the world but also to lend themselves scientific credence. Heinrich von Treitschke, who popularized the term “anti-Semitism” in Germany, used it to argue that Jews, no matter how areligious they were and how “German” they had become in the manners how they lived their lives, were always different from the Germans and a danger to the national German character since they, as a people without a homeland, were comparable, in his mind, to parasites undermining “Germanness”.
Within this whole context, anti-Semitism whether as a self-descriptor, a name for a political movement (like Wilhelm Marr’s League of anti-Semites), or as the name used by people opposing these ideas, always meant exclusively Jews and not Arabs since like the term “Aryan” as it was used by the völkisch movements, it had been almost completely divorced from its original context and use in linguistics.
In essence, with the development of modern racism eschewing old categories and instead embracing scientific and pseudo-scientific explanations and categories not just for why people were different but also why they had different “worth”, a special context emerged for Jews, which while using terms and monikers with a different original meaning, developed their own connotations and meanings. Anti-Semitism refers to Jews exclusively because the people who coined the term as a political label and name for their theory used it to describe Jews exclusively and divorced it from its linguistic meaning.
Like the völkisch movement and the Nazis using the term “Aryan” as a descriptor for the German race and not in its original meaning of Iranians, anti-Semitism took on its meaning because the term was used to convey this meaning historically. That, combined with the fact that neither Jews nor Arabs are Semites strictly speaking but that Hebrew and Arabic are Semitic languages in a linguistic use, is the history behind the usage of the term to describe modern, race-based prejudice and hatred towards Jews.