This is more a linguistics question than a historical one exactly, but I’ll give it a shot. As you note, the word “synagogue” is Greek. And just as you say, Greek was a common language of the Eastern Mediterranean in ancient times.
There were communities of Greek-speaking Jews in ancient times, too. From there the word passed into Latin and other European languages. It’s used many times in the Christian bible, which would keep the term in currency even in parts of Europe without Jewish communities.
However, many Jewish communities didn’t necessarily use the term “synagogue” at all. Ancient Jewish texts in Hebrew/Aramaic tend to use בית־הכנסת (roughly meaning “assembly place”).
This is the term in Jewish texts in Hebrew and Modern Hebrew. Derivatives of this term also were widely used in Middle Eastern Jewish communities, such as “Kanis”. The first attestation of written Yiddish uses בית־כנסת as well. Ashkenazi communities generally used “shul”, which is cognate with “school” in English (and also means “school” in Yiddish).
Jews from Spain and Portugal generally did use a derivative of “synagogue”, such as “esnoga”, though. And of course, it’s used in English. While many Jewish religious terms are borrowed directly from Hebrew, many aren’t, so it’s not that unusual.
The Yiddish “bentsh”, meaning “bless”, still used widely in Ashkenazic Jewish communities, is ultimately Latin. Yiddish words for praying aren’t Hebrew either (the most common, “daven”, has no obvious etymology–but Western Yiddish “orn” is Latin).
The now-obscure English word “porge” for removing forbidden fats from meat is from Judeo-Spanish (nowadays various forms of the Hebrew “nikkur” are usually used), related to “purge”.
In general, though, the trend over the past century or so has been towards more Hebrew/Yiddish. It’s pretty unusual to refer to Jewish holidays by translated names, for example, but it wasn’t 100 years ago. As noted above “nikkur” has replaced “porging”.
“Synagogue” has mostly resisted this trend. I say “mostly” because using “shul” in English has gotten more common over time (at least, that’s my perception). I don’t think there’s an obvious answer why. I suspect that the reason is that the term is neutral. Reform Jews (and many Conservative Jews) refer to the synagogue as “Temple”, a trend that began in 19th century German Reform Judaism.
While people who use the term are usually not consciously choosing it for theological reasons, it does have theological undertones for many Jews and is often thought of as Reform-specific (it’s also used by Italian Jews, but not for that reason and without that connotation–the Great Synagogue of Rome is “Tempio Maggiore” in Italian).
Terms like “shul” are Ashkenazi specific, may be thought of as suggesting an Orthodox synagogue, and aren’t necessarily familiar to all English-speaking Jews. Plus “synagogue” is much more entrenched as an English term than, say, the English names of Jewish holidays.
So, “synagogue” sticks around in English as the default. Jews among themselves might use “temple” or “shul” or “shil” (the same as shul but in a different Yiddish dialect) or “bet knesset”, but “synagogue” still has enough currency to stick around, and the Hebrew or Yiddish alternatives aren’t widely enough used or known to replace it.