Recently the prevailing language-as-resource metaphor has been problematised and theorised. Using the philosophical theory of inferentialism, we trace an epistemological dimension of multilingualism in mathematics education and add it to the current language-as-resource discussions. With data from two different settings--a mathematics classroom in Sweden and a workshop in an indigenous settlement in Colombia--we show that in encounters between language practices and plural mathematics, the semantic and the epistemological are two sides of the same coin. Inferentialism captures such encounters without dichotomising either languages or mathematics. We contend that epistemological issues move beyond the scope of language-as-resource approaches, but they are not paths to improving school achievement. Neither are they matters of distinguishing between formal and informal language use. Rather, an epistemological dimension is about shaping meta-understandings of language diversity that are liberated from mathematics as fixed and pre-stablished knowledge.