Amagire

ingesting asbestos probably does increase your risk of gastrointestinal tract cancers, according to various studies including this one I'm linking because it was specifically done on lighthouse keepers.

however, all the studies seem to be on people who had asbestos fibers in their drinking water, I didn't see any other sources of ingestion mentioned. the EPA limit on asbestos on drinking water is 7 million fibers per liter, which is, per this not terribly helpful pdf, something like 210 micrograms. so if someone drinks 2 liters of water per day they're getting about 0.15 grams per year if I didn't screw up the math.

I don't know how much mass a mortar typically loses into food; it seems like probably more than 0.15 grams/year but I honestly have no idea and it's probably very dependent on the exact type of pestle and how you use it and how often you use it... I can't fill in that part of the equation and give you an exact answer. At any rate that's the amount the EPA says is safe; the actual risk threshold is not really known.

(for comparison, bad drinking water can have 600 million fibers per liter, which if my math is holding up is like 13 grams a year, which has got to be more than your mortar loses. This is obviously not good but at least it shows it's within the range that people were tolerating without having obvious massive problems.)

this study fed rats 360 mg/day of asbestos without seeing any cancers; they only observed them for six months plus it's rats, so that's limited in its usefulness, but it does at least tell us that even absurd amounts of asbestos ingestion don't cause any kinds of toxicity besides the irritation/inflammation from the fibers.

there was an asbestos contamination incident in New York that maybe increased the risk of pancreatic cancer (but that's based off only 9 cases because it wasn't a massive elevation and pancreatic cancer isn't super common at baseline); they did not find any increase in other cancers. that was at concentrations up to 304.5 million fibers per liter.

so to the extent that I have any conclusion... the only likely relevant risk is GI cancer, and the risk isn't zero so you should not continue using that mortar, but it probably isn't catastrophically high either.

Having worked a fair bit with it in non-culinary capacities, serpentine is used to describe a comically broad range of greenish rocks, some of which contain meaningful quantities of asbestos some which do not, sometimes mixed up with an entirely different kind of greenish rock like soapstone, and it's not trivial at a glance to figure out the kind you've got especially from a finished product. I know a handful of old gemologist dudes who could probably offer a decent guess from a good enough picture, but their general advice would be against it, and it's generally a soft stone poorly suited for that job anyway.