Scammers have long since used bots already for text based scams (though dumb ones). Phone calls are a lot harder, though. And there’s also “pig butchering” scams, which are the long cons. Most commonly those are fake relationships. I suspect those long cons would have a hard time convincing someone for months, as human scammers manage to do.
I suspect that scammers will have a harder time utilizing AI, though. For one thing, scammers are often not that technologically advanced. They can put together some basic scripts, but utilizing AI is difficult. They could use established AI, but it’d almost surely be against their ToS, so the AI will likely try to filter scam attempts out.
That said, it might be just a matter of time. Today, developing your own AI has a barrier to entry, but in the future, it is likely to get a lot easier. And with enough advancements, we could see AI being so good that fooling someone for months may be possible. Especially once AI gets good at generating video (long con scams usually do have scammers video chat their victims).
And honestly, most scams have a hundred red flags anyway. As long as the AI doesn’t outright say something like “as a large language model…”, you could probably convince a non zero number of victims (and maybe even if the AI fucks up like that – I mean, somehow people get convinced the IRS takes app store gift cards, so clearly you don’t have to be that convincing).
The whole CSAM issue is why I’d never personally run an instance, nor any other kind of server that allows users to upload content. It’s an issue I have no desire to have to deal with moderating nor the legal risks of the content even existing on a server I control.
While I’d like to hope that law enforcement would be reasonable and understand “oh, you’re just some small time host, just delete that stuff and you’re good”, my opinion on law enforcement is in the gutter. I wouldn’t trust law enforcement not to throw the book at me if someone did upload illegal content (or if I didn’t handle it correctly). Safest to let someone else deal with that risk.
And even if you can win some case in court, just having to go to court can be ludicrously expensive and risk high impact negative press.