Why I believe AGI (Artificial General Intelligence) is impossible

There is a lot of discussion these days about "Artificial General Intelligence" (AGI), by which term a purely hypothetical phenomenon better described as "Technology-based Consciousness" is usually meant. I refer to it as "hypothetical," because, like "extraterrestrial intelligent beings," it is a category the existence, or even the possibility, of which remains to be demonstrated. In the case of  ETs, my gut instinct is that they exist but are rare. In the case of AGI, or better, my term, TBC, I believe, in the absence of evidence one way or the other, that there is no such thing, never has been anywhere in the universe, and likely never will be. 

My reasons, which are purely philosophical as opposed to scientific, center around what I think of as the "Star Trek transporter paradox." If such a transporter actually existed, there is no real reason why the person entering the "Send" station wouldn't simply remain in existence after the signal to recreate his scanned "image" was sent, to emerge as another version of him. Or, if the scanning were destructive, I have to ask, in what sense have you not just killed him? How do you actually know that the replica that emerges from the "Receive" end isn't an entirely different person, who just happens to have a mental image of the prior life of the "sent" version of "himself?" But what is the actual continuity from the point of view of the person that got in the transporter in the first place? I would argue that there is no way to know. The qualia of seeming continuity between one moment to the next could well be terminated for the person getting in; and the person emerging only has the illusion of continuity. There is simply no way to know. Very sophisticated philosophical treatments of the nature of consciousness and the perception of time, including some ancient but highly complex and subtle ruminations by Buddhist sages, have dealt with this issue, but I would argue that no one really knows what the real essence of the perception of continuous consciousness is. We only know that it seemingly emerges in minds, but, so far anyway, insofar as anyone can determine, only in minds of biological origin.

People will express the opposite view quite easily, but I am unconvinced. I have never perceived any reason to believe that a computer (such as ChatGPT, which I have interacted with a fair amount), which can be turned off so that it is an inert object like a brick, then turned back on, is in any way conscious. Even "a little." A system can mimic the externalities of an intelligent mind, which could lead you to infer the existence of consciousness, but the externalities are not actually evidence for any kind of subjective experience at all. I would argue, in fact, that there are indications to suggest that there is no subjective awareness whatsoever. As I said, my gut instinct is that there is simply no there there. The "smartest" computer is only a universal computing machine running an elaborate algorithm. The same may be true of our own minds; that can be argued forever too. But I remain unconvinced that a machine can be built that suddenly, and for no apparent reason, experiences emergent subjective consciousness. I just don't buy it. 

I suppose on some level it doesn't matter. If some biosphere-derived natural conscious being evolved somewhere and built a "machine civilization" entirely out of computer operated effectuators (spacecraft?) --that proceeded to self-replicate entirely without subjective experience, our experience of that "civilization," were we to encounter it, might be indistinguishable from an encounter with conscious minds. For that matter, solipsism aside, we wouldn't really know if an actual alien life form was conscious or merely a sophisticated algorithm, even if it were unquestionably biological. Our "theory of mind" gives us pretty good confidence that our fellow humans, and even animals more or less closely related to us, have some form of subjective consciousness, but beyond that, there is a vast forest of the unknown. And even that, I think, is an ever so slight-seeming leap of faith, because we can only assume that other instances of what appear to be minds actually have their own inner experience, entirely inaccessible to us, as individuals.

Regardless, these considerations have led me to the working assumption that artificial, technology-based consciousness is an intriguing, comprehensible idea, but one which does not correspond to any reality, existing at any time, anywhere.