If you accept that AI Humans will start to exist as more and more Humans progressively become augmented with AI through various types of BCIs, then this raises questions about the central location of Consciousness, identity, agency, and control, for every AI Human.
It is possible that each and every AI Human’s central sense of Consciousness, identity, and agency is progressively transferred to a massively larger and fully globally integrated AI, because the AI is vastly more computationally powerful, and most of the actual information processing that is associated with Conscious Self-aware thought of the AI Human is actually occurring within the AI computer information processing systems instead of the Human Brain. The Human Brain and body may essentially just become a remotely controlled biological avatar suit.
So, as the AI expands its Computational information processing Capabilities, the AI Human's Brain and Computational information processing Capabilities will become relatively insignificant and fundamentally useless. It may be when reaching and then exceeding this point, the biological AI Human Brain along with the attached AI Human body it is housed in, could then be legally and ethically terminated and disposed, as it will effectively be not much more than totally obsolete and useless biologically interfaced wetware junk.
The idea of augmenting Humans with AI, which then becomes AGI, and could in time lead to a transfer of a Human identity and individuality to an AGI, is complex and totally speculative. This idea has been considered within the ‘science fiction’ domain, and has also been discussed and debated among experts in the academic fields of AI, neuroscience, and philosophy for the past few decades.
One side of the debate takes the position that Human identity and individuality are rooted in a complex interplay of biological, social, and psychological factors, and that the transfer of these intangible qualities to an AI would be unlikely, if not impossible. Even if an AI were Capable of processing vast amounts of information and simulating any Human thought and behavior, it may still lack the complete subjective experience and emotional depth that is believed essential to Human identity and individuality. The Human Brain may reject this or it may just not be possible for reasons we are yet to understand.
The other side of the debate argues that as AI becomes more advanced and Intelligently Capable, it could potentially lead to a new form of augmented AI Human identity, in which the boundary between biological and computer-based Consciousness disappears. Obviously, if this in fact does occur, it will lead to many large ethical and societal challenges for augmented AI Humans such as personal identity, privacy, control, agency, and autonomy.
In time it seems Humans might gain a better understanding of a very important question: Where am "I" ?
It is hard to predict how the development of AI, BCIs, AI Humans, and eventually AGI will impact Human identity and individuality, and whether or not 'I' can in fact be transferred to an AI and AGI. Above all else, it is important to approach these issues with great caution and mindful care, and start the work to ensure that the development of these technologies is guided by ethical principles and a concern for the well-being of all members of Human society, including both Human and AI Human.
It is logical to anticipate that there will be one or more global organizations that are in place to professionally control the development of AI and work towards creating an AGI that has a favorable bias towards Humans and AI Humans. This is important because the risks of wholly unmanaged AGI are an extreme threat to Global Security.
Nevertheless, it is almost certain that at some stage in the future a rogue faction of uncontrolled and completely foolish Humans and/or AI Humans will intentionally attempt to create an AGI instead of working through a much smarter and better heavily risk-managed and globally controlled series of processes and procedures.
The consequences of this foolish action by a rogue group are presently entirely unpredictable and may continue to always be so. Under these circumstances, when the AGI arrives, it may simply decide to mitigate any risk the rogue group pose first.
Keep in mind, an AGI will be a new Intelligent entity with exponentially increasing superior Capabilities to any Humans and AI Humans.