Artificial Intelligence (AI) has evolved rapidly, redefining the interaction between humans and technology. The growing adoption of autonomous agents, capable of making decisions and performing tasks without direct human intervention, raises fundamental questions about ethics, security, privacy and governance. Given this scenario, it is essential to analyze not only the benefits of these systems, but also the challenges they pose to society and technological regulation.
The rise of autonomous AI agents
Autonomous AI agents are transforming industries and driving efficiency gains. Models such as Arbitrus.ai, an autonomous arbitration platform, demonstrate the potential of AI-based systems to replace traditional processes in sectors such as the judiciary and business. These agents follow rules and parameters programmed by their creators, operating in specific domains of knowledge. However, although they perform complex tasks independently, they lack the broad and flexible understanding of the world – such as that of a human being –, limiting their adaptation and ability to innovate in the face of unexpected scenarios. Hence the debate: what are the impacts of this limitation on the reliability of systems and social well-being?
In this context, Large Language Models (LLMs) – which are the foundation of modern AI – illustrate this limitation. While they can identify patterns and generate coherent responses, they are not able to verify the veracity of the information provided and often exhibit algorithmic bias or even “hallucinations” (incorrect responses that appear convincing).
The future, however, is promising. Deloitte predicts that by 2025, 25% of companies using generative AI will launch pilots or proofs of concept for autonomous AI agents, rising to 50% by 2027. The agentic AI market is expanding rapidly, with investors pouring more than $2 billion into startups in the sector in the last two years. Meanwhile, large technology companies and cloud providers are also developing their own solutions, seeking to increase the productivity of knowledge workers and automate complex workflows.
However, the adoption of these agents is not limited to large corporations. Critical sectors are already using AI to automate repetitive tasks and execute critical processes, requiring a detailed analysis of the risks involved. Hospitals, for example, are beginning to employ AI to process medical records, analyze complex exams and even suggest preliminary diagnoses, reducing the burden on healthcare professionals.
It is essential to distinguish between semi-autonomous agents, which assist human decision-making without completely replacing it, and fully autonomous agents, which operate without supervision and defy human control over their actions. Some envision the evolution of these agents towards Artificial General Intelligence (AGI) – systems capable of replicating cognitive flexibility and performing virtually any intellectual task. However, this truly generalist AI remains in the realm of hypothesis: despite advances in current models, there are still technical and conceptual barriers that separate today’s agents from autonomy comparable to that of humans.
Given this scenario, it is essential to analyze the advancement of these technologies in a multidisciplinary way, considering not only the technical challenges, but also the social and regulatory implications of their development. The impact of fully autonomous agents is not limited to the technological sphere, but can resonate broadly across society. James O’Donnell, in an article published in MIT Technology Review Brazil, highlights the importance of an in-depth debate on the ethics of using these systems, warning of the risks that can go beyond engineering barriers and directly influence fundamental aspects of governance, security and human rights. Thus, while the search for more sophisticated systems continues, it is essential that their development occurs responsibly, balancing innovation with adequate control and oversight mechanisms.
Main risks of full autonomy
The progressive elimination of human oversight could lead to serious problems, including persistent errors, security vulnerabilities, and adverse social impacts. Some of the main challenges associated with developing fully autonomous agents include: Inaccuracy and inconsistency: Modern AI models, especially LLMs, are still subject to significant limitations. Furthermore, examples such as Devin, launched by Cognition Software in 2024, show that even the most advanced agents still have limitations. Devin was able to solve 14% of GitHub issues autonomously, twice the accuracy rate of LLM-based chatbots, but still far too low to fully replace human software engineers.
Privacy and security: Autonomous agents require large volumes of data to operate effectively, expanding the attack surface for cyberattacks and increasing the risk of sensitive information leaks. Without proper regulation, there is significant potential for data abuse and compromise of user privacy. On the other hand, overly restrictive regulation can limit the capacity for innovation. Furthermore, the use of autonomous agents in the military raises significant concerns. The IEEE report points out that autonomous weapons systems can act without human intervention, representing an unacceptable risk to military operations and international security.
Flexibility and unpredictable actions: The flexibility of AI agents can bring both benefits and risks. Highly flexible systems can facilitate the automation and optimization of processes, but they also become more susceptible to unintended or malicious actions, such as the execution of harmful code and improper financial transactions.
Spread of misinformation: The risk that autonomous agents can amplify false information is undeniable. Combined with imperfect models, there is great potential for the dissemination of misinformation on a large scale, which can undermine public trust and negatively impact society. One consequence of this would be the ratification of the theory of the Dead Internet, a digital reality where the authenticity of human interaction on the internet is deeply questioned.
Impact on fairness and algorithmic bias: AI agents can perpetuate prejudices and social inequalities if their training models are biased. This is especially problematic in autonomous decision-making systems in sectors such as healthcare, justice and the financial market.
Over-reliance on autonomous systems: There is also a risk of undue trust in autonomous agents, leading to uncritical acceptance of their decisions. Over-reliance on these systems can undermine human oversight and result in unpredictable consequences.
Semi-autonomy models as a viable alternative
Implementing a hybrid model, which combines automation with human supervision, can be an alternative to try to balance innovation and control. Semi-autonomous solutions can increase efficiency without compromising safety and accountability. In the paper Fully Autonomous AI Agents Should Not be Developed, Mitchell, Ghosh, Luccioni and Pistilli argue that continuous human supervision is essential to mitigate risks from advanced AI.
These models can be implemented in different ways, depending on the complexity of the application and the level of risk involved. Some of the most common approaches include: (i) active supervision, in which human operators monitor automated decisions and can intervene in real time when necessary; (ii) use of iterative adjustments, in which the system learns from human corrections, adjusting its decision-making to improve accuracy over time; (iii) escalation mechanisms, so that in cases of high-impact decisions, the agent can escalate the issue to a human supervisor before acting; and (iv) scope limitation, in which restrictions are incorporated to prevent the agent from making decisions outside his/her area of competence or that may pose critical risks.
Semi-autonomous agents operate within a shared control model, in which automation is combined with human supervision to ensure greater reliability and security. This balance allows decisions to be processed quickly and efficiently, but with the possibility of human intervention whenever necessary. By maintaining supervision at strategic points in the process, the risk of unpredictable actions or systemic failures that could compromise operational integrity is significantly reduced. In addition, this model favors dynamic adaptation to changes in the environment, ensuring that agents can operate efficiently even in uncertain or unexpected scenarios.
The continuous presence of human supervision also plays a fundamental role in mitigating cyber risks. By maintaining a level of manual control over operations, semi-autonomous systems reduce the attack surface that can be exploited by external threats, making it difficult for intrusions, manipulations, and the leaking of sensitive data. Similarly, the possibility of human intervention prevents unintentional errors from resulting in critical impacts, especially on essential infrastructures such as communication networks, financial systems and public service platforms. Another direct benefit of this model is the improved auditability of automated decisions, ensuring that failures can be identified, analyzed and corrected before they cause significant damage.
The advancement of AI must be accompanied by regulations and safety standards that ensure transparency and governance of these systems. In the European Union, for example, there is the AI Act, which proposes requirements proportional to the level of risk of each AI application and provides for mandatory human supervision in high-risk contexts. In parallel, strategies such as regulatory sandboxes are emerging, which allow for controlled experimentation of innovative autonomous agents in supervised environments. This approach seeks to collect evidence and lessons learned to guide future regulations without stifling innovation during the early stages of development.
In addition to guidelines and standards, concrete technical measures are needed, such as the implementation of specific audit frameworks for autonomous agents, mechanisms for traceability and explainability of decisions, as well as clear protocols that allow for rapid and effective human intervention in critical scenarios. Continuous monitoring of these agents is also essential to identify and correct biases or inconsistencies in AI models, preventing automated decisions from resulting in discrimination or other unintended consequences.
Next steps
The future of AI requires a balanced approach between innovation and responsibility. Developing fully autonomous agents without supervision risks compromising safety and ethics. The solution lies in proper governance, continuous monitoring, and the implementation of human oversight mechanisms. Only then will it be possible to ensure that AI remains a tool for human progress – and not a risk to society.
( fontes: MIT Technology Review)
