Automated Decision-Making in Screening: AI and the Limits of Judgement in Governance

Artificial intelligence (AI)–enabled tools are increasingly used across employment, compliance, and risk-management contexts to support screening and preliminary decision-making. Organisations often adopt these tools to manage scale and volume, particularly where large amounts of information must be processed efficiently. However, these systems were never designed to exercise judgement, assess whether someone is suitable in their specific context, or provide ongoing assessments of risk. Instead, they screen and sort information by identifying patterns in existing data (for example, ranking applications or flagging cases for further review). These tools operate within broader decision-making processes that still depend on human judgement and oversight (Mittelstadt et al., 2016).

The discussion below focuses on commonly used decision-support tools, such as automated shortlisting, risk-flagging, and text-analysis systems, rather than autonomous or experimental AI technologies. Understanding the limits of these tools is not a critique of their use, nor of technological innovation more broadly. Instead, it supports proportionate, evidence-informed governance in environments where trust, accountability, and potential harm intersect.

What AI-Based Screening Systems Are Designed to Do in Risk Assessment

AI-based screening systems, as discussed here, refer to automated tools used to support preliminary assessment, prioritisation, or triage in decision-making processes. These systems are usually deployed early in a workflow to help manage large volumes of information, not to replace professional judgement.

Examples include:

• automated recruitment tools that rank applications for further review

• systems that flag potential compliance or safeguarding risks based on predefined indicators

• text-analysis tools that scan written material for patterns associated with elevated risk

In each case, the system produces classifications, scores, or alerts that are intended to inform subsequent human consideration. These tools are commonly described as decision-support systems, rather than autonomous decision-makers (Bennett Moses & Chan, 2018).

Their function is to sort and highlight information based on learned statistical associations. They do not evaluate context, assess intent, or exercise normative judgement. As a result, AI-based screening systems are effective only within clearly defined and limited purposes. They are not designed to account for unrecorded information, changes in circumstances over time, or the relational and ethical considerations that underpin high-trust decision-making environments.

Bias and Risk in AI-Based Screening Systems

Bias in AI-based screening systems has been identified in the scholarly literature as a predictable outcome of data-driven design, rather than an isolated technical fault (Mittelstadt et al., 2016). Because these systems learn from historical data, they may reflect and reproduce existing social, economic, or institutional inequalities embedded within that data. Further risks arise when automated screening systems rely on proxy variables (indirect indicators used to draw conclusions about a person). Examples may include location, educational background, language patterns, or employment gaps. When these variables are built into systems that are difficult to explain or understand, it becomes harder to question or challenge the outcomes they produce (Bennett Moses & Chan, 2018). These limitations do not negate the usefulness of AI-based tools for defined, lower-risk purposes. However, they do limit the weight that can reasonably be placed on automated outputs, particularly where decisions affect individuals’ rights, opportunities, or access to trusted roles.

Why AI-Based Systems Cannot Replace Human Judgement

Automated classification and human judgement are conceptually different.

Judgement involves contextual reasoning, ethical evaluation, and the ability to interpret ambiguity, capabilities that automated systems do not possess (Mittelstadt et al., 2016).

Large language models (LLMs), such as widely used tools like ChatGPT, illustrate this distinction clearly. These systems generate text by predicting statistically likely sequences of words based on patterns in large volumes of existing language data. While their outputs may appear fluent, confident, or authoritative, they do not reflect understanding, intent, or evaluative judgement in the human sense. In practice, these tools can be useful for drafting, summarising, or exploring information. However, their outputs remain probabilistic rather than evidentiary. For this reason, they should not be treated as verified information or used as standalone bases for decisions that carry legal, professional, or safeguarding implications (Bennett Moses & Chan, 2018).

Governance Implications of AI Screening and Verification

As with other screening mechanisms, over-reliance on AI-based tools can create a false sense of assurance. When automated outputs are treated as determinative rather than indicative, opportunities for professional judgement, contextual assessment, and early identification of emerging risk may be reduced (Australian Law Reform Commission [ALRC], 2023). Automated decision-support tools are most effective when embedded within layered systems of accountability, transparency, and human oversight (Bennett Moses & Chan, 2018).

Verification remains essential where decisions affect people’s rights, reputations, or access to vulnerable populations. Although these issues are especially significant in high-trust environments, they apply to any setting where automated tools influence decisions with serious personal, legal, or organisational consequences.

Limits of AI in Screening and Decision-Making

AI-based screening and decision-support tools can play a legitimate role in organisational decision-making. However, these systems were never intended to replace human judgement, contextual assessment, or ongoing evaluation of suitability (Mittelstadt et al., 2016; Bennett Moses & Chan, 2018). Recognising the design limits of automated systems allows organisations to move beyond assumptions of objectivity and toward proportionate, evidence-informed governance. In practice, safeguarding is strongest when technological tools are understood as one safeguard among many, rather than as single points of assurance.

References

Australian Law Reform Commission. (2023). Confronting complexity: Automated decision-making and administrative law (ALRC Report No. 141).

Bennett Moses, L., & Chan, J. (2018). Using big data for legal and law enforcement decisions: Testing the new tools. University of New South Wales Law Journal, 41(2), 643–678.

Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 1–21. https://doi.org/10.1177/2053951716679679

→ Why Screening Is Not the Same as Ongoing Risk Assessment

These limitations are particularly relevant in screening contexts, as explored in our article on why screening is not the same as ongoing risk assessment.

→ The Cost of Assumptions

They also reflect broader challenges in relying on incomplete information, as discussed in our article on the cost of assumptions and the value of verification.

→ When Trust Becomes a Blind Spot

This intersects with how trust and oversight evolve over time, as examined in our article on when trust becomes a blind spot in risk and governance.