Full Text
1. Introduction
Artificial intelligence has moved from experimental research
environments into everyday organizational and public infrastructure.
Predictive models support fraud detection, credit assessment, medical
image analysis, recommendation, recruitment, content moderation,
logistics and the allocation of public resources. Many of these systems
are based on machine learning methods that infer patterns from data
rather than relying exclusively on rules written by domain experts.
Their ability to capture complex relationships can improve predictive
performance, but complex models may be difficult for users, developers
and affected individuals to understand.
The tension between predictive performance and interpretability has
become a central issue in responsible AI. A model may achieve high
accuracy on a test dataset while providing little insight into the basis
of an individual prediction. In a clinical setting, a clinician may need
to know whether a risk score is driven by a clinically meaningful factor
or by an irrelevant artefact in the image. In credit assessment, an
applicant may need a meaningful explanation of an adverse decision and
information about what could be changed. In public administration,
officials need to understand whether a model has systematically
disadvantaged a group or relied on an inappropriate proxy.
Explainable artificial intelligence refers broadly to methods and
practices that make the behaviour, outputs and limitations of AI systems
understandable to relevant stakeholders. The field includes
intrinsically interpretable models, post hoc explanation techniques,
visualization, counterfactual reasoning, model documentation,
human-computer interaction and governance. Doshi-Velez and Kim (2017)
argued that interpretability should be studied through rigorous
evaluation rather than treated as an intuitive label. Lipton (2018)
similarly showed that the term interpretability is used to describe
several distinct goals, including transparency, decomposability and post
hoc explanation.
The demand for explanation is not simply a response to technical
complexity. It is connected to longstanding principles of scientific
reasoning, professional responsibility, procedural justice and
democratic accountability. Explanations can help people detect errors,
assess whether a system is appropriate for a task, learn from model
behaviour and decide whether to rely on a recommendation. Yet
explanations can also mislead. A plausible account may not faithfully
represent the model's computation, and a visually appealing explanation
may create confidence without improving understanding. Rudin (2019)
therefore argued that, in high-stakes settings, researchers should
consider using interpretable models instead of explaining black-box
models whenever feasible.
This review examines XAI as a technical and institutional discipline.
It asks: (1) How should explainability and interpretability be
understood? (2) What principal families of explanation methods are
available? (3) How are these methods applied across
information-intensive sectors? (4) What technical, ethical and
organizational limitations affect their usefulness? (5) How should
institutions evaluate and govern explainable systems? The analysis
focuses on literature published by 31 December 2022, so that the
discussion reflects the research foundations available entering
2023.
2. Review Methodology
An integrative literature review was used to bring together technical
and conceptual contributions from machine learning, computer science,
information systems, human-computer interaction, law and ethics. The
field of XAI is methodologically diverse. It includes mathematical
methods for attributing predictions, studies of human understanding,
empirical assessments of explanation interfaces, normative analyses of
accountability and applied work in specific domains. A single
quantitative synthesis would not be appropriate because the reviewed
studies address different models, datasets, users and evaluation
outcomes.
The review scope was limited to literature available on or before 31
December 2022. Search concepts included explainable artificial
intelligence, interpretable machine learning, model interpretability,
feature attribution, local explanation, global explanation,
counterfactual explanation, saliency maps, algorithmic transparency,
explanation evaluation, fairness and human-AI interaction. Priority was
given to peer-reviewed journal articles, major conference papers,
established books and widely cited technical or policy reports with
traceable publication details.
The selected literature was organized into five themes: conceptual
foundations; technical explanation methods; applications; evaluation and
human factors; and governance, ethics and risk. Within each theme, the
review compares the assumptions and limitations of different approaches.
It distinguishes methods that explain a model's internal structure from
methods that approximate or probe its outputs. It also distinguishes
explanation quality from model quality: a model can be accurate but
poorly explained, and an explanation can be understandable but
unfaithful.
The review is integrative rather than exhaustive. It does not claim
that every XAI technique or application has been covered. Instead, it
selects influential work that illustrates the main methodological
families and recurring debates. The emphasis is on principles that
information and engineering organizations can use when selecting,
evaluating and governing explainability mechanisms.
3. Conceptual
Foundations of Explainability
The words explainability, interpretability, transparency and
intelligibility are often used interchangeably, but they do not
necessarily refer to the same property. Transparency may refer to
whether the model's structure or operation can be inspected.
Interpretability often refers to the degree to which a human can
understand how inputs relate to outputs. Explainability may refer to a
process that produces an account of a model's behaviour for a particular
audience. Lipton (2018) observed that interpretability can refer to
different goals, such as simulatability, decomposability and algorithmic
transparency. Treating these goals as identical can result in weak
evaluation and unclear claims.
An interpretable model is designed so that its operation can be
understood directly, at least within a defined scope. Examples include
sparse linear models, shallow decision trees, rule lists and generalized
additive models. A post hoc explanation is generated after a model has
been trained, often by approximating its behaviour or measuring how
changes to inputs affect outputs. The distinction matters because a post
hoc explanation may describe a simplified representation rather than the
exact model. The user should know whether the explanation is a faithful
account of the model or an approximation designed to be
understandable.
Explainability is also audience-dependent. A machine learning
engineer may want to inspect feature interactions, gradients and
training behaviour. A domain expert may need to understand whether the
model's reasoning is compatible with professional knowledge. A decision
subject may need a concise, actionable explanation of the factors that
influenced an outcome. A regulator may require evidence of governance,
validation and consistency. Miller (2019), drawing on social science
research into explanation, emphasized that human explanations are
selective, contrastive and shaped by the explainer's audience and
purpose. XAI design should therefore begin by asking who needs an
explanation and what question it must answer.
The scope of an explanation can be local or global. A local
explanation concerns one prediction or a small region around an
instance. A global explanation describes the model's overall behaviour,
influential features or decision structure. These forms answer different
questions. A local explanation may clarify why one loan application was
rejected, but it does not establish that the model is fair across a
population. A global feature ranking may describe broad model behaviour
but fail to explain a particular decision. A responsible system may
require both, alongside subgroup analysis and documentation.
Explanations can also be descriptive or prescriptive. A descriptive
explanation identifies factors associated with an existing prediction. A
prescriptive or actionable explanation indicates what changes could lead
to a different outcome. Counterfactual explanations often take this
form. However, the factors that influence a prediction are not
necessarily causes of the outcome in the real world. A model may use a
proxy variable that is predictive but not appropriate to change.
Explanations should not imply causal relationships unless the underlying
analysis supports causal inference.
4. Principal Families of XAI
Methods
XAI methods can be grouped in several ways: model-specific versus
model-agnostic; intrinsic versus post hoc; local versus global; and
attribution-based versus example-based or rule-based. These categories
overlap. A feature attribution method may be local and model-agnostic,
while a saliency method may be local and model-specific. The categories
are useful for organizing the field, but the choice of method should
follow the question being asked rather than the popularity of a
technique.
Intrinsic interpretability is achieved by selecting a model whose
structure can be examined directly. Linear regression, logistic
regression, decision trees, rule-based classifiers and generalized
additive models are common examples. Their parameters or decision paths
can often be inspected without a separate explanation algorithm.
However, simple models are not automatically understandable.
High-dimensional feature spaces, correlated predictors, complex
preprocessing and interactions can make even a linear model difficult to
interpret. The term interpretable should therefore be evaluated in
relation to the user, task and data representation.
Post hoc model-agnostic methods aim to explain a model without
requiring access to its internal architecture. They can be applied to
many black-box predictors through input-output queries. LIME, introduced
by Ribeiro et al. (2016), approximates a model locally around an
instance using an interpretable surrogate. The method perturbs the
input, observes the model's predictions and fits a simple model weighted
toward samples near the instance. LIME is flexible, but the explanation
depends on the perturbation distribution, distance measure,
interpretable representation and local sampling process. If the
neighbourhood is unrealistic or the surrogate fits poorly, the
explanation may be misleading.
SHAP, proposed by Lundberg and Lee (2017), uses Shapley values from
cooperative game theory to attribute a prediction among input features.
The method provides a unified framework for several additive feature
attribution approaches and has model-specific and model-agnostic
implementations. Shapley-based explanations have attractive theoretical
properties, including consistency under defined assumptions. However,
their values depend on how missing features are represented and how
feature dependence is handled. A mathematically principled attribution
is not automatically a causal explanation, nor does it prove that a
model is fair.
Feature permutation methods estimate the importance of a variable by
measuring the change in model performance when that variable's values
are disrupted. They are relatively intuitive and can be used with many
model classes. Yet correlated features can complicate interpretation: if
two variables contain overlapping information, permuting one may have
little effect even though the information represented by the pair is
important. Feature importance is also dependent on the evaluation
dataset and chosen performance metric. It should not be interpreted as a
universal ranking of causal influence.
Gradient-based methods are frequently used to explain deep neural
networks. They examine how the output changes with respect to input
features, often using gradients or related quantities. Integrated
Gradients, developed by Sundararajan et al. (2017), attributes
predictions by integrating gradients along a path from a baseline input
to the observed input. The method was designed to satisfy properties
such as sensitivity and implementation invariance. Its results
nevertheless depend on the choice of baseline and the suitability of the
path for the data domain.
For image classification, saliency maps and class activation methods
highlight regions that contribute to a model's prediction. Grad-CAM,
introduced by Selvaraju et al. (2017), uses gradients flowing into
convolutional layers to produce a coarse localization map for a target
class. Such visualizations can help domain experts inspect whether a
model attends to relevant regions. However, visual plausibility is not
proof of faithfulness. A heatmap may look convincing while being
insensitive to changes in model parameters or labels. Adebayo et al.
(2018) proposed sanity checks showing that some saliency methods can
fail to reflect learned model behaviour under controlled randomization
tests.
Counterfactual explanations describe a minimally changed instance
that would alter the model's output. Wachter et al. (2017) developed a
framework for explaining individual predictions through counterfactual
examples. In a credit context, an explanation might state that a
different debt-to-income ratio or repayment history would change a
predicted category. The usefulness of such an explanation depends on
whether the suggested changes are feasible, lawful, stable and within
the individual's control. A system should not recommend changing
immutable or protected characteristics, nor should it present a
mathematically possible change as a realistic path to a better
outcome.
Example-based explanations use representative instances, prototypes,
criticisms or influential training examples to communicate model
behaviour. A user may understand a prediction by seeing similar cases
and contrasting cases. This can be intuitive, but the examples must be
selected carefully. Similarity metrics may not align with human
concepts, and a model may rely on features that are not visible in the
example display. Example-based methods can also expose sensitive
training data if individual records are revealed. Privacy-preserving
selection and aggregation are therefore important.
Rule extraction and surrogate models attempt to express complex model
behaviour through human-readable rules or simpler approximations. A
decision tree may be trained to imitate a neural network, or rules may
be extracted from a trained model. The resulting representation can help
users understand broad patterns, but approximation error must be
measured. A surrogate that agrees with the original model on common
cases may fail in rare but consequential regions. The explanation should
disclose its scope and fidelity rather than imply that the surrogate is
the original decision process.
Healthcare is a prominent application area because clinical decisions
combine high consequences, specialist knowledge and complex data. AI
systems have been developed for medical imaging, risk prediction,
triage, pathology and patient monitoring. Explanations may help
clinicians assess whether a model's output is consistent with the image
or patient record, identify artefacts and decide when to seek additional
evidence. Visual explanations can highlight image regions, while feature
attribution can indicate which patient variables contributed to a risk
estimate. However, clinicians must not assume that a highlighted region
is clinically causal. Models may exploit acquisition artefacts,
demographic proxies or patterns that do not generalize across
hospitals.
In finance, machine learning supports fraud detection, credit
scoring, anti-money-laundering monitoring and risk management.
Explainability can support internal model validation, regulatory review,
customer communication and the investigation of unexpected decisions. A
local explanation may identify factors associated with a credit
decision, while global analysis may reveal whether the model relies
excessively on variables that act as proxies for protected
characteristics. Counterfactual explanations can help an applicant
understand what factors are potentially changeable. Yet the institution
must distinguish between factors that are predictive, factors that are
legally permissible and factors that are genuinely actionable.
Public administration uses data-driven systems in areas such as
eligibility screening, tax compliance, welfare administration,
immigration, policing and resource allocation. These settings raise
particular concerns because affected people may have limited ability to
opt out or challenge a decision. An explanation should make the basis of
a decision accessible and support meaningful review. It should also
identify the responsible public authority and the process for correction
or appeal. A generic statement that an algorithm produced a score is not
a meaningful explanation of an administrative decision.
In cybersecurity, machine learning models detect malicious traffic,
anomalous behaviour, phishing and malware. Security analysts need to
understand why an alert was generated, which features or events
contributed to it, and whether the alert corresponds to a real threat.
Explainability can help analysts prioritize investigations and identify
false positives. However, explanation interfaces may expose detection
logic to attackers. Security teams must balance analyst transparency
with the need to protect sensitive detection rules and infrastructure.
Explanations should be role-based and should not disclose unnecessary
operational detail to unauthorized users.
Education systems use predictive analytics for learning
recommendations, student-risk identification, automated assessment and
adaptive learning. Explanations can help teachers understand why a
learner is flagged as needing support and help students understand
feedback. Yet educational predictions may reflect socioeconomic
disadvantage, unequal access to devices or historical grading patterns.
A risk score should not become a self-fulfilling label. Educators should
use explanations to guide inquiry and support, not to replace
professional judgement or define a student's future potential.
In information retrieval and recommendation, explainability can
clarify why a document, product, article or video has been recommended.
Explanations may identify the user's stated interests, content
similarity or the behaviour of comparable users. They can improve user
control and help people identify recommendations that are irrelevant or
inappropriate. However, recommendation explanations may oversimplify
ranking mechanisms or conceal commercial objectives. Platforms should
distinguish explanations of predicted relevance from explanations of
business rules, sponsored placement and personalization policy.
In human resources, automated systems may screen applications, rank
candidates or estimate employee attrition. Explanation is essential
because employment decisions affect livelihoods and may reproduce
historical inequalities. Feature attribution can show which variables
influenced a ranking, but it may not reveal whether the model's training
labels reflect discriminatory practices. A fair and accountable process
requires assessment of data provenance, job relevance, group
performance, accessibility and the availability of human review.
Explanations cannot repair an invalid selection criterion.
Manufacturing and engineering use AI for predictive maintenance,
quality inspection, process optimization and autonomous control.
Operators may need explanations to understand why a machine is predicted
to fail or why a control system recommends a particular adjustment. In
safety-critical settings, the explanation must be timely, accurate and
aligned with operational procedures. An overly detailed explanation may
increase cognitive burden during an incident, while a simplistic one may
omit critical uncertainty. The design should reflect the operator's task
and the time available for intervention.
6. Evaluating Explanation
Quality
A major challenge in XAI is the absence of a single accepted measure
of explanation quality. An explanation can be faithful to the model but
too complex for its intended audience. It can be easy to understand but
fail to represent the model. It can be stable but unhelpful, or
actionable but based on an unrealistic counterfactual. Doshi-Velez and
Kim (2017) proposed a framework for evaluating interpretability that
distinguishes application-grounded, human-grounded and functionally
grounded evaluation. This distinction remains useful because different
evaluation settings answer different questions.
Application-grounded evaluation involves domain experts performing
real tasks in realistic settings. For example, clinicians may use
explanations to decide whether to trust an imaging model, or auditors
may use them to investigate a credit model. This approach has strong
practical relevance but can be expensive and difficult to standardize.
Human-grounded evaluation uses simplified tasks with human participants
to compare explanations or measure understanding. Functionally grounded
evaluation uses proxy measures, such as fidelity, sparsity or
consistency, without requiring human subjects. Each approach provides
useful evidence, but none alone establishes that an explanation is
suitable for deployment.
Fidelity measures how accurately an explanation represents the model
being explained. For a surrogate model, fidelity may be measured by
agreement between the surrogate and the original model over a defined
sample. For attribution methods, evaluation may examine whether removing
highly attributed features changes the prediction more than removing
low-attribution features. These tests require care because input
perturbations can create unrealistic examples. A method can appear
faithful under one perturbation scheme and fail under another.
Stability concerns whether similar inputs produce similar
explanations when the model's behaviour is similar. Large explanation
changes caused by tiny input changes can make a system difficult to
trust. However, stability should not be treated as an unconditional
goal. If a small change crosses a genuine decision boundary, an
explanation may appropriately change. Evaluation should therefore relate
explanation variation to changes in model output and the data
context.
Comprehensibility concerns whether intended users can understand the
explanation. It can be assessed through task performance, user studies,
comprehension questions, response time and subjective measures.
Simplicity metrics such as the number of features or rules may be
useful, but they are only proxies. A short explanation can be confusing
if it uses unfamiliar terminology, while a longer explanation may be
understandable when presented progressively. The interface, user's
expertise and decision context all affect comprehension.
Actionability is important when explanations are intended to guide a
user toward a different outcome. A counterfactual should identify
changes that are feasible, permissible and relevant. It should not
recommend altering immutable characteristics or suggest that a person
can control factors outside their influence. Explanations should also
avoid implying that changing one feature guarantees a particular
real-world result when the model only estimates a probability.
Fairness evaluation asks whether explanation methods behave
appropriately across groups and whether explanations reveal or obscure
discriminatory model behaviour. A model may produce explanations that
appear neutral while relying on proxy variables. Group-level analyses
can compare attribution patterns, error rates and the availability of
recourse. However, fairness is a normative and contextual concept.
Statistical parity, equalized error rates and individual fairness
represent different principles and can conflict. Explanation methods
cannot determine which principle an institution should adopt.
Human reliance and trust calibration are also important. An
explanation that increases trust is not necessarily beneficial if the
model is wrong. Users should trust a system when it is appropriate and
question it when it is not. Studies of explanation interfaces should
therefore measure whether explanations improve the ability to detect
model errors, not merely whether users report greater confidence. Kaur
et al. (2020) examined how explanations affect reliance on AI systems
and highlighted the importance of evaluating actual user behaviour
rather than assuming that more explanation produces better
decisions.
7. Limitations and Failure
Modes
Post hoc explanations can be unfaithful. A surrogate model may
approximate the original model only in a narrow region, and feature
attribution can depend on assumptions about feature independence or the
baseline distribution. If users are not told these assumptions, they may
interpret the explanation as a direct account of the model's internal
reasoning. The explanation interface should state what method was used,
what it explains and what it does not establish.
Explanations can be unstable. Small changes in an input, model
parameters or sampling choices may lead to different feature rankings or
visualizations. Slack et al. (2020) demonstrated that explanation
methods can be manipulated in ways that make biased models appear more
benign under certain conditions. This illustrates that explanations
themselves can become targets of strategic manipulation. Independent
testing and adversarial evaluation are therefore necessary, particularly
when organizations have incentives to present models favourably.
Explanations can be incomplete. A feature ranking may show the most
influential variables but omit interactions, nonlinearities or data
preprocessing steps. A heatmap may highlight a region without explaining
why the region matters. A counterfactual may identify a minimal change
but fail to account for dependencies among variables. Explanations
should not be presented as complete accounts of complex systems when
they are only partial views.
Explanations may create a false sense of causality. Feature
importance indicates a relationship within a model, not necessarily a
causal effect in the world. If a model predicts loan default using
postcode, the explanation does not establish that changing postcode
would causally change repayment behaviour. Causal claims require an
appropriate causal model, assumptions and identification strategy. The
distinction is essential when explanations are used to recommend
interventions or justify decisions.
There is also a risk of information overload. Detailed technical
explanations may overwhelm users, particularly in time-sensitive
settings. A clinician or security analyst may need a concise indication
of the key evidence, while an auditor may require a detailed record. A
single explanation format cannot serve all audiences. Progressive
disclosure can provide a summary first and allow users to inspect
technical details when needed.
Privacy risks arise when explanations expose sensitive attributes,
training examples or information about other individuals. Example-based
explanations can reveal records that were used during training. Feature
attribution may reveal sensitive information about a person when
combined with external knowledge. Organizations should evaluate privacy
leakage and avoid displaying identifiable examples without a lawful
basis and appropriate safeguards.
Security risks include explanation-based model extraction and gaming.
If a system reveals too much about its decision boundary, an attacker
may infer model behaviour or manipulate inputs to evade detection. In
hiring or credit settings, detailed explanations can help individuals
understand and challenge decisions, but they can also reveal exploitable
thresholds. The solution is not to eliminate explanations, but to design
access, detail and review processes according to the risks and rights
involved.
Finally, explainability can distract from deeper system problems. A
model trained on unrepresentative data, deployed in an unsuitable
context or used to pursue an unjust objective does not become acceptable
merely because it has an explanation interface. Selbst et al. (2019)
warned that fairness interventions can fail when they abstract away from
the social context in which a system operates. XAI must therefore be
integrated with data governance, impact assessment, institutional
accountability and the ability to decide not to automate.
8. Governance,
Standards and Organizational Practice
Organizations should treat explainability as a lifecycle requirement.
At project initiation, teams should define the decision being supported,
affected stakeholders, possible harms and the level of explanation
required. During data preparation, they should document provenance,
quality, representativeness and known limitations. During model
development, they should compare interpretable alternatives with more
complex models and record why a particular approach was selected. During
validation, they should test predictive performance, explanation
fidelity, subgroup behaviour, robustness and user comprehension.
Model documentation can support this lifecycle. Mitchell et al.
(2019) proposed model cards as structured documents describing intended
use, evaluation procedures, performance across relevant groups and
limitations. Datasheets for datasets, proposed by Gebru et al. (2021),
provide a complementary approach to documenting dataset motivation,
composition, collection and recommended uses. These documents do not
solve all governance problems, but they make assumptions and limitations
more visible and provide a basis for review.
Governance should assign clear responsibilities. Data stewards are
responsible for data quality and permitted use; model developers
document design choices and limitations; domain experts assess
contextual validity; risk and compliance teams examine legal and policy
requirements; system owners monitor deployment; and senior
decision-makers remain accountable for institutional use. The existence
of a technical explanation should not transfer responsibility from the
organization to the model or its vendor.
Users should be able to challenge consequential decisions. A
meaningful contestation process requires more than a generic
explanation. It should identify the responsible decision-maker, provide
access to relevant information, allow correction of inaccurate data and
offer a route to human reconsideration. The explanation should be
understandable to the affected person and should not require advanced
technical knowledge. Organizations should monitor whether appeals reveal
systematic errors and use the findings to improve the system.
Procurement also matters. When an organization purchases an AI
product, it should ask what explanation methods are available, what
their limitations are, whether explanations can be independently tested,
how model updates are communicated and whether audit records can be
retained. Vendor claims such as fully transparent or completely
explainable should be translated into testable requirements. Contracts
should address access to documentation, incident notification, data
processing, model changes and support for regulatory or independent
review.
An explanation policy should specify which systems require
explanations, which audiences receive them, how explanations are
validated and how conflicts are escalated. The policy should distinguish
low-risk assistive tools from systems that materially affect rights or
access to services. It should also define when an interpretable model is
preferred, when a post hoc method may be acceptable and when the
organization should avoid deploying an AI system because its behaviour
cannot be adequately understood or governed.
9.
Proposed Layered Framework for Explainable Intelligent Systems
The review proposes a six-layer framework for designing and governing
explainable AI. The first layer is purpose and context. It defines the
decision, user groups, affected people, legal setting and consequences
of error. The second layer is model transparency. It records the model
class, features, training data, preprocessing, assumptions and known
limitations. The third layer is explanation generation. It selects
methods that fit the model and the intended question, such as
coefficients, decision paths, feature attribution, saliency maps,
examples or counterfactuals.
The fourth layer is explanation validation. It evaluates fidelity,
stability, comprehensibility, actionability, fairness and privacy. The
fifth layer is human interpretation and contestability. It provides
interfaces appropriate to different users, supports expert review and
enables affected individuals to question outcomes. The sixth layer is
governance and monitoring. It assigns responsibility, records decisions,
monitors drift and incidents, and ensures that explanations remain
appropriate as the model and operating environment change.
Figure 1. Layered framework for explainable intelligent systems
1. Purpose and context: decision, stakeholders, consequences and
rights
2. Model transparency: data, features, architecture, assumptions and
limits
3. Explanation generation: intrinsic, attribution, example, rule or
counterfactual
4. Explanation validation: fidelity, stability, usability, fairness and
privacy
5. Human interpretation: role-specific interfaces, review and
contestation
6. Governance and monitoring: accountability, audit, drift and incident
response
Feedback flows from monitoring and user challenge to model review and
redesign.
The framework is iterative rather than linear. User feedback may
reveal that an explanation is incomprehensible or that a model relies on
an inappropriate feature. Monitoring may identify drift that invalidates
earlier explanations. An appeal may reveal inaccurate input data or an
unjustified decision rule. These findings should trigger review of the
model, explanation method or deployment purpose. The framework therefore
treats explainability as an ongoing organizational capability rather
than a one-time technical deliverable.
Table 1 presents the relationship between common XAI methods, their
typical strengths and their principal cautions.
10. Discussion
The literature demonstrates that explainability is not a single
measurable attribute that can be added to any AI system through a
standard visualization. It is a relationship among a model, an
explanation method, a user, a task and an institutional setting. A
method that is useful for a data scientist may be unsuitable for a
patient or loan applicant. An explanation that supports debugging may
not satisfy the requirements of procedural fairness. The first design
decision should therefore be the explanation purpose, not the selection
of a popular library or visualization.
A second finding is that model transparency and post hoc explanation
should not be treated as equivalent. Intrinsically interpretable models
can provide direct insight into the decision process, but they may not
always achieve the predictive performance required for a particular
task. Complex models may capture nonlinear patterns, but post hoc
explanations introduce approximation and validation challenges. The
choice should be made through comparative evaluation, with particular
caution in high-impact settings. Rudin's (2019) argument for
interpretable models is especially important where a simpler model can
achieve adequate performance and where the cost of an opaque decision is
high.
Third, explanation quality is inseparable from data quality and
institutional context. If training data encode historical
discrimination, a feature attribution method may faithfully explain a
harmful pattern without making the decision acceptable. If a model is
deployed in a setting different from its training environment, an
accurate explanation of its current computation may still be irrelevant
to the real-world decision. XAI should be embedded within broader impact
assessment, fairness analysis and organizational governance.
Fourth, explanations should support calibrated reliance rather than
unconditional trust. Users need to know when the system is likely to be
useful, when it is uncertain and when it should be ignored. A
well-designed explanation can help a professional recognize a model's
limitations, but it can also increase overreliance if presented as
authoritative. User studies should measure error detection and decision
quality, not only trust ratings or perceived transparency.
Finally, the field needs more consistent reporting. Research papers
should describe the model, explanation algorithm, perturbation or
baseline assumptions, evaluation datasets, user population and known
limitations. Applied studies should report how explanations changed
decisions and whether the changes improved outcomes. Without such
detail, it is difficult to compare methods or translate findings into
institutional practice.
11. Research Gaps and
Future Directions
One research priority is the development of stronger evaluation
protocols. Many explanation methods are assessed using different
datasets and metrics, making direct comparison difficult. Shared
evaluation suites should include fidelity tests, stability tests,
adversarial checks, subgroup analysis and user-centred measures. These
suites should be adapted to specific domains rather than relying on a
single generic benchmark.
A second priority is causal explainability. Current feature
attribution methods generally describe model associations rather than
causal mechanisms. More work is needed to connect explanations with
causal inference while clearly stating assumptions and limitations.
Counterfactual methods should distinguish changes that alter a model
prediction from interventions that would alter a real-world outcome.
This distinction is vital when explanations are used to recommend
action.
A third priority is explanation under distribution shift. Models may
encounter new populations, devices, policies or operating conditions
after deployment. Explanation methods may become less reliable when data
distributions change. Research should examine how explanation stability
and fidelity evolve over time and how monitoring systems can detect when
previously validated explanations are no longer appropriate.
A fourth priority concerns privacy-preserving explanation.
Organizations need to provide meaningful accounts of decisions without
disclosing personal information or sensitive model details. Techniques
such as aggregation, controlled access, differential privacy and secure
computation may contribute, but they can also reduce explanation
precision. Research should evaluate the trade-offs between privacy,
fidelity and usability.
A fifth priority is inclusive and accessible explanation design.
Explanations should be tested with people who have different levels of
technical literacy, language proficiency, disability status and domain
knowledge. Research should explore plain-language explanations,
multilingual interfaces, visual and non-visual formats, and methods that
allow users to ask follow-up questions. Accessibility should be treated
as a core quality attribute.
A sixth priority is the integration of explainability with
organizational decision processes. Many studies focus on the explanation
artifact rather than the work system in which it is used. Future
research should examine how explanations affect professional workflows,
accountability, appeals, organizational learning and the distribution of
responsibility. It should also investigate when explanations lead
institutions to correct models, change policies or abandon inappropriate
automation.
Finally, XAI research should study strategic behaviour. Organizations
may use explanations to legitimize decisions, while individuals may
adapt their behaviour to influence model outputs. Attackers may
manipulate explanation systems or exploit disclosed decision boundaries.
The field needs stronger threat models and independent audits to ensure
that explanations cannot be easily gamed or used as public-relations
substitutes for substantive accountability.
12. Conclusion
Explainable artificial intelligence is essential to the responsible
use of machine learning in information-intensive and high-impact
settings. The field includes interpretable models, post hoc
explanations, feature attribution, counterfactual reasoning, visual
methods, example-based explanations and human-centred evaluation. These
approaches can help developers debug systems, professionals assess
recommendations, institutions document model behaviour and affected
individuals understand or challenge decisions.
The review has shown that no explanation method is universally
suitable. Methods differ in their assumptions, scope, fidelity,
stability, audience and vulnerability to manipulation. A feature
attribution score is not a causal effect; a heatmap is not proof of
model reasoning; a counterfactual is not necessarily a feasible
intervention; and a simple surrogate is not the original model.
Organizations must communicate these limits and validate explanations in
the context where they will be used.
The proposed six-layer framework connects purpose, model
transparency, explanation generation, validation, human interpretation
and governance. It places explainability within the full lifecycle of an
intelligent system and emphasizes accountability, contestability and
continuous monitoring. For high-stakes applications, institutions should
prefer intrinsically interpretable models when they can meet the task
requirements, and should not use explanation techniques to justify
systems that are fundamentally unsuitable or ungovernable.
As AI becomes more deeply embedded in organizational and public
infrastructure, explainability should be understood as a condition for
informed use, not a guarantee of correctness or fairness. Transparent
intelligent systems require sound data, appropriate models, honest
explanations, capable users and accountable institutions. The future of
XAI will depend on bringing these elements together so that
computational efficiency does not come at the expense of human
understanding and responsible decision-making.
Table 1. Common explanation methods, strengths and
implementation cautions
| Method family |
Typical contribution |
Important caution |
| Intrinsic interpretable models |
Direct inspection of coefficients, rules or decision paths |
Complex features and interactions can still reduce human
understanding |
| LIME and local surrogates |
Approximates behaviour around an individual prediction |
Sensitive to perturbations, neighbourhood definition and surrogate
fidelity |
| SHAP values |
Feature attribution with a principled additive framework |
Dependence assumptions and attribution are not causal
explanations |
| Integrated Gradients |
Attributes deep model outputs along a baseline path |
Baseline selection affects interpretation |
| Saliency and Grad-CAM |
Highlights image regions relevant to a model output |
Visual plausibility does not establish faithfulness |
| Counterfactual explanations |
Shows changes associated with a different prediction |
Suggested changes may be infeasible, impermissible or
non-causal |
| Example-based explanations |
Uses prototypes or similar cases to communicate behaviour |
Similarity may be misleading and examples may expose private
data |
| Rule extraction and surrogates |
Communicates complex behaviour through compact rules |
Approximation may fail in rare or consequential regions |
References
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., &
Kim, B. (2018). Sanity checks for saliency maps. Advances in Neural
Information Processing Systems, 31.
https://proceedings.neurips.cc/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html
Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A
survey on explainable artificial intelligence (XAI). IEEE Access, 6,
52138–52160. https://doi.org/10.1109/ACCESS.2018.2870052
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik,
S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R.,
Chatila, R., & Herrera, F. (2020). Explainable Artificial
Intelligence (XAI): Concepts, taxonomies, opportunities and challenges
toward responsible AI. Information Fusion, 58, 82–115.
https://doi.org/10.1016/j.inffus.2019.12.012
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019). Machine
learning interpretability: A survey on methods and metrics. Electronics,
8(8), Article 832. https://doi.org/10.3390/electronics8080832
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of
interpretable machine learning. arXiv.
https://doi.org/10.48550/arXiv.1702.08608
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach,
H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets.
Communications of the ACM, 64(12), 86–92.
https://doi.org/10.1145/3458723
Gunning, D., & Aha, D. (2019). DARPA’s explainable artificial
intelligence (XAI) program. AI Magazine, 40(2), 44–58.
https://doi.org/10.1609/aimag.v40i2.2850
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F.,
& Pedreschi, D. (2018). A survey of methods for explaining black box
models. ACM Computing Surveys, 51(5), Article 93.
https://doi.org/10.1145/3236009
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., &
Wortman Vaughan, J. (2020). Interpreting interpretability: Understanding
data scientists’ use of interpretability tools for machine learning.
Proceedings of the 2020 CHI Conference on Human Factors in Computing
Systems, 1–14. https://doi.org/10.1145/3313831.3376219
Lipton, Z. C. (2018). The mythos of model interpretability. Queue,
16(3), 31–57. https://doi.org/10.1145/3236386.3241340
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to
interpreting model predictions. Advances in Neural Information
Processing Systems, 30.
https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html
Miller, T. (2019). Explanation in artificial intelligence: Insights
from the social sciences. Artificial Intelligence, 267, 1–38.
https://doi.org/10.1016/j.artint.2018.07.007
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L.,
Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model
cards for model reporting. Proceedings of the Conference on Fairness,
Accountability, and Transparency, 220–229.
https://doi.org/10.1145/3287560.3287596
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I
trust you?” Explaining the predictions of any classifier. Proceedings of
the 22nd ACM SIGKDD International Conference on Knowledge Discovery and
Data Mining, 1135–1144. https://doi.org/10.1145/2939672.2939778
Rudin, C. (2019). Stop explaining black box machine learning models
for high stakes decisions and use interpretable models instead. Nature
Machine Intelligence, 1, 206–215.
https://doi.org/10.1038/s42256-019-0048-x
Selbst, A. D., boyd, d., Friedler, S. A., Venkatasubramanian, S.,
& Vertesi, J. (2019). Fairness and abstraction in sociotechnical
systems. Proceedings of the Conference on Fairness, Accountability, and
Transparency, 59–68. https://doi.org/10.1145/3287560.3287598
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D.,
& Batra, D. (2017). Grad-CAM: Visual explanations from deep networks
via gradient-based localization. Proceedings of the IEEE International
Conference on Computer Vision, 618–626.
https://doi.org/10.1109/ICCV.2017.74
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H.
(2020). Fooling LIME and SHAP: Adversarial attacks on post hoc
explanation methods. Proceedings of the AAAI/ACM Conference on AI,
Ethics, and Society, 180–186.
https://doi.org/10.1145/3375627.3375830
Sokol, K., & Flach, P. (2020). Explainability fact sheets: A
framework for systematic assessment of explainable approaches.
Proceedings of the 2020 Conference on Fairness, Accountability, and
Transparency, 56–67. https://doi.org/10.1145/3351095.3372870
Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic
attribution for deep networks. Proceedings of the 34th International
Conference on Machine Learning, 70, 3319–3328.
https://proceedings.mlr.press/v70/sundararajan17a.html
Wachter, S., Mittelstadt, B., & Russell, C. (2017).
Counterfactual explanations without opening the black box: Automated
decisions and the GDPR. Harvard Journal of Law & Technology, 31(2),
841–887.
https://jolt.law.harvard.edu/assets/articlePDFs/v31/Counterfactual-Explanations-without-Opening-the-Black-Box-Sandra-Wachter-et-al.pdf