ACT I
A green PR can still be wrong
Implementation accelerates. Context discovery and review capacity do not.
Common situations
Knowledge moves
When people move to another company, their tacit knowledge leaves with them.
Ownership shifts
Team rotations, Reorganizations change responsibility and accountability.
Harness Engineering
AI Agents turn backlog items into code at industrial speed.
Then Software engineer using an AI agent picks up a seemingly local story...
The story looks local
Rename customerId to clientId.
record Order(String clientId, BigDecimal total) {}
The agent implements it quickly. Repository checks pass.
Every repository check can be green—and production can still break
✓ Compiles
✓ Unit + integration tests
✓ Security scans
✓ Acceptance criteria
💥 Production breaks
The checks proved the implementation—not the whole consequence.
How can a change be repository-correct and
still be system-wrong?
The blast radius is larger than the repository
Mobile API
OpenAPI + Jackson
ORDER
SERVICE
customerId → clientId
Database
Flyway migration
Kafka event
schema
Fraud + notifications
Data lake + ETL
BI + documentation
The repository cannot always be the unit of correctness.
System architecture mirrors communication structures
Organizations design systems that mirror their communication structures.
Paraphrased from Melvin Conway, 1968.
AI amplifies the cost of weak communication boundaries
Fragmented system context
→
Agents generate changes faster
→
Hidden inconsistency spreads faster
Faster generation does not repair a weak boundary—it propagates its
gaps faster than people can discover and review them.
AI accelerates technology—not the whole system
People
Ownership
Communication
Process
Specification
Validation
AI accelerates technology.
System correctness still depends on all three.
System awareness changes the implementation strategy
Repository-only view
Rename customerId
to clientId
System-aware view
- Introduce
clientId compatibly
- Read and write both names
- Migrate and observe consumers
- Remove
customerId after adoption
Same requirement. Different context. Different plan.
Parallel Change:
expand → migrate → contract
Agents see tasks first; dependency failures live beyond them
ORGANIZATION -> ownership · decisions · business constraints
SYSTEM -> consumers · contracts · other repositories
REPOSITORY -> code · tests
TASK
Most coding agents start at task and repository.
Many
dependency failures live outside them.
A green PR is evidence—not a decision
Usually visible
Diff
Repository tests
CI status
Agent summary
Often missing
External consumers
Uncertainty
Applicable controls
Decision authority
Without system context and decision authority, human approval becomes cosmetic oversight.
ACT II
HITL at the PR: from review to decision
Human review only counts when reviewers see the evidence, own the risk, and can say no.
Why green PRs can still fool humans
Automation bias
Green and polished feels safe
Correlated evidence
The agent writes code and tests
Context gap
Consequences live elsewhere
Queue pressure
Generation outpaces review
A human presence does not guarantee a human judgment.
Meaningful HITL requires sight, accountability, and power to act
See enough
Context + evidence
Be accountable
Competence + authority
Be able to act
Time + intervention
Without them, approval is human-as-a-button.
1 — Classify the change before choosing the review
Start with impact and uncertainty. Autonomy, reversibility, data, rights, safety, sector, and applicable law can raise the level.
REVIEW LEVEL 1
Local and reversible
Docs · formatting · contained tests
Policy-defined review
REVIEW LEVEL 2
Application behavior
Code owner + test evidence
REVIEW LEVEL 3
Cross-system or sensitive
Contracts · schema · personal data · security
Specialist reviewers
REVIEW LEVEL 4
Rights, safety, or regulated operations
Accountable owner + controlled release
The goal is effective control where consequences matter, not approval everywhere.
2 — Show the evidence, not just the diff
Before
Requirement and scope
Known consumers
Risk classification
Decision
Compatibility strategy
Assumptions and limits
Required owners
After
Consumer evidence
Rollout and rollback
Monitoring and audit trail
The diff proposes the change; the evidence package makes the consequence reviewable.
Challenge the evidence before accepting the claim
“The customerId migration was validated successfully.”
Cosmetic review
Checks are green.
The diff looks reasonable.
APPROVE
Effective review
Which consumers were checked?
What remains unverified?
How is compatibility preserved?
What triggers rollback?
3 — Route the PR to accountable reviewers
API · event · schema
Domain + system owners
Auth · secrets · dependencies
Security + platform
Personal data · retention · logs
Privacy + data owners
Money · rights · safety · regulation
Business + compliance + safety
The right reviewer depends on the risk, not on who is available.
4 — Decide: approve, modify, escalate, or stop
APPROVE
Evidence is sufficient
MODIFY
Request changes or more proof
ESCALATE
Another owner must decide
STOP
Risk or uncertainty is unacceptable
A reviewer needs real alternatives to “Approve.”
5 — Observe: merge is not the end of the loop
Monitor, rollback, preserve evidence, and improve the control that failed.
Agent-created PR
→
Controlled release
→
Observe outcome
Prediction ↔ outcome ↔ corrective action
ACT III
Governance that changes engineering decisions
Classify first, then let triggers turn regulation into reviews, evidence, and release gates.
Governance is a control stack
HITL is the decision mechanism connecting policy to execution.
Laws & obligations
Standards
Frameworks
↓
Organizational governance · policy · risk appetite · decision rights · owners
↓
Engineering controls · permissions · PR gates · tests · reviewer routing · rollout
↓
Evidence and outcomes · decision · deployment · monitoring · audit · corrective action
Classification comes before compliance claims
Regulated AI system
Article 14 requires effective human oversight for applicable high-risk AI systems.
AI-assisted development
Using an AI coding agent does not by itself classify the software being changed as a high-risk AI system.
Classify the deployed system and use case first.
Apply human review, intervention, auditability, and rollback as engineering controls regardless.
Regulation (EU) 2024/1689
Three trigger families keep governance actionable
AI system risk
EU AI Act
Classify the system and use case · define oversight when applicable
Personal data
GDPR
Minimization · access · retention · rights · accountable data owner
Cybersecurity and resilience
Cyber Resilience Act (CRA) · NIS2 · DORA
Supply chain · vulnerabilities · continuity · incidents
These are routing signals—not final applicability determinations.
EU AI Act: classify the system before choosing controls
Route when
An AI system or GPAI model is placed on the EU market, used in the EU, or its output affects people in the EU.
Classify
Provider or deployer · intended purpose · prohibited practice · Annex III/I high-risk signal · transparency · GPAI.
Engineer evidence
Human oversight · model and data lineage · logs · accuracy and robustness tests · monitoring · rollback or disablement.
High-risk milestones: Annex III · 2 Dec 2027; Annex I product AI · 2 Aug 2028. Classification and legal interpretation require qualified owners. Regulation (EU) 2024/1689
EU AI Act · More Java issues
Examples:
| Issue |
Risk |
Fix |
AI Act hook |
| 1 · No human oversight |
Agent-generated or model-driven change to financial logic reaches production on a green pipeline |
Merge blocked until a competent reviewer approves, modifies, rejects, or stops; prompt, model version, diff, and tests exposed |
Art. 14: human oversight |
| 2 · No traceability of AI outputs |
Model output drives a decision, but model version, input, and result are not recorded |
Decision log with model version, input reference, output, confidence, and reviewer, retained under a defined policy |
Art. 12 · 26: logging, deployer log retention |
| 3 · AI interaction not disclosed |
A chatbot or generated content is presented to users as human or unmarked |
Disclosure flag in the API and UI; generated content labeled and machine-readable where required |
Art. 50: transparency obligations |
| 4 · AI feature shipped unclassified |
A scoring or ranking feature reaches users with no intended-purpose or risk classification |
Classification record (role, purpose, Annex III signal) required by a CI gate before release |
Art. 5 · 6: prohibited practices, high-risk classification |
GDPR: personal data changes the design boundary
Route when
A Java service collects, stores, transforms, exposes, logs, exports, or deletes data linked to a natural person.
Decide with owners
Purpose and lawful basis · controller or processor role · sensitive data · DPIA · transfers · retention.
Engineer evidence
Data inventory · narrow DTOs · field authorization · rights workflows · deletion propagation · privacy-safe observability.
The GDPR has applied since 25 May 2018.
Privacy, DPO, legal, security, and data owners decide lawful basis, roles, DPIA, transfers, and risk acceptance. Regulation (EU) 2016/679
GDPR · More Java issues (1/2)
Examples:
| Issue |
Risk |
Fix |
GDPR hook |
| 1 · Entity exposed by the API |
findById returns the whole CustomerProfile; every stored field becomes API data |
Purpose-specific CustomerSummary DTO with authorization and access audit |
Art. 5(1)(c) · 25: minimisation, data protection by design |
| 2 · PII in logs and traces |
log.info("Customer {}", customer) writes email and address to logs, traces, and metrics |
Log ids only; masking layer and field-level redaction policy; no PII in observability |
Art. 5(1)(f) · 32: confidentiality, security of processing |
| 3 · Erasure not propagated |
repository.delete(id) removes the row, but copies stay in caches, search indexes, topics, backups, and analytics |
Erasure workflow that fans out to every store and records completion |
Art. 17: right to erasure |
GDPR · More Java issues (2/2)
Examples:
| Issue |
Risk |
Fix |
GDPR hook |
| 4 · Retention never enforced |
Data kept forever; no retention period or purge job |
retentionUntil per purpose and a scheduled purge with a recorded outcome |
Art. 5(1)(e): storage limitation |
| 5 · No access or export path |
Data subject requests are answered with ad hoc SQL |
DataSubjectExport gathers data across services into a portable format with an audit trail |
Art. 15 · 20: access, portability |
| 6 · Purpose creep |
Data collected for billing is reused for marketing or model training with no check |
Purpose tags on fields and a policy check at the point of use; basis or consent recorded |
Art. 5(1)(b) · 6: purpose limitation, lawful basis |
Cyber Resilience Act (CRA): secure the product lifecycle
Route when
Software or a connected Java component may be a product with digital elements made available on the EU market.
Decide with owners
Product classification · manufacturer role · important or critical category · conformity route · support period.
Engineer evidence
Threat model · secure defaults · SBOM · vulnerability handling · signed updates · user guidance · end-of-support.
Article 14 reporting applies from 11 Sep 2026; full application starts 11 Dec 2027.
Product, product-security, legal, compliance, support, and market-access owners decide applicability and conformity. Regulation (EU) 2024/2847
CRA · More Java issues
Examples:
| Issue |
Risk |
Fix |
CRA hook |
| 1 · Secrets in the artifact |
Hardcoded API key or signing key inside the JAR or properties |
Inject secrets at runtime; block secrets in the artifact with a build scan |
Annex I: confidentiality, protection of data |
| 2 · Insecure defaults |
Default password, open admin surface, or trustAllCerts shipped in the product |
Secure-by-default config; SecurityDefaults .validate(config) fails on weak settings |
Annex I: secure by default, minimal attack surface |
| 3 · SBOM drift |
Fat JAR built with latest or version ranges; no inventory of what shipped |
Pinned dependency lock; CycloneDX SBOM generated in CI and attached to the release |
Annex I: SBOM, component transparency |
| 4 · Unsafe deserialization |
new ObjectInputStream(in) .readObject() on update or plugin metadata |
Typed DTO with schema validation and an ObjectInputFilter |
Annex I: limit attack vectors, input validation |
NIS2: cybersecurity is a service and supply-chain concern
Route when
A system supports a possible essential or important entity, critical sector, managed service, or digital infrastructure provider.
Decide with owners
Entity and sector classification · Member State law · significant-incident reporting · provider responsibility.
Engineer evidence
Asset inventory · secure configuration · incident workflow · supply-chain review · access control · restore and continuity tests.
EU transposition deadline: 17 Oct 2024; verify the applicable national law.
Legal, security, risk, resilience, continuity, and executive owners decide entity scope and reporting duties. Directive (EU) 2022/2555
NIS2 · More Java issues (1/2)
Examples:
| Issue |
Risk |
Fix |
NIS2 hook |
| 1 · Incident as log line |
logger.error("Security issue", e); no triage, owner, or reporting handoff |
SecurityIncident classified by severity, evidence captured, owners notified, containment opened |
Art. 23: significant-incident reporting |
| 2 · Reporting clock ignored |
Detection time and report deadlines are not recorded |
ReportingDeadlines.from (incident.detectedAt()) computes the 24h, 72h, and 1-month milestones and alerts owners |
Art. 23: early warning 24h · notification 72h · final report 1 month |
| 3 · Unvetted supplier dependency |
Third-party SDK or SaaS client added with no inventory, owner, or review |
SupplierRegister entry (component, provider, criticality, review, exit plan) enforced by a CI check |
Art. 21(2)(d): supply-chain security |
NIS2 · More Java issues (2/2)
Examples:
| Issue |
Risk |
Fix |
NIS2 hook |
| 4 · Overprivileged access |
Service account or admin API with a static token, no MFA, no least privilege |
Role-scoped access, short-lived credentials, MFA on admin paths, access logged |
Art. 21(2)(i, j): access control, MFA |
| 5 · Untested recovery |
Backup job exists but nobody has restored from it; no continuity runbook |
Scheduled restore test records RTO and RPO and stores the result as evidence |
Art. 21(2)(c): continuity, backup, crisis management |
| 6 · Unmanaged vulnerabilities and config |
Unpatched dependencies and drifting configuration; no asset inventory |
Asset inventory, CI scanning, patch SLA by severity, and drift detection |
Art. 21(2)(e, f): vulnerability handling, effectiveness assessment |
DORA: prove digital operational resilience
Route when
A Java system supports an in-scope financial entity, important business service, critical ICT function, or ICT provider relationship.
Decide with owners
Applicability · function criticality · incident reporting · provider criticality · outsourcing and concentration risk.
Engineer evidence
ICT inventory · detection and escalation · restore and failover tests · change control · provider monitoring · exit plans.
DORA has applied since 17 Jan 2025.
Legal, compliance, risk, resilience, security, procurement, and business owners decide applicability and risk acceptance. Regulation (EU) 2022/2554
DORA · More Java issues (1/2)
Examples:
| Issue |
Risk |
Fix |
DORA hook |
| 1 · No resilience on a critical dependency |
paymentNetwork.settle(batch) with no timeout, idempotency, bounded retry, circuit breaker, or fallback |
ResiliencePolicy with timeout, bounded retry, and circuit breaker; idempotent replay and deferAndEscalate |
Art. 9 · 11: ICT risk management, response and recovery |
| 2 · Incidents not classified or reportable |
Failure logged as error; no major-incident criteria, clock, or report handoff |
IncidentClassifier applies the criteria (clients affected, duration, data loss, criticality) and routes to the reporting workflow |
Art. 17–19: incident classification and reporting |
| 3 · Concentration on one ICT provider |
One cloud or payment provider hardcoded across the code; no abstraction or exit path |
Port/adapter boundary plus a ProviderRegister with criticality, alternative provider, and tested exit plan |
Art. 28 · 30: third-party risk, exit strategies |
DORA · More Java issues (2/2)
Examples:
| Issue |
Risk |
Fix |
DORA hook |
| 4 · Recovery never tested |
Failover and restore exist on paper but were never exercised; RTO and RPO unknown |
Scheduled failover and restore drills record RTO and RPO and store the results as evidence |
Art. 11 · 24–25: continuity, resilience testing |
| 5 · Uncontrolled ICT change |
Direct production deploy or config change with no approval, rollback, or audit trail |
Gated change pipeline with approval, automated rollback, and a change record linked to the release |
Art. 9: ICT change management |
| 6 · No ICT asset or dependency inventory |
Unknown libraries, services, and data flows behind an important function |
Inventory mapping each function to components, dependencies, owner, and criticality, built from the SBOM and service catalog |
Art. 8: identification, ICT asset mapping |
EU compliance calendar: when must projects be ready?
| Regulation |
Project trigger |
Application milestone |
| GDPR |
Processes personal data |
25 May 2018 · applies |
| NIS2 |
Supports an in-scope essential or important entity |
17 Oct 2024 · EU transposition deadline; check national law |
| DORA |
Falls within EU financial-sector scope |
17 Jan 2025 · applies |
| EU AI Act |
Provides or deploys an in-scope AI system or GPAI model |
2 Feb 2025 · prohibitions + AI literacy
2 Aug 2025 · GPAI + governance
2 Aug 2026 · transparency + enforcement
2 Dec 2027 · Annex III high-risk AI
2 Aug 2028 · Annex I product AI
|
| CRA |
Places a product with digital elements on the EU market |
11 Sep 2026 · reporting
11 Dec 2027 · full application |
Dates are legal milestones—readiness starts earlier. Confirm the organization, role, sector, use case, market, and Member State before deciding applicability.
The rename raises three different governance questions
What can break?
OpenAPI · Kafka · Flyway
Compatibility plan + consumer owners
How was it produced?
AI-assisted development
Generated artifacts + limits + human decision trace
What is the system?
Applicable AI · data · resilience classification
Specialist owner + required controls
Development governance controls the agent-created PR. System regulation depends on what the deployed system does and where it operates.
Source: Leavitt's Alignment Model (1965) >> People, Process and Technology
Framework
