AI Security Management Prep

A company building a narrow fraud-detection tool assumes GPAI obligations don't apply, and they're right. But when they later license a foundation model for an open-ended internal chatbot, GPAI obligations — model evaluation, incident tracking, cybersecurity — apply in a way their narrow tool never triggered, because generality itself determines this category, not risk level.

A logistics company's routing AI starts systematically deprioritizing deliveries to a lower-income zip code because of biased historical data. Engineers notice the pattern, but no one has authority to halt the system — governance was never formalized beyond "the data science team handles it." When a complaint escalates to a lawsuit, the board — who holds ultimate, undelegable accountability regardless of who built the model — has no oversight structure to show regulators.

A competitor secretly contributes manipulated samples to a public dataset that a company later uses to fine-tune its customer-service model. Months later, the model responds strangely to a specific, obscure phrase — a backdoor the poisoned samples planted — letting the competitor trigger embarrassing outputs on demand. Because nobody validated the training data's provenance before fine-tuning, the sabotage went undetected until it was already deployed.

A bank's AI-driven loan-denial system has a "human review" step, but reviewing officers are told to approve the AI's call in over 95% of cases and are evaluated on throughput, not accuracy. When a discrimination complaint is investigated, regulators determine the review was rubber-stamping, not genuine independent judgment — meaning the EU AI Act's HITL requirement was violated in substance even with a human technically "in the loop."

A bank can describe exactly which layers and weights activated to produce a loan denial — genuine explainability — but the loan officer still can't tell the applicant anything meaningful about why their situation led to that outcome. Explainability without an interpretability layer leaves perfect technical detail and no usable answer.

A company's customer-facing chatbot processes documents uploaded by users, and an attacker embeds hidden instructions inside a PDF telling the model to ignore its safety guidelines and reveal internal system prompts. Because the malicious instructions arrived inside processed content rather than the visible chat message, this is indirect prompt injection — easy to miss if a team only screens direct user input.

A demand-forecasting model performs reliably for two years, then starts over-predicting after a competitor shifts customer behavior. Investigation determines this is concept drift, not data drift — the inputs look statistically similar, but the input-output relationship changed. The distinction matters because the fix differs from what a data-drift diagnosis would prescribe.