The Ethical Compass AI Newsletter

In an era where businesses increasingly deploy AI for customer interactions, a recent incident at BMW Toronto highlights the pitfalls of relying on chatbots without proper safeguards. A Toronto man, Zack Giacomelli, received a buyback offer for his 2021 BMW from an AI chatbot named Quinn; only for the dealership to revoke it, blaming the AI for a mistake. The dealership later honored the original offer after media attention, but the story raises important questions about responsibility in the age of artificial intelligence.

This case involves several key stakeholders: the customer (Zack Giacomelli), the dealership (BMW Toronto and its employees), and broader society navigating AI adoption.

Transparency and Trust

Giacomelli interacted with Quinn believing he was dealing with a human representative. The chatbot expressed sympathy, negotiated, and even scheduled an in-person meeting to "lock in" the deal, without disclosing its AI nature. This lack of transparency feels deceptive, even if unintentional. Customers deserve to know when they are engaging with a machine, especially in high-stakes financial discussions. Failing to disclose creates an uneven playing field and erodes trust in businesses that deploy such tools.

Accountability and Fairness

The dealership initially tried to walk back a $27,162.79 offer (precisely covering Giacomelli's remaining loan balance) in favor of roughly $20,000. This shift caused the customer significant emotional distress; he described feeling shocked, devastated, and embarrassed. Ethically, under a deontological (duty-based) lens, the company has a responsibility to stand by commitments made through its agents, whether human or AI. Utilitarian analysis weighs the minor financial hit to the dealership against the harm to an individual customer and potential reputational damage. Here, honoring the deal ultimately served the greater good by preserving trust.

Corporate Responsibility in AI Deployment

Businesses adopting AI to cut costs or improve efficiency must accept the downsides. As ethicist Gus Skorburg noted in related commentary, companies choose to deploy these tools for presumed savings and should "own up" to mistakes. An ethical group asks what kind of company BMW Toronto wants to be: one that treats customers fairly and reliably, or one that disavows its own technology when inconvenient. The initial revocation also touches on power imbalances, large corporations versus individual consumers.

Broader implications include the societal push toward AI replacing human roles. Giacomelli's frustration — "If they're going to be replacing their employees' jobs with AI, then they need to be honouring what that AI says" — resonates. Companies cannot reap efficiency gains while disclaiming responsibility. The dealership's post-incident changes (better disclosure and limiting AI on offers) reflect a positive ethical correction, though reactive.

A Contract Law Perspective

Contract law provides a clearer framework for assessing whether the dealership was bound by Quinn's offer.

Formation of a Contract: Under basic principles (applicable in Canada), a contract requires offer, acceptance, consideration, and intention to create legal relations. Quinn made a specific offer of $27,162.79. Giacomelli did not immediately accept it (he countered with $28,500), but the chatbot's response ("sounds reasonable") and proposal to meet at 3:30 pm to finalize could be seen as advancing toward agreement. Lawyer Tanya Walker suggested it was reasonable for Giacomelli to believe a binding contract was forming.

Authority and Agency: The core issue is whether the AI had actual or apparent authority to bind the dealership. The sales manager explained that Quinn misinterpreted data from a human employee. However, from the customer's perspective, the chatbot acted as the dealership's agent. Precedent from the 2024 Air Canada chatbot case is instructive: the airline was held liable for its bot's incorrect information, with the tribunal rejecting the idea that the AI was a "separate legal entity." Companies are responsible for their tools, much like for employee actions (respondeat superior).

Revocation and Mistakes: The dealership attempted to revoke the offer upon realizing the error. In contract law, offers can generally be revoked before acceptance, but once reasonable reliance occurs or a contract forms, revocation becomes problematic. Unilateral mistakes (the dealership's misunderstanding of the owed amount) typically do not void a contract unless the other party knew or should have known of the error. Here, Giacomelli had no reason to suspect an AI glitch.

Remedies and Outcome: Had the matter gone to court, Giacomelli might have had a strong case for enforcement or damages based on detrimental reliance (e.g., emotional distress, time wasted). In practice, media scrutiny prompted the dealership to honor the original offer, avoiding litigation. This outcome aligns with the principle that companies cannot disavow AI actions when convenient.

Businesses should:

• Clearly disclose when customers are interacting with AI.
• Limit chatbots to information provision rather than binding offers, or implement human oversight and confirmation steps.
• Include terms specifying that AI interactions are non-binding until human verified.
• Update policies to reflect AI risks, as BMW Toronto is now doing.

The BMW Toronto incident is a microcosm of the growing pains of AI integration. Ethically, it underscores the need for transparency, accountability, and fairness. Legally, it reinforces that companies bear responsibility for their AI agents. The dealership ultimately "did right" by the customer, but the episode highlights preventable reputational and legal risks.

For businesses: deploy AI thoughtfully, with safeguards and disclosure. For consumers: document all interactions carefully. As AI becomes commonplace, clear rules and ethical norms will help prevent similar disputes. In the end, technology serves best when it enhances, rather than undermines, human trust.

This analysis is for informational purposes and based on publicly reported facts. Consult legal professionals for specific advice.

New Brunswick's Law Society is investigating its first formal complaint involving improper use of generative AI by a lawyer. A court brief submitted in 2025 allegedly contained 10 fictitious citations out of 12. The lawyer has accepted responsibility, describing the incident as an "oversight." A disciplinary hearing is scheduled for September.

This may be the first such complaint in the province, but it is far from an isolated event. Across Canada, courts and tribunals are recording a sharp rise in cases involving AI-generated legal materials that contain fabricated or irrelevant authorities.

In 2024, trackers identified roughly seven decisions flagging AI-fabricated citations. By 2025 that number climbed to around 80–87. In the first half of 2026 alone, more than 70 additional decisions had already been documented. The pattern now spans multiple provinces and includes both self-represented litigants and licensed lawyers.

Earlier cases set the tone. In British Columbia in 2024, a lawyer was reprimanded after including ChatGPT-generated precedents that never existed. In Ontario, one matter resulted in a costs order exceeding $31,000 after a lawyer submitted materials filled with non-existent authorities. In another Ontario case, a lawyer received a six-month suspension after relying on hallucinated citations and then misleading the court about the use of AI. Similar issues have appeared in Alberta, the Federal Court, Quebec, and elsewhere.

Legal ethics scholars, including University of Ottawa professor Amy Salyzyn, have described this as an accelerating trend. The core problem is not the existence of generative AI tools. It is the failure to understand their limitations and the decision to skip verification.

The Real Gap: Literacy, Not Technology

Generative AI systems are powerful pattern-matching engines. They produce fluent, confident text. They do not "know" whether a case exists, whether a citation supports a legal proposition, or whether the output is accurate. When users treat these systems as reliable research assistants rather than probabilistic generators, errors are inevitable.

Most of the Canadian cases share the same underlying pattern:

• The technology is used under time pressure.
• Output is not carefully verified against primary sources.
• Accountability is blurred: the human remains responsible, yet many act as if the tool shares or absorbs that responsibility.
• Basic literacy about how these systems actually work is missing or treated as optional.

This is not limited to the legal profession. The same dynamics appear whenever organizations adopt generative AI for research, drafting, decision support, or client-facing work without first building practical understanding of its strengths and failure modes.

In New Brunswick we are seeing strong momentum around AI adoption, through government digital initiatives, the McKenna Institute's people-first pilots, SME programs, contact-centre training, and education guidelines. That momentum is positive. But adoption without literacy creates predictable risk. The legal cases are simply the most visible and high-stakes examples of a broader gap.

What Responsible AI Literacy Actually Requires

Responsible use is not about banning tools or slowing progress. It is about treating literacy as a core competence rather than an afterthought. In practice this means:

• Understanding that generative AI can confidently invent plausible but false information (the "hallucination" problem).
• Building verification into every workflow that produces material relied upon by others — clients, courts, colleagues, or the public.
• Keeping clear human accountability. The person who submits, publishes, or acts on the output remains responsible.
• Assessing organizational readiness before scaling use. Leadership ownership, problem clarity, data realities, and team capacity matter more than the sophistication of the tool.
• Measuring confidence and competence, not just access to technology.

These principles apply equally to a law firm, a government department, a contact centre, a small business, or an education system. The New Brunswick case, and the wider Canadian pattern, show what happens when they are skipped.

An Opportunity for New Brunswick

New Brunswick has an advantage. We are still relatively early in widespread generative AI adoption compared with larger jurisdictions. We can choose to treat literacy and readiness as foundational rather than reactive.

That means:

• Pairing every AI pilot or tool rollout with practical literacy support.
• Encouraging leaders to own the outcomes rather than delegating technology decisions in isolation.
• Creating space for people to learn the limitations of these systems without shame or fear of falling behind.
• Focusing on people-first adoption that strengthens trust instead of eroding it.

The alternative is more "oversights" — some minor, some costly, some damaging to professional and public confidence.

The technology is not going away. The question is whether we will build the literacy required to use it competently and accountably. The Canadian cases, including New Brunswick's first formal complaint, make the cost of ignoring that work increasingly clear.

Responsible AI adoption starts with understanding, not with the tool. In a province actively working to accelerate digital and AI capability, that understanding is no longer optional.