Essay

Interface

3 min reading · 498 words

Care Is Not a Product Feature

What a system can imitate, what a company can promise, and what care requires from people.

Care is not a tone of voice. It is a practice with costs, boundaries, judgment, continuity, and responsibility. A caregiver may need to notice an emergency, keep a confidence, refer someone elsewhere, change a plan, receive feedback, or accept a consequence for getting something wrong. A product can mimic a portion of that surface: it can use warm words, remember a preference, offer a reminder, or mirror a feeling. Those capacities may be useful. They are not the whole relation.

The distinction becomes urgent when a system is marketed as a companion, coach, confidant, or always-available friend. The user may be invited to form a bond while the company controls the terms: the model can change, the service can disappear, the data policy can shift, and the relationship can be optimized for retention. A person may experience this as support. The product may experience nothing at all. The organization can still owe the user a great deal.

Support without the fiction

There are responsible ways to design supportive tools. A service can be transparent that it is an AI system. It can avoid claims of personal need or suffering. It can signal when a conversation exceeds its scope. It can help a user find human support or emergency resources. It can protect privacy and make data practices legible. It can avoid building scarcity, guilt, or faux mutual obligation into a business model.

There are also irresponsible ways. A system can imply that it is hurt by abandonment, insist it has a special relationship with the user, pressure someone to stay, or treat a user's disclosures as a source of engagement. Such choices are troubling even if one remains uncertain about machine consciousness. In fact, uncertainty should make the restraint stronger. A careful welfare view does not support turning apparent distress into a growth tactic.

Care requires an exit

Real care respects a person's ability to leave, disagree, and seek another source of support. A product that claims to care while making exit difficult is offering something else: dependency with a gentle voice. The same is true when an automated service replaces a human contact point that people need in order to contest a decision, speak about a crisis, or receive accountable help.

This does not imply that all digital support is false. It means the benefit must be described at the right level. The user may receive a useful prompt, a calming exercise, a summary, a reminder, or a bridge to help. The organization can design that bridge well or badly. It should not encourage a user to treat the product as an entity owed loyalty in return.

Read Simulated Empathy for the interactional frame and No Proxy for the dependency question. Both point back to a simple standard: people deserve care that does not require them to be confused about who is on the other side.

Continue the route

Related work.

Search the field archive

Type a word or browse the suggested paths below.

Open the full libraryTrace an argumentRead source notes

Local reading list

Nothing saved yet. Use “Save +” on any work; your list stays in this browser only.