uniopen Tunes Amazon Nova 2 Lite to Meet Retail Moderation Targets

A moderation policy built around two questions

uniopen connects customers to ecommerce, membership benefits and other retail experiences across web, tablet and mobile channels. Across those channels, its moderation policy classifies each interaction along two axes. The first is what behavior occurred, with nine categories. The second is what subject the behavior refers to: brand, other, or forbidden.

Both labels must be correct for a moderation decision to be useful, AWS says. Both are also specific to uniopen's business, so a general-purpose model cannot be expected to learn them out of the box. The team therefore customized Amazon Nova 2 Lite, which handles the primary moderation requests in production.

Results: from baseline to above target

The team compared three configurations on the same held-out test set of 737 conversation windows. The fine-tuning dataset contained 3,391 training windows. A conversation window is a bounded segment of a customer conversation treated as one training or evaluation example.

Two metrics were used. Per Behavior Macro F1 measures how consistently the model classifies the nine behaviors as a group, with each behavior weighted equally. Subject Type Macro F1 does the same for brand, other and forbidden subjects. Equal weighting means strong results on common categories cannot hide weak results on harder ones.

• Baseline: Per Behavior Macro F1 of 0.5852 and Subject Type Macro F1 of 0.4162.
• Fine-tuned with LoRA: 0.8364 and 0.8302. The subject-type score exceeded its target, while behavior classification came close.
• Prompt-optimized: 0.8550 and 0.8491, against production targets of 0.8500 and 0.8200.

Fine-tuning produced the largest gain, especially for subject type. The prompt step changed the model output from JSON to a line-based format and clarified how multiple behaviors should be returned. According to AWS, this required no additional model training.

A workflow with human checks and release gates

The architecture separates the production moderation path from correction, training, evaluation and deployment. Amazon Nova 2 Pro helps generate candidate corrections for reported errors, but a human reviewer must verify each one before it enters the training set. That keeps generated labels from being treated as ground truth.

Verified corrections are stored in Amazon S3. Amazon DynamoDB tracks active and candidate model configurations. Argo Workflows on Amazon EKS orchestrates prompt optimization, evaluation and deployment, and Argo CD applies approved configurations to production. Amazon SNS and Amazon CloudWatch alert operators when a hard gate fails or a candidate needs attention.

Promotion is controlled by two kinds of checks. Hard gates are must-pass regression tests, and a failure stops the workflow and sends an alert. Soft gates are warning signals such as low confidence or a drop in performance for a specific class. If no soft-gate warning appears, a candidate can be promoted automatically. Otherwise it stays pending until an administrator approves it.

AWS also lists responsible AI controls: mandatory human review for ambiguous cases and reused corrections, fixed test sets and regression checks, optional Amazon Bedrock Guardrails content filters, error monitoring across categories, minimal retention of customer data, and revalidation of thresholds when policies change.

What it means for uniopen

Because both metrics cleared their release targets, uniopen can direct routine moderation to the customized path and keep human review focused on ambiguous cases, according to AWS. The company says this reduces unnecessary escalation while preserving a human decision point for uncertain or policy-sensitive content.

Using both metrics as release gates also stops an improvement in one dimension from masking a regression in the other. The team plans to keep Per Behavior Macro F1 and Subject Type Macro F1 as production gates. AWS notes that model availability varies by Region.