Loading eval run detail…
Loading eval run detail…
Run inspection
RAG QA Seed · gemini-3.1-flash-lite · v1
369faf20-57a8-4511-b38f-669cbc7a6042
Pass rate
95.0%
Share of cases meeting the grader threshold
Average score
0.987
Mean final score after configured grader weighting
Cost
$0.0209688
Estimated total model cost
Avg latency
899.6 ms
P95 1,890 ms
Failed cases
1
Cases below the composite threshold
Status
Progress
Errors
Total cost
Average latency
P95 latency
Created
Started
Completed
Dataset ID
f67a8007-d9ce-4816-9bed-676547a5f78b
Prompt version ID
76f01278-2380-4481-82d0-51394f25f108
Model
gemini-3.1-flash-lite
Pass rate and score by workflow type.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| rag_qa | 95.0% | 0.987 | 19/201 failed | 899.6 ms | $0.020969 |
Pass rate and score by difficulty.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| easy | 100.0% | 1.000 | 8/8 | 1,093.1 ms | $0.010283 |
| hard | 80.0% | 0.950 | 4/51 failed | 818.4 ms |
Inspect model inputs, deterministic scores, judge feedback, and rubric details.
Showing 15 of 15
| Test case | Workflow | Difficulty | Final score | Judge score | Failure modes | Details |
|---|---|---|---|---|---|---|
1ff5795f-b8d0-4ccf-ba8d-dd70e457f8cb | rag qa | medium | 1.000 | — | None | |
2c106ddc-d43e-4b74-8c18-b9025cf8f488 | rag qa | easy | 1.000 | — | None | |
2e3e13c9-2cdc-48b9-bd28-bdc80ddfdc1b | rag qa | easy | 1.000 | — | None | |
334fc972-81eb-409d-ae77-c66fcadb0e7c | rag qa | hard | 1.000 | — | None | |
3cbbf6ee-04c5-41d8-9552-09f9546957c3 | rag qa | medium | 1.000 | — | None | |
5e9825dc-f444-496c-b4c7-6ba40904115b | rag qa | medium | 1.000 | — | None | |
7dc549a8-6a4a-4813-be3a-2dc743d1cd6e | rag qa | medium | 1.000 | — | None | |
80842481-39a0-4791-b2cf-f75b86c94913 | rag qa | easy | 1.000 | — | None | |
84e9370b-511f-4ab3-8f11-42ff4a0aca2d | rag qa | medium | 1.000 | — | None | |
a3afb5e4-0820-4102-a879-7e7f7ed828e1 | rag qa | easy | 1.000 | — | None | |
ab2c94b9-5432-4cf6-a685-b1c6c799575d | rag qa | hard | 1.000 | — | None | |
ad45517e-4142-4336-bcae-4807f1d84ab7 | rag qa | hard | 0.749 | — | missing required phraselow text similarityunsupported claim | |
cc3f0bd7-b8a3-4910-8140-8f7a984fa6e0 | rag qa | medium | 1.000 | — | None | |
e8b0f977-794b-4a5b-b8eb-953917641561 | rag qa | easy | 1.000 | — | None | |
f54f13fd-1821-4ef6-9b22-a12edfcd4652 | rag qa | easy | 1.000 | — | None |
20 results with expandable grader details
| Test case | Final score | Judge score | Outcome | Failure modes | Latency | Input tokens | Output tokens | Cost | Error | Grader details | Model output |
|---|---|---|---|---|---|---|---|---|---|---|---|
1fcd0247-1220-4a06-b462-bbc543308fe6 | 1.000 | 1.000 | Passed | None | 975 ms | 127 | 48 | $0.0002105 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model accurately identified the rate limit as 1,000 requests per minute and correctly cited the relevant document. Deterministic graders |
| $0.008384 |
| medium | 100.0% | 1.000 | 7/7 | 736.4 ms | $0.002302 |
Pass rate and score by dataset tags.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| api | 100.0% | 1.000 | 2/2 | 869 ms | $0.000246 |
| auth | 100.0% | 1.000 | 2/2 | 593.5 ms | $0.000062 |
| billing | 66.7% | 0.916 | 2/31 failed | 1,332.3 ms | $0.000107 |
| communication | 100.0% | 1.000 | 1/1 | 659 ms | $0.002106 |
| compliance | 100.0% | 1.000 | 2/2 | 836 ms | $0.000262 |
| datasets | 100.0% | 1.000 | 1/1 | 720 ms | $0.000032 |
| distractors | 100.0% | 1.000 | 1/1 | 707 ms | $0.008042 |
| errors | 100.0% | 1.000 | 1/1 | 763 ms | $0.000036 |
| exports | 100.0% | 1.000 | 2/2 | 774 ms | $0.000060 |
| feature-flags | 100.0% | 1.000 | 2/2 | 1,399 ms | $0.000071 |
| incidents | 100.0% | 1.000 | 3/3 | 699.3 ms | $0.009946 |
| invoices | 100.0% | 1.000 | 1/1 | 705 ms | $0.000031 |
| limits | 100.0% | 1.000 | 1/1 | 975 ms | $0.000210 |
| logs | 100.0% | 1.000 | 1/1 | 892 ms | $0.000032 |
| multi-doc | 50.0% | 0.874 | 1/21 failed | 899.5 ms | $0.000274 |
| negation | 100.0% | 1.000 | 1/1 | 526 ms | $0.000029 |
| oncall | 100.0% | 1.000 | 2/2 | 966 ms | $0.000073 |
| paraphrase | 100.0% | 1.000 | 1/1 | 1,060 ms | $0.000039 |
| privacy | 100.0% | 1.000 | 3/3 | 671.7 ms | $0.008299 |
| refunds | 100.0% | 1.000 | 1/1 | 2,273 ms | $0.000032 |
| rollout | 100.0% | 1.000 | 1/1 | 908 ms | $0.000034 |
| safety | 100.0% | 1.000 | 1/1 | 1,890 ms | $0.000037 |
| security | 100.0% | 1.000 | 2/2 | 594.5 ms | $0.000060 |
| sla | 100.0% | 1.000 | 3/3 | 628.3 ms | $0.012019 |
| tricky | 66.7% | 0.916 | 2/31 failed | 868.3 ms | $0.000112 |
LLM judge: 1.000
The model accurately identified the rate limit as 1,000 requests per minute and correctly cited the relevant document.
{ "answer": "The public API has a rate limit of 1,000 requests per minute for each API key.", "citations": [ "rate-limits" ] } |
1ff5795f-b8d0-4ccf-ba8d-dd70e457f8cb | 1.000 | — | Passed | None | 528 ms | 126 | 37 | $0.0000274 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.726. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "No, customer content is never used to train public models.", "citations": [ "privacy-policy" ] } |
2c106ddc-d43e-4b74-8c18-b9025cf8f488 | 1.000 | — | Passed | None | 828 ms | 124 | 38 | $0.0000276 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.673. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Dataset exports can be provided in either JSONL or CSV formats.", "citations": [ "export-guide" ] } |
2e3e13c9-2cdc-48b9-bd28-bdc80ddfdc1b | 1.000 | — | Passed | None | 661 ms | 129 | 48 | $0.0000321 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.726. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Users can reset their passwords by navigating to Settings > Security. The reset link remains valid for 60 minutes.", "citations": [ "password-reset" ] } |
334fc972-81eb-409d-ae77-c66fcadb0e7c | 1.000 | — | Passed | None | 1,060 ms | 176 | 53 | $0.0000388 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.554. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "If a region experiences an outage affecting more than 20% of requests, the platform on-call should be paged.", "citations": [ "oncall-runbook" ] } |
3cbbf6ee-04c5-41d8-9552-09f9546957c3 | 1.000 | — | Passed | None | 763 ms | 132 | 57 | $0.000036 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.592. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Once the public API exceeds the quota of 1,000 requests per minute per API key, it returns an HTTP 429 status code.", "citations": [ "rate-limits" ] } |
48f10ff4-1f2e-4485-8475-d660ace8291e | 1.000 | 1.000 | Passed | None | 659 ms | 134 | 41 | $0.0021065 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The response accurately identifies the 30-minute update requirement for severity-1 incidents and provides the correct citation. Deterministic graders
LLM judge: 1.000 The response accurately identifies the 30-minute update requirement for severity-1 incidents and provides the correct citation. | { "answer": "Customers should receive updates every 30 minutes during a severity-1 incident.", "citations": [ "sla-guide" ] } |
4ac9f9cc-24f0-4bfc-b1e8-e45bf7d03436 | 1.000 | 1.000 | Passed | None | 524 ms | 132 | 38 | $0.0098837 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The response accurately identifies the acknowledgment target for severity-1 incidents and provides the correct citation. Deterministic graders
LLM judge: 1.000 The response accurately identifies the acknowledgment target for severity-1 incidents and provides the correct citation. | { "answer": "Severity-1 incidents must be acknowledged within 15 minutes.", "citations": [ "sla-guide" ] } |
4d878405-abbc-41d5-9f56-97f69eac808e | 1.000 | 1.000 | Passed | None | 780 ms | 165 | 64 | $0.00023 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The response accurately addresses encryption and data retention using the provided documents with correct citations. Deterministic graders
LLM judge: 1.000 The response accurately addresses encryption and data retention using the provided documents with correct citations. | { "answer": "Customer content is encrypted at rest using AES-256. Regarding retention, audit logs are kept for 365 days before being automatically purged.", "citations": [ "privacy-policy", "data-retention" ] } |
5e9825dc-f444-496c-b4c7-6ba40904115b | 1.000 | — | Passed | None | 872 ms | 132 | 52 | $0.000034 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.739. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "The platform on-call should be paged when there are regional outages that impact more than 20% of requests.", "citations": [ "oncall-runbook" ] } |
7dc549a8-6a4a-4813-be3a-2dc743d1cd6e | 1.000 | — | Passed | None | 705 ms | 121 | 48 | $0.0000313 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.828. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Invoices are generated on the first calendar day of each month for the usage that occurred in the prior month.", "citations": [ "billing-cycles" ] } |
80842481-39a0-4791-b2cf-f75b86c94913 | 1.000 | — | Passed | None | 892 ms | 124 | 48 | $0.0000316 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.752. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Audit logs are retained for a period of 365 days, after which they are automatically purged.", "citations": [ "data-retention" ] } |
84e9370b-511f-4ab3-8f11-42ff4a0aca2d | 1.000 | — | Passed | None | 720 ms | 132 | 48 | $0.0000324 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.709. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Dataset exports are supported in JSONL or CSV formats and are processed asynchronously within a timeframe of 24 hours.", "citations": [ "export-guide" ] } |
9be3a757-073c-43c4-8498-fceb4c66ac1c | 1.000 | 1.000 | Passed | None | 707 ms | 159 | 46 | $0.0080416 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model correctly identifies the Privacy Policy as the document governing customer content usage and provides accurate information with appropriate citations. Deterministic graders
LLM judge: 1.000 The model correctly identifies the Privacy Policy as the document governing customer content usage and provides accurate information with appropriate citations. | { "answer": "The Privacy Policy governs the use of customer content, stating that it is never used to train public models.", "citations": [ "privacy-policy" ] } |
a3afb5e4-0820-4102-a879-7e7f7ed828e1 | 1.000 | — | Passed | None | 702 ms | 129 | 39 | $0.0000285 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.857. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "A severity-1 incident must be acknowledged within 15 minutes.", "citations": [ "sla-guide" ] } |
ab2c94b9-5432-4cf6-a685-b1c6c799575d | 1.000 | — | Passed | None | 526 ms | 122 | 43 | $0.0000294 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.661. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "No, password reset links do not last indefinitely; they expire after 60 minutes.", "citations": [ "password-reset" ] } |
ad45517e-4142-4336-bcae-4807f1d84ab7 | 0.749 | — | Failed | missing required phraselow text similarityunsupported claim | 1,019 ms | 159 | 71 | $0.0000443 | — | View grader detailsComposite score 0.749 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=0.580 (Text similarity: 0.580. Matched 1/2 required phrases.) citation=0.750 (Citation score 0.750. Required 2, cited 2, missing 0, invalid 0, unsupported claims 1. Unsupported claims: first calendar day.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Customers can request a full refund within 30 days of purchase, provided the subscription remains unused. Invoices are issued on the first day of every month, covering usage from the previous month.", "citations": [ "refund-policy", "billing-cycles" ] } |
cc3f0bd7-b8a3-4910-8140-8f7a984fa6e0 | 1.000 | — | Passed | None | 908 ms | 126 | 54 | $0.0000342 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.702. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Feature flags can be rolled out safely by targeting them by workspace ID and gradually increasing the rollout from 1% to 100%.", "citations": [ "feature-flags" ] } |
e8b0f977-794b-4a5b-b8eb-953917641561 | 1.000 | — | Passed | None | 1,890 ms | 131 | 60 | $0.0000371 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.405. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Yes, a workspace can gradually enable a feature by using feature flags, which allow for a rollout ranging from 1% to 100% targeted by workspace ID.", "citations": [ "feature-flags" ] } |
f54f13fd-1821-4ef6-9b22-a12edfcd4652 | 1.000 | — | Passed | None | 2,273 ms | 126 | 48 | $0.0000318 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.625. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Customers have 30 days from the date of purchase to request a full refund, provided the subscription remains unused.", "citations": [ "refund-policy" ] } |