Root Cause Analysis
Queried: Jun 18, 2026 – Jun 20, 2026 | Generated:
DANGER
Slow external dependency: Invoke averaging 106s per call
Key Findings
- P95 = 397.6s | Average = 106.8s over last 24h
- Slowest dependency: Invoke — avg 105.9s, P95 319.6s (61 calls)
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Reduce HttpClient timeout for Invoke and add exponential backoff retry. Consider provisioned capacity (e.g. Azure OpenAI PTU) for consistent latency.
- P95 exceeds 90s — review function timeout in host.json and HttpClient.Timeout settings. Fail fast and retry rather than waiting for the full timeout.
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 148.2s
Peak P95: 397.6s
at hour 10:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 397.6s | 4523111e… | |||
| 396.7s | 4523111e… | |||
| 362.0s | 4523111e… | |||
| 355.0s | 4523111e… | |||
| 332.4s | 4523111e… | |||
| 332.4s | 4523111e… | |||
| 319.7s | 4523111e… | |||
| 318.9s | 4523111e… | |||
| 246.8s | 4523111e… | |||
| 246.7s | 4523111e… |
All from 1 instance
Warning / Error Traces
38
| Sev | Message | Count | Last Seen | |
|---|---|---|---|---|
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'Dickerson, Donna S.;Dinges, Dennis R;Edwards, Robyn;Egan, Janet M.;Eggers, Margaret;Ehret, Jason;Falcetta, Dorothy S.;Fletcher, Michael;Forbes, Grace M;Gaddy, William H.;Gill, Jeffrey S;Granger, Daniel J;Gray, Danny;Griggs, Sharee L.;Grubisic, Vukosava;Hall, Anna E.;Hanon, Tabatha J;Hart, Priscilla;Hatcher, Thomas', PageId: 05f85f79-3796-4847-b33b-28855e1f0b08, ClassifiedAs: Multi Orders. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'DEPUY, BETTY', PageId: 79588b2b-a0d9-4448-be3b-40e45ae3806d, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'HALLOCK, TOMMIE', PageId: f6a78a1c-9d80-406d-bdc2-45b69fcfda71, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'PEAL, BYTHELLA', PageId: ba581aa6-30e4-4587-9758-a22453fdebc7, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'BAILEY, MARGARETT', PageId: 10b69990-30a9-496e-ab9f-af28ad05d615, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'Allen, David;Alsultani, Thaer;Anderson, Willard;Ansehl, Donna L;Basile, Barbara A;Besher, Linda M.;Blevins, Robert E.;Bobo, Nicole;Brogdon, Micheal W;Brown, Hae O.;Brown, Karl R;Buhrmester, Barbara;Chancellor, Adolphus;Clymer, Ann F.;Cole, Jocelyn M.;Crabtree, James;Crow, Bluedolphin;Davis, Nancy', PageId: 41b3a33b-2412-4a36-8f51-c4cdd5d221b8, ClassifiedAs: Multi Orders. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'KUKIC, MIRZETA', PageId: fdc9033e-0e95-4d30-a809-92b5f67d4249, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'CROMARTIE, MARGARET', PageId: 98df873c-0f82-42af-bbfa-45d161fa11fe, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'MAI, AN', PageId: 3f2173c8-75d7-45f0-8515-1a2b34866bec, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'Heierman, Brian D.;Helfrich, Susan K;Henson, Jermaine A;Hudson, Brenda D.;Jeske, Steven H;Johnson, Daniel L.;Jones, Betsy M;Jones, Latifah F;Kim, Scarlet;Knapp, Michael D.;Knaust, Betty;Kosulandich, Priscilla;Kranawetter, Allen D;Kupfer, William J;Labriere, Joann;Leachman, Mark;Lee, Rebecca;Lindner, Ronald A;Lindwedel, David R', PageId: 95a7cf9c-24d2-472b-8835-fe942a19dcf2, ClassifiedAs: Multi Orders. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'BARR JR., ARTHUR', PageId: 120a8e0b-0b92-40b3-8433-f3038e15cb92, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'ADAMS, FLORINDA', PageId: c0c72761-6284-4609-8bef-525ded25b78b, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'DEIE, DANIEL', PageId: 4299039f-609d-442d-aebd-df1c41d20829, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'Phung, Phillip;Powell, Clarence F;Ramey, Jacquelyn M;Rensing, Robin;Sampson, Charles L.;Sanyangore, Cleopus P.;Stacy, Robert Charles;Stasik, Dorothy C;Stewart, James D.;Strickland, Louise M.;Szewczuk, Lois A;Velten, Cheryl L;Watkins, Willie;Wells, Patricia M;White, Jordache;Whitfield, Lacarl;Williams, Edward L.;Williams, Marvin;Williams, Samuel A', PageId: 0a5814b2-0f09-4e57-963b-6afece025e5d, ClassifiedAs: Multi Orders. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'GRIBBIN, DELICIA', PageId: 68473c5e-e72d-447a-ae57-8e8086a68158, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'Williams, Susan;Winstead, Mickey T;Woodard Jr, Sherman;Woody, Janis K', PageId: 78b21db7-c44d-40f9-9e68-ef4a29a46979, ClassifiedAs: Multi Orders. Review for possible name mismatch. | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'WILLIAMS, BEVERLY', PageId: 10c031f1-cdb0-44e1-b0d7-bc85125f67ea, ClassifiedAs: Face sheet. Review for possible name mismatch. | 2 | ||
| Warning | No DocExtractionConfig found for ClassifiedAs: BlankPage. Falling back to regex. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: 0b7c3aad-cc15-4899-a128-57f0774b5cb9 | 2 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'Gonzales, Maria', PageId: d8c4fc8d-bce2-4e72-bea5-2e32ab0c3bd0, ClassifiedAs: ORDER. Review for possible name mismatch. | 1 | ||
| Warning | DocumentNormalization: lone page with no group peers. PatientName: 'CARVAJAL, MANUEL', PageId: c89957a0-8a54-45ab-9803-59a928eefd1b, ClassifiedAs: Face sheet. Review for possible name mismatch. | 1 |
Swallowed Exceptions
No exceptions in the last 24h.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| Invoke | InProc | 61 | 105.9s | 319.6s | 0 | 0.0% |
| devreferralqopenai.openai.azure.com | HTTP | 199 | 65.2s | 111.7s | 0 | 0.0% |
| cachesync-api.azurewebsites.net | HTTP | 185 | 124ms | 305ms | 0 | 0.0% |
| BlobBaseClient.DownloadContent | InProc | Microsoft.Storage | 985 | 9ms | 18ms | 0 | 0.0% |
| BlobBaseClient.Exists | InProc | Microsoft.Storage | 1,148 | 8ms | 21ms | 0 | 0.0% |
| devreferralq.blob.core.windows.net | Azure blob | 2,133 | 8ms | 20ms | 0 | 0.0% |
Show 10 internal / host RPC calls
| Name | Calls | Avg | P95 |
|---|---|---|---|
| POST /Settlement/Complete | 4 | 4851ms | 19229ms |
| POST /Settlement/RenewSessionLock | 7 | 1579ms | 7845ms |
| POST /Settlement/Complete | 1 | 181ms | 181ms |
| POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream | 1 | 129ms | 129ms |
| POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream | 1 | 119ms | 119ms |
| POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream | 1 | 118ms | 118ms |
| POST /Settlement/Complete | 12 | 68ms | 169ms |
| POST /Settlement/Complete | 44 | 54ms | 110ms |
| POST /Settlement/RenewSessionLock | 3 | 37ms | 99ms |
| POST /Settlement/RenewSessionLock | 9 | 10ms | 12ms |
DANGER
High exception volume (62 in 24h) — top: InvalidOperationException
Key Findings
- P95 = 45.4s | Average = 1.7s over last 24h
- Slowest dependency: Invoke — avg 2.0s, P95 3.8s (62 calls)
- High-failure dependency: PUT devreferralq — 14% failure rate (62/429 calls)
- 62 swallowed exceptions in 24h — all reported success=true, masking real failures
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Fix exception swallowing — let failures propagate so success=false is recorded and the Service Bus message is dead-lettered for reprocessing instead of silently dropped.
- Investigate 14% failure rate on PUT devreferralq: check endpoint health, RBAC permissions, payload size, and schema compatibility.
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 5.5s
Peak P95: 45.4s
at hour 14:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 45.4s | 4523111e… | |||
| 6.0s | 4523111e… | |||
| 4.4s | 4523111e… | |||
| 3.8s | 4523111e… | |||
| 3.2s | 4523111e… | |||
| 2.8s | 4523111e… | |||
| 2.7s | 4523111e… | |||
| 2.1s | 4523111e… | |||
| 2.0s | 4523111e… | |||
| 1.9s | 4523111e… |
All from 1 instance
Warning / Error Traces
No warning or error traces in the last 24h.
Swallowed Exceptions
62
Type:
System.InvalidOperationExceptionMessage: An invalid request URI was provided. Either the request URI must be an absolute URI or BaseAddress must be set.
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| Invoke | InProc | 62 | 2.0s | 3.8s | 0 | 0.0% |
| cachesync-api.azurewebsites.net | HTTP | 186 | 354ms | 285ms | 0 | 0.0% |
| BlobClient.Upload | InProc | Microsoft.Storage | 367 | 50ms | 140ms | 0 | 0.0% |
| devreferralq.blob.core.windows.net | Azure blob | 491 | 42ms | 138ms | 62 | 12.6% |
| BlobBaseClient.DownloadTo | InProc | Microsoft.Storage | 62 | 33ms | 70ms | 0 | 0.0% |
| BlobContainerClient.CreateIfNotExists | InProc | Microsoft.Storage | 62 | 6ms | 7ms | 0 | 0.0% |
Show 1 internal / host RPC calls
| Name | Calls | Avg | P95 |
|---|---|---|---|
| POST /Settlement/Complete | 62 | 58ms | 114ms |
WARNING
P95 duration elevated: 51s peak
Key Findings
- P95 = 50.6s | Average = 8.7s over last 24h
- Slowest dependency: Invoke — avg 8.9s, P95 16.7s (328 calls)
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 16.8s
Peak P95: 50.6s
at hour 21:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 51.7s | 4523111e… | |||
| 50.7s | 4523111e… | |||
| 50.6s | 4523111e… | |||
| 50.6s | 4523111e… | |||
| 50.2s | 4523111e… | |||
| 45.5s | 4523111e… | |||
| 44.8s | 4523111e… | |||
| 43.3s | 4523111e… | |||
| 36.4s | 4523111e… | |||
| 27.9s | 4523111e… |
All from 1 instance
Warning / Error Traces
No warning or error traces in the last 24h.
Swallowed Exceptions
No exceptions in the last 24h.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| Invoke | InProc | 328 | 8.9s | 16.7s | 0 | 0.0% |
| DocumentAnalysisClient.AnalyzeDocument | InProc | Microsoft.CognitiveServices | 328 | 7.0s | 7.4s | 0 | 0.0% |
| AnalyzeDocumentOperation.WaitForCompletion | InProc | Microsoft.CognitiveServices | 328 | 6.5s | 6.8s | 0 | 0.0% |
| cachesync-api.azurewebsites.net | HTTP | 711 | 732ms | 372ms | 0 | 0.0% |
| doc-intelligence-referralq.cognitiveservices.azure.com | HTTP | 1,037 | 237ms | 728ms | 0 | 0.0% |
| AnalyzeDocumentOperation.UpdateStatus | InProc | Microsoft.CognitiveServices | 709 | 148ms | 322ms | 0 | 0.0% |
| BlobBaseClient.DownloadTo | InProc | Microsoft.Storage | 328 | 24ms | 56ms | 0 | 0.0% |
| devreferralq.blob.core.windows.net | Azure blob | 1,186 | 14ms | 47ms | 0 | 0.0% |
| BlobClient.Upload | InProc | Microsoft.Storage | 858 | 12ms | 22ms | 0 | 0.0% |
Show 1 internal / host RPC calls
| Name | Calls | Avg | P95 |
|---|---|---|---|
| POST /Settlement/Complete | 328 | 67ms | 128ms |
WARNING
P95 duration elevated: 71s peak
Key Findings
- P95 = 70.5s | Average = 7.5s over last 24h
- Slowest dependency: Invoke — avg 7.0s, P95 15.9s (462 calls)
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 14.7s
Peak P95: 70.5s
at hour 18:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 70.5s | 4523111e… | |||
| 35.6s | 4523111e… | |||
| 23.1s | 4523111e… | |||
| 23.1s | 4523111e… | |||
| 23.1s | 4523111e… | |||
| 22.7s | 4523111e… | |||
| 22.2s | 4523111e… | |||
| 21.7s | 4523111e… | |||
| 20.7s | 4523111e… | |||
| 20.2s | 4523111e… |
All from 1 instance
Warning / Error Traces
No warning or error traces in the last 24h.
Swallowed Exceptions
No exceptions in the last 24h.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| Invoke | InProc | 462 | 7.0s | 15.9s | 0 | 0.0% |
| fax-receiver-api-dev.azurewebsites.net | HTTP | 111 | 685ms | 7.1s | 0 | 0.0% |
SUCCESS
Function is performing within normal bounds
Key Findings
- P95 = 21.3s | Average = 2.1s over last 24h
- Slowest dependency: Invoke — avg 1.8s, P95 7.4s (460 calls)
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 4.9s
Peak P95: 21.3s
at hour 22:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 21.4s | 4523111e… | |||
| 21.3s | 4523111e… | |||
| 20.6s | 4523111e… | |||
| 17.2s | 4523111e… | |||
| 17.2s | 4523111e… | |||
| 17.2s | 4523111e… | |||
| 17.2s | 4523111e… | |||
| 17.1s | 4523111e… | |||
| 17.1s | 4523111e… | |||
| 17.1s | 4523111e… |
All from 1 instance
Warning / Error Traces
No warning or error traces in the last 24h.
Swallowed Exceptions
No exceptions in the last 24h.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| Invoke | InProc | 460 | 1.8s | 7.4s | 0 | 0.0% |
| cachesync-api.azurewebsites.net | HTTP | 1,223 | 263ms | 963ms | 0 | 0.0% |
| BlobBaseClient.DownloadContent | InProc | Microsoft.Storage | 448 | 82ms | 431ms | 0 | 0.0% |
| devreferralq.blob.core.windows.net | Azure blob | 448 | 81ms | 430ms | 0 | 0.0% |
Show 3 internal / host RPC calls
| Name | Calls | Avg | P95 |
|---|---|---|---|
| POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream | 1 | 119ms | 119ms |
| POST /Settlement/Complete | 337 | 99ms | 336ms |
| POST /Settlement/Complete | 123 | 88ms | 193ms |
SUCCESS
Function is performing within normal bounds
Key Findings
- P95 = 3.5s | Average = 0.5s over last 24h
- Slowest dependency: Invoke — avg 0.4s, P95 0.7s (82 calls)
- High-failure dependency: Invoke — 24% failure rate (20/82 calls)
- 20 swallowed exceptions in 24h — all reported success=true, masking real failures
- 21 error-level trace messages logged
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Fix exception swallowing — let failures propagate so success=false is recorded and the Service Bus message is dead-lettered for reprocessing instead of silently dropped.
- Investigate 24% failure rate on Invoke: check endpoint health, RBAC permissions, payload size, and schema compatibility.
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.7s
Peak P95: 3.5s
at hour 10:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 3.5s | 4523111e… | |||
| 3.2s | 4523111e… | |||
| 1.0s | 4523111e… | |||
| 0.8s | 4523111e… | |||
| 0.8s | 4523111e… | |||
| 0.7s | 4523111e… | |||
| 0.6s | 4523111e… | |||
| 0.5s | 4523111e… | |||
| 0.5s | 4523111e… | |||
| 0.5s | 4523111e… |
All from 1 instance
Warning / Error Traces
84
| Sev | Message | Count | Last Seen | |
|---|---|---|---|---|
| Error | Message processing error (Action=ProcessMessageCallback, EntityPath=groupingqueue, Endpoint=devsbreferralq.servicebus.windows.net) | 20 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 5b3eb979-df06-456f-a2ad-e6de89683cbb. Creating a single packet as a possible supplemental document. | 20 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 0e688650-4849-4a55-a128-1f46b6496247. Creating a single packet as a possible supplemental document. | 14 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 4b68c167-b4ed-4d1c-902a-a76d49f48e5a. Creating a single packet as a possible supplemental document. | 5 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 4f923857-4980-4cf6-82ac-f8255c8a5251. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 6 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 2 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 296490f8-4c64-4f1b-ba9b-e50644f463de. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 5 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 4 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId d4a3c62e-f1a6-4b79-b4cd-4be7d8395224. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 3 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 1 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. | 2 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId c843ac34-3cfb-4806-84b8-92c7e2e6fbac. Creating a single packet as a possible supplemental document. | 1 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 1a93bd64-7862-43be-a372-75fb453d1014. Creating a single packet as a possible supplemental document. | 1 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId acfb2a8a-2b3e-4ae3-bf2e-b1994753ce9b. Creating a single packet as a possible supplemental document. | 1 | ||
| Error | API call failed. DocumentId: 5b3eb979-df06-456f-a2ad-e6de89683cbb, StatusCode: InternalServerError, TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, Service: PatientPacketProcessor. Response: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"6c557943b3b9","details":[]} | 1 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 0cd139c8-e634-46bc-959b-0c74ad2ee8cf. Creating a single packet as a possible supplemental document. | 1 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 946bd030-514f-48f2-8b1a-588166bb48a4. Creating a single packet as a possible supplemental document. | 1 | ||
| Warning | DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 688e9037-a9b0-490f-b4b9-02be42a7ce2f. Creating a single packet as a possible supplemental document. | 1 |
Swallowed Exceptions
20
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"6c557943b3b9","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"6ba05767f233","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"d9f9c44234f9","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"de3e81ab1b4c","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"af2f8870e3e5","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"7f84747b2d49","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"617d3c5b6879","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"7842637b9b7d","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"dc191ea84733","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
Type:
System.ExceptionMessage: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"3acb1ea698cc","details":[]}
Last seen:
Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| Invoke | InProc | 82 | 355ms | 705ms | 20 | 24.4% |
| cachesync-api.azurewebsites.net | HTTP | 266 | 99ms | 275ms | 20 | 7.5% |
SUCCESS
Function is performing within normal bounds
Key Findings
- P95 = 1.7s | Average = 0.7s over last 24h
- Slowest dependency: Invoke — avg 0.6s, P95 1.2s (67 calls)
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.9s
Peak P95: 1.7s
at hour 21:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 1.7s | 4523111e… | |||
| 1.6s | 4523111e… | |||
| 1.5s | 4523111e… | |||
| 1.2s | 4523111e… | |||
| 1.2s | 4523111e… | |||
| 1.1s | 4523111e… | |||
| 1.0s | 4523111e… | |||
| 0.9s | 4523111e… | |||
| 0.9s | 4523111e… | |||
| 0.9s | 4523111e… |
All from 1 instance
Warning / Error Traces
97
| Sev | Message | Count | Last Seen | |
|---|---|---|---|---|
| Warning | Facility validation failed. FacilityName: Garnet Hill Rehab & Skilled Care, FacilityAddress: [not extracted], State: , PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. | 28 | ||
| Warning | Facility validation failed. FacilityName: [not extracted], FacilityAddress: [not extracted], State: [not extracted], PacketId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32, DocumentId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32. | 12 | ||
| Warning | Facility validation failed. FacilityName: [not extracted], FacilityAddress: [not extracted], State: [not extracted], PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. | 10 | ||
| Warning | Facility validation failed. FacilityName: Garnet Hill Rehab & Skilled Care, FacilityAddress: 1420 McCreary Road, State: , PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. | 9 | ||
| Warning | Facility validation failed. FacilityName: Garnet Hill Rehab & Skilled Care, FacilityAddress: [not extracted], State: [not extracted], PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. | 7 | ||
| Warning | Facility validation failed. FacilityName: , FacilityAddress: [not extracted], State: , PacketId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32, DocumentId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32. | 6 | ||
| Warning | Facility validation failed. FacilityName: HERITAGE PARK OF KATY NURSING AND REHABILITATION, FacilityAddress: [not extracted], State: [not extracted], PacketId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf, DocumentId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf. | 2 | ||
| Warning | Facility validation failed. FacilityName: , FacilityAddress: [not extracted], State: , PacketId: 81ea2e7d-7429-4056-b41d-0efc91c8ea02, DocumentId: 81ea2e7d-7429-4056-b41d-0efc91c8ea02. | 2 | ||
| Warning | Patient not found in target system. Group: 1/1, FirstName: PATRICK, LastName: S***, PacketId: 4f923857-4980-4cf6-82ac-f8255c8a5251, DocumentId: 4f923857-4980-4cf6-82ac-f8255c8a5251, PageIds: [24784f84-d7e1-47aa-873d-25e8a6a243f0, bfc72bc8-e406-407a-b318-80805dcdf429, 23c83109-5597-49ab-8553-513239369eb7]. | 2 | ||
| Warning | Facility validation failed. FacilityName: , FacilityAddress: [not extracted], State: , PacketId: 19ad5cc4-974e-449f-8a68-568556d13343, DocumentId: 19ad5cc4-974e-449f-8a68-568556d13343. | 2 | ||
| Warning | Validation failed. Method: ValidateAsync, Validator: ClassificationValidator, StatusCode: 54, Reason: One or more pages could not be classified., DocumentId: 36430251-e289-4bf0-b6a3-be8dcfc1cb21, TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c | 2 | ||
| Warning | Facility validation failed. FacilityName: HERITAGE PARK OF KATY NURSING AND REHABILITATION, FacilityAddress: [not extracted], State: [not extracted], PacketId: 27a397be-d66c-44ac-8624-be4c9cad542b, DocumentId: 27a397be-d66c-44ac-8624-be4c9cad542b. | 2 | ||
| Warning | Patient not found in target system. Group: 1/1, FirstName: Annette, LastName: K***, PacketId: c843ac34-3cfb-4806-84b8-92c7e2e6fbac, DocumentId: c843ac34-3cfb-4806-84b8-92c7e2e6fbac, PageIds: [1cba47b5-72a2-438d-9726-e7f66dc3a2a7, c2ca1a8c-d6ba-42d5-80af-6bce2c95f10b, 4f894a11-2c8a-43fa-8f76-290b605a4e33, 314499e9-0c6b-40d4-99cc-e7bbb12e13df]. | 2 | ||
| Warning | Patient not found in target system. Group: 1/2, FirstName: JEANLONO, LastName: C***, PacketId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, DocumentId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, PageIds: [c5354f15-1ffd-4907-be1c-e9b19f278dd1, 1b38e4a0-af02-4782-b6e5-4a12fcf394d1, 4814b2b9-4a3e-420a-8d07-6d42c0a87ea6, c67a7a42-60f5-496a-848f-1445b0e5ee37]. | 2 | ||
| Warning | Patient not found in target system. Group: 1/2, FirstName: Frank, LastName: C***, PacketId: 5b3eb979-df06-456f-a2ad-e6de89683cbb, DocumentId: 5b3eb979-df06-456f-a2ad-e6de89683cbb, PageIds: [51734c03-b66c-4e72-93ea-192e4af64ec6]. | 2 | ||
| Warning | Patient not found in target system. Group: 2/2, FirstName: Joanlene, LastName: C***, PacketId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, DocumentId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, PageIds: [dadfc7c4-e529-4e59-b3ea-d72e0b571d60]. | 2 | ||
| Warning | Facility validation failed. FacilityName: Rolling Hills Healthcare Center, FacilityAddress: 3625 St. Joseph Road, State: , PacketId: 36430251-e289-4bf0-b6a3-be8dcfc1cb21, DocumentId: 36430251-e289-4bf0-b6a3-be8dcfc1cb21. | 2 | ||
| Warning | Validation failed. Method: ValidateAsync, Validator: PatientNameValidator, StatusCode: 54, Reason: One or more pages are missing a valid patient name., DocumentId: f8531cd4-df19-409a-98f6-5a586b2412e8, TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c | 1 | ||
| Warning | Patient not found in target system. Group: 3/4, FirstName: FAYE, LastName: I***, PacketId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf, DocumentId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf, PageIds: [ad7ddca9-0a9c-4093-a64a-12eba64060e9, 6b3bd842-b5df-461f-8fb0-7bd52231c3eb]. | 1 | ||
| Warning | Patient not found in target system. Group: 1/1, FirstName: Michael G., LastName: C***, PacketId: 767db6df-7c49-4558-aee6-37ced3338824, DocumentId: 767db6df-7c49-4558-aee6-37ced3338824, PageIds: [22a84300-3945-428f-9ff6-8c628e0d5c69, 411f50cd-cd26-475b-8235-70bae28f88e6, 63a70200-7d48-4f9d-8449-3e9ad6ec2176, 7ad5f7f2-86f8-44c9-989d-0d7a54c486b2]. | 1 |
Swallowed Exceptions
No exceptions in the last 24h.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| Invoke | InProc | 67 | 616ms | 1.2s | 0 | 0.0% |
| cachesync-api.azurewebsites.net | HTTP | 985 | 34ms | 279ms | 0 | 0.0% |
Show 3 internal / host RPC calls
| Name | Calls | Avg | P95 |
|---|---|---|---|
| POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream | 1 | 129ms | 129ms |
| POST /Settlement/Complete | 57 | 62ms | 115ms |
| POST /Settlement/Complete | 10 | 60ms | 179ms |
SUCCESS
Function is performing within normal bounds
Key Findings
- P95 = 0.1s | Average = 0.0s over last 24h
- Slowest dependency: BlobContainerClient.CreateIfNotExists — avg 0.0s, P95 0.0s (3 calls)
- High-failure dependency: Azure blob: referralqstorage/deer-oaks — 50% failure rate (3/6 calls)
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Investigate 50% failure rate on Azure blob: referralqstorage/deer-oaks: check endpoint health, RBAC permissions, payload size, and schema compatibility.
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.0s
Peak P95: 0.1s
at hour 9:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 0.3s | 4523111e… | |||
| 0.1s | 4523111e… | |||
| 0.1s | 4523111e… | |||
| 0.1s | 4523111e… | |||
| 0.0s | 4523111e… | |||
| 0.0s | 4523111e… | |||
| 0.0s | 4523111e… | |||
| 0.0s | 4523111e… | |||
| 0.0s | 4523111e… | |||
| 0.0s | 4523111e… |
All from 1 instance
Warning / Error Traces
No warning or error traces in the last 24h.
Swallowed Exceptions
No exceptions in the last 24h.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| BlobContainerClient.CreateIfNotExists | InProc | Microsoft.Storage | 3 | 20ms | 31ms | 0 | 0.0% |
| BlobContainerClient.GetBlobsByHierarchy | InProc | Microsoft.Storage | 3 | 19ms | 32ms | 0 | 0.0% |
| referralqstorage.blob.core.windows.net | Azure blob | 6 | 13ms | 31ms | 3 | 50.0% |
SUCCESS
Function is performing within normal bounds
Key Findings
- P95 = 0.4s | Average = 0.0s over last 24h
- Slowest dependency: cachesync-api.azurewebsites.net — avg 0.6s, P95 7.0s (68 calls)
- High-failure dependency: PUT devreferralq — 50% failure rate (128/256 calls)
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Investigate 50% failure rate on PUT devreferralq: check endpoint health, RBAC permissions, payload size, and schema compatibility.
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.1s
Peak P95: 0.4s
at hour 21:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 7.7s | wn1mdwk0… | |||
| 7.2s | wn1mdwk0… | |||
| 7.2s | wn1mdwk0… | |||
| 7.1s | wn1mdwk0… | |||
| 7.1s | wn1mdwk0… | |||
| 3.2s | wn1mdwk0… | |||
| 3.1s | wn1mdwk0… | |||
| 1.1s | wn1mdwk0… | |||
| 0.6s | wn1mdwk0… | |||
| 0.5s | wn1mdwk0… |
All from 1 instance
Warning / Error Traces
6
| Sev | Message | Count | Last Seen | |
|---|---|---|---|---|
| Warning | CacheSync attempt 1/3 returned 400. Retrying in 1s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: b1e77eb8-20fd-4819-b17d-e3a5b2f07037 | 1 | ||
| Warning | CacheSync attempt 2/3 returned 400. Retrying in 2s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: b1e77eb8-20fd-4819-b17d-e3a5b2f07037 | 1 | ||
| Warning | Failed to notify document update API after 3 attempts for tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c and document b1e77eb8-20fd-4819-b17d-e3a5b2f07037. Final status: BadRequest | 1 | ||
| Warning | CacheSync attempt 1/3 returned 400. Retrying in 1s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: 5059c0f3-f65a-464a-a01e-54a41ada693f | 1 | ||
| Warning | CacheSync attempt 2/3 returned 400. Retrying in 2s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: 5059c0f3-f65a-464a-a01e-54a41ada693f | 1 | ||
| Warning | Failed to notify document update API after 3 attempts for tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c and document 5059c0f3-f65a-464a-a01e-54a41ada693f. Final status: BadRequest | 1 |
Swallowed Exceptions
No exceptions in the last 24h.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| cachesync-api.azurewebsites.net | HTTP | 68 | 642ms | 7.0s | 6 | 8.8% |
| rq-portal-api.azurewebsites.net | HTTP | 47 | 385ms | 401ms | 0 | 0.0% |
| BlobClient.Upload | InProc | Microsoft.Storage | 128 | 33ms | 131ms | 0 | 0.0% |
| devreferralq.blob.core.windows.net | Azure blob | 256 | 22ms | 95ms | 128 | 50.0% |
| BlobContainerClient.CreateIfNotExists | InProc | Microsoft.Storage | 128 | 12ms | 29ms | 0 | 0.0% |
DANGER
High exception volume (69 in 24h) — top: PostgresException
Key Findings
- P95 = 0.7s | Average = 0.2s over last 24h
- Slowest dependency: devsbreferralq.servicebus.windows.net/extractentitiesqueue — avg 0.1s, P95 0.2s (65 calls)
- 69 swallowed exceptions in 24h — all reported success=true, masking real failures
- All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
- Fix exception swallowing — let failures propagate so success=false is recorded and the Service Bus message is dead-lettered for reprocessing instead of silently dropped.
- Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.2s
Peak P95: 0.7s
at hour 22:00
Top 10 Slowest Executions
| Time | Duration | OK? | Instance (short) | |
|---|---|---|---|---|
| 44.3s | wn1mdwk0… | |||
| 43.8s | wn1mdwk0… | |||
| 43.8s | wn1mdwk0… | |||
| 43.5s | wn1mdwk0… | |||
| 43.5s | wn1mdwk0… | |||
| 42.8s | wn1mdwk0… | |||
| 38.8s | wn1mdwk0… | |||
| 38.3s | wn1mdwk0… | |||
| 36.4s | wn1mdwk0… | |||
| 28.9s | wn1mdwk0… |
All from 1 instance
Warning / Error Traces
1144
| Sev | Message | Count | Last Seen | |
|---|---|---|---|---|
| Warning | Authorization header missing. Path: / | 498 | ||
| Warning | Tenant claim missing on authenticated identity. Path: /, Method: GET | 498 | ||
| Warning | FacilityMasterValidationService: no local or PaceHR match for facility Garnet Hill Rehab & Skilled Care, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. | 52 | ||
| Warning | [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Garnet+Hill+Rehab+%26+Skilled+Care | 35 | ||
| Warning | [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Garnet+Hill+Rehab+%26+Skilled+Care&AddressLine1=1420+McCreary+Road | 13 | ||
| Warning | Authorization header missing. Path: /favicon.ico | 11 | ||
| Warning | [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=HERITAGE+PARK+OF+KATY+NURSING+AND+REHABILITATION | 4 | ||
| Warning | FacilityMasterValidationService: no local or PaceHR match for facility HERITAGE PARK OF KATY NURSING AND REHABILITATION, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. | 4 | ||
| Warning | [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Garnet+Hill+Rehab+%26+Skilled+Care&City=Wylie&AddressLine1=1420+McCreary+Road | 3 | ||
| Warning | TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: DANIEL, LastName: DEMISSIE | 3 | ||
| Warning | FacilityMasterValidationService: no local or PaceHR match for facility Rolling Hills Healthcare Center, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. | 3 | ||
| Warning | [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Rolling+Hills+Healthcare+Center&AddressLine1=3625+St.+Joseph+Road | 3 | ||
| Warning | TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: Frank, LastName: Cabido | 3 | ||
| Warning | TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: Annette, LastName: Korb | 2 | ||
| Warning | TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: JEANLONO, LastName: COLIRON | 2 | ||
| Warning | FacilityMasterValidationService: no local or PaceHR match for facility The Cedars SNF, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. | 2 | ||
| Warning | FacilityMasterValidationService: no local or PaceHR match for facility Fox Hollow Post Acute, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. | 2 | ||
| Warning | TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: Joanlene, LastName: Collron | 2 | ||
| Warning | [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Fox+Hollow+Post+Acute&AddressLine1=310+America+Drive | 2 | ||
| Warning | Slow DB connection acquisition: 6881ms (threshold 5000ms). Pool may be under pressure. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c | 2 |
Swallowed Exceptions
69
Type:
Npgsql.PostgresExceptionMessage: 23502: null value in column "patient_dob" of relation "patient_packets" violates not-null constraint
DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
Last seen:
Root Cause: Database connectivity or query error
Fix: Check Postgres connection string, firewall rules, and query syntax.
Type:
System.TimeoutExceptionMessage: The operation has timed out
Inner: The operation has timed out.
Last seen:
Root Cause: Operation timed out waiting for a resource
Fix: Increase timeout or investigate slow dependencies. Add retry with backoff.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
| Endpoint | Type | Calls | Avg | P95 | Failures | Fail % |
|---|---|---|---|---|---|---|
| pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net | HTTP | 156 | 38ms | 109ms | 3 | 1.9% |
| devsbreferralq.servicebus.windows.net/extractentitiesqueue | Azure Service Bus | 130 | 25ms | 229ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/Extractor | Azure Service Bus | 124 | 19ms | 75ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/enrollmentpacketqueue1 | Azure Service Bus | 12 | 16ms | 46ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/patientvalidatequeue | Azure Service Bus | 146 | 16ms | 62ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/groupingqueue | Azure Service Bus | 126 | 15ms | 39ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/enrollmentpacketqueue | Azure Service Bus | 206 | 13ms | 40ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/pageclassificationqueue | Azure Service Bus | 920 | 12ms | 87ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/missingdocqueue | Azure Service Bus | 1,544 | 7ms | 32ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/OCRQueue | Azure Service Bus | 746 | 6ms | 28ms | 0 | 0.0% |
| devsbreferralq.servicebus.windows.net/ActionDeterminatorQueue | Azure Service Bus | 20 | 6ms | 37ms | 0 | 0.0% |