Loading…

Root Cause Analysis

Queried: Jun 18, 2026 – Jun 20, 2026  |  Generated:
From
To
DANGER Slow external dependency: Invoke averaging 106s per call
Key Findings
  • P95 = 397.6s | Average = 106.8s over last 24h
  • Slowest dependency: Invoke — avg 105.9s, P95 319.6s (61 calls)
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Reduce HttpClient timeout for Invoke and add exponential backoff retry. Consider provisioned capacity (e.g. Azure OpenAI PTU) for consistent latency.
  • P95 exceeds 90s — review function timeout in host.json and HttpClient.Timeout settings. Fail fast and retry rather than waiting for the full timeout.
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 148.2s Peak P95: 397.6s at hour 10:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
397.6s 4523111e…
396.7s 4523111e…
362.0s 4523111e…
355.0s 4523111e…
332.4s 4523111e…
332.4s 4523111e…
319.7s 4523111e…
318.9s 4523111e…
246.8s 4523111e…
246.7s 4523111e…
All from 1 instance
Warning / Error Traces 38
SevMessageCountLast Seen
Warning DocumentNormalization: lone page with no group peers. PatientName: 'Dickerson, Donna S.;Dinges, Dennis R;Edwards, Robyn;Egan, Janet M.;Eggers, Margaret;Ehret, Jason;Falcetta, Dorothy S.;Fletcher, Michael;Forbes, Grace M;Gaddy, William H.;Gill, Jeffrey S;Granger, Daniel J;Gray, Danny;Griggs, Sharee L.;Grubisic, Vukosava;Hall, Anna E.;Hanon, Tabatha J;Hart, Priscilla;Hatcher, Thomas', PageId: 05f85f79-3796-4847-b33b-28855e1f0b08, ClassifiedAs: Multi Orders. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'DEPUY, BETTY', PageId: 79588b2b-a0d9-4448-be3b-40e45ae3806d, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'HALLOCK, TOMMIE', PageId: f6a78a1c-9d80-406d-bdc2-45b69fcfda71, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'PEAL, BYTHELLA', PageId: ba581aa6-30e4-4587-9758-a22453fdebc7, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'BAILEY, MARGARETT', PageId: 10b69990-30a9-496e-ab9f-af28ad05d615, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'Allen, David;Alsultani, Thaer;Anderson, Willard;Ansehl, Donna L;Basile, Barbara A;Besher, Linda M.;Blevins, Robert E.;Bobo, Nicole;Brogdon, Micheal W;Brown, Hae O.;Brown, Karl R;Buhrmester, Barbara;Chancellor, Adolphus;Clymer, Ann F.;Cole, Jocelyn M.;Crabtree, James;Crow, Bluedolphin;Davis, Nancy', PageId: 41b3a33b-2412-4a36-8f51-c4cdd5d221b8, ClassifiedAs: Multi Orders. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'KUKIC, MIRZETA', PageId: fdc9033e-0e95-4d30-a809-92b5f67d4249, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'CROMARTIE, MARGARET', PageId: 98df873c-0f82-42af-bbfa-45d161fa11fe, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'MAI, AN', PageId: 3f2173c8-75d7-45f0-8515-1a2b34866bec, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'Heierman, Brian D.;Helfrich, Susan K;Henson, Jermaine A;Hudson, Brenda D.;Jeske, Steven H;Johnson, Daniel L.;Jones, Betsy M;Jones, Latifah F;Kim, Scarlet;Knapp, Michael D.;Knaust, Betty;Kosulandich, Priscilla;Kranawetter, Allen D;Kupfer, William J;Labriere, Joann;Leachman, Mark;Lee, Rebecca;Lindner, Ronald A;Lindwedel, David R', PageId: 95a7cf9c-24d2-472b-8835-fe942a19dcf2, ClassifiedAs: Multi Orders. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'BARR JR., ARTHUR', PageId: 120a8e0b-0b92-40b3-8433-f3038e15cb92, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'ADAMS, FLORINDA', PageId: c0c72761-6284-4609-8bef-525ded25b78b, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'DEIE, DANIEL', PageId: 4299039f-609d-442d-aebd-df1c41d20829, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'Phung, Phillip;Powell, Clarence F;Ramey, Jacquelyn M;Rensing, Robin;Sampson, Charles L.;Sanyangore, Cleopus P.;Stacy, Robert Charles;Stasik, Dorothy C;Stewart, James D.;Strickland, Louise M.;Szewczuk, Lois A;Velten, Cheryl L;Watkins, Willie;Wells, Patricia M;White, Jordache;Whitfield, Lacarl;Williams, Edward L.;Williams, Marvin;Williams, Samuel A', PageId: 0a5814b2-0f09-4e57-963b-6afece025e5d, ClassifiedAs: Multi Orders. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'GRIBBIN, DELICIA', PageId: 68473c5e-e72d-447a-ae57-8e8086a68158, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'Williams, Susan;Winstead, Mickey T;Woodard Jr, Sherman;Woody, Janis K', PageId: 78b21db7-c44d-40f9-9e68-ef4a29a46979, ClassifiedAs: Multi Orders. Review for possible name mismatch. 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'WILLIAMS, BEVERLY', PageId: 10c031f1-cdb0-44e1-b0d7-bc85125f67ea, ClassifiedAs: Face sheet. Review for possible name mismatch. 2
Warning No DocExtractionConfig found for ClassifiedAs: BlankPage. Falling back to regex. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: 0b7c3aad-cc15-4899-a128-57f0774b5cb9 2
Warning DocumentNormalization: lone page with no group peers. PatientName: 'Gonzales, Maria', PageId: d8c4fc8d-bce2-4e72-bea5-2e32ab0c3bd0, ClassifiedAs: ORDER. Review for possible name mismatch. 1
Warning DocumentNormalization: lone page with no group peers. PatientName: 'CARVAJAL, MANUEL', PageId: c89957a0-8a54-45ab-9803-59a928eefd1b, ClassifiedAs: Face sheet. Review for possible name mismatch. 1
Swallowed Exceptions

No exceptions in the last 24h.

External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
Invoke InProc 61 105.9s 319.6s 0 0.0%
devreferralqopenai.openai.azure.com HTTP 199 65.2s 111.7s 0 0.0%
cachesync-api.azurewebsites.net HTTP 185 124ms 305ms 0 0.0%
BlobBaseClient.DownloadContent InProc | Microsoft.Storage 985 9ms 18ms 0 0.0%
BlobBaseClient.Exists InProc | Microsoft.Storage 1,148 8ms 21ms 0 0.0%
devreferralq.blob.core.windows.net Azure blob 2,133 8ms 20ms 0 0.0%
Show 10 internal / host RPC calls
NameCallsAvgP95
POST /Settlement/Complete 4 4851ms 19229ms
POST /Settlement/RenewSessionLock 7 1579ms 7845ms
POST /Settlement/Complete 1 181ms 181ms
POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream 1 129ms 129ms
POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream 1 119ms 119ms
POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream 1 118ms 118ms
POST /Settlement/Complete 12 68ms 169ms
POST /Settlement/Complete 44 54ms 110ms
POST /Settlement/RenewSessionLock 3 37ms 99ms
POST /Settlement/RenewSessionLock 9 10ms 12ms
DANGER High exception volume (62 in 24h) — top: InvalidOperationException
Key Findings
  • P95 = 45.4s | Average = 1.7s over last 24h
  • Slowest dependency: Invoke — avg 2.0s, P95 3.8s (62 calls)
  • High-failure dependency: PUT devreferralq — 14% failure rate (62/429 calls)
  • 62 swallowed exceptions in 24h — all reported success=true, masking real failures
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Fix exception swallowing — let failures propagate so success=false is recorded and the Service Bus message is dead-lettered for reprocessing instead of silently dropped.
  • Investigate 14% failure rate on PUT devreferralq: check endpoint health, RBAC permissions, payload size, and schema compatibility.
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 5.5s Peak P95: 45.4s at hour 14:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
45.4s 4523111e…
6.0s 4523111e…
4.4s 4523111e…
3.8s 4523111e…
3.2s 4523111e…
2.8s 4523111e…
2.7s 4523111e…
2.1s 4523111e…
2.0s 4523111e…
1.9s 4523111e…
All from 1 instance
Warning / Error Traces

No warning or error traces in the last 24h.

Swallowed Exceptions 62

Type: System.InvalidOperationException
Message: An invalid request URI was provided. Either the request URI must be an absolute URI or BaseAddress must be set.
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
Invoke InProc 62 2.0s 3.8s 0 0.0%
cachesync-api.azurewebsites.net HTTP 186 354ms 285ms 0 0.0%
BlobClient.Upload InProc | Microsoft.Storage 367 50ms 140ms 0 0.0%
devreferralq.blob.core.windows.net Azure blob 491 42ms 138ms 62 12.6%
BlobBaseClient.DownloadTo InProc | Microsoft.Storage 62 33ms 70ms 0 0.0%
BlobContainerClient.CreateIfNotExists InProc | Microsoft.Storage 62 6ms 7ms 0 0.0%
Show 1 internal / host RPC calls
NameCallsAvgP95
POST /Settlement/Complete 62 58ms 114ms
WARNING P95 duration elevated: 51s peak
Key Findings
  • P95 = 50.6s | Average = 8.7s over last 24h
  • Slowest dependency: Invoke — avg 8.9s, P95 16.7s (328 calls)
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 16.8s Peak P95: 50.6s at hour 21:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
51.7s 4523111e…
50.7s 4523111e…
50.6s 4523111e…
50.6s 4523111e…
50.2s 4523111e…
45.5s 4523111e…
44.8s 4523111e…
43.3s 4523111e…
36.4s 4523111e…
27.9s 4523111e…
All from 1 instance
Warning / Error Traces

No warning or error traces in the last 24h.

Swallowed Exceptions

No exceptions in the last 24h.

External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
Invoke InProc 328 8.9s 16.7s 0 0.0%
DocumentAnalysisClient.AnalyzeDocument InProc | Microsoft.CognitiveServices 328 7.0s 7.4s 0 0.0%
AnalyzeDocumentOperation.WaitForCompletion InProc | Microsoft.CognitiveServices 328 6.5s 6.8s 0 0.0%
cachesync-api.azurewebsites.net HTTP 711 732ms 372ms 0 0.0%
doc-intelligence-referralq.cognitiveservices.azure.com HTTP 1,037 237ms 728ms 0 0.0%
AnalyzeDocumentOperation.UpdateStatus InProc | Microsoft.CognitiveServices 709 148ms 322ms 0 0.0%
BlobBaseClient.DownloadTo InProc | Microsoft.Storage 328 24ms 56ms 0 0.0%
devreferralq.blob.core.windows.net Azure blob 1,186 14ms 47ms 0 0.0%
BlobClient.Upload InProc | Microsoft.Storage 858 12ms 22ms 0 0.0%
Show 1 internal / host RPC calls
NameCallsAvgP95
POST /Settlement/Complete 328 67ms 128ms
WARNING P95 duration elevated: 71s peak
Key Findings
  • P95 = 70.5s | Average = 7.5s over last 24h
  • Slowest dependency: Invoke — avg 7.0s, P95 15.9s (462 calls)
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 14.7s Peak P95: 70.5s at hour 18:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
70.5s 4523111e…
35.6s 4523111e…
23.1s 4523111e…
23.1s 4523111e…
23.1s 4523111e…
22.7s 4523111e…
22.2s 4523111e…
21.7s 4523111e…
20.7s 4523111e…
20.2s 4523111e…
All from 1 instance
Warning / Error Traces

No warning or error traces in the last 24h.

Swallowed Exceptions

No exceptions in the last 24h.

External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
Invoke InProc 462 7.0s 15.9s 0 0.0%
fax-receiver-api-dev.azurewebsites.net HTTP 111 685ms 7.1s 0 0.0%
SUCCESS Function is performing within normal bounds
Key Findings
  • P95 = 21.3s | Average = 2.1s over last 24h
  • Slowest dependency: Invoke — avg 1.8s, P95 7.4s (460 calls)
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 4.9s Peak P95: 21.3s at hour 22:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
21.4s 4523111e…
21.3s 4523111e…
20.6s 4523111e…
17.2s 4523111e…
17.2s 4523111e…
17.2s 4523111e…
17.2s 4523111e…
17.1s 4523111e…
17.1s 4523111e…
17.1s 4523111e…
All from 1 instance
Warning / Error Traces

No warning or error traces in the last 24h.

Swallowed Exceptions

No exceptions in the last 24h.

External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
Invoke InProc 460 1.8s 7.4s 0 0.0%
cachesync-api.azurewebsites.net HTTP 1,223 263ms 963ms 0 0.0%
BlobBaseClient.DownloadContent InProc | Microsoft.Storage 448 82ms 431ms 0 0.0%
devreferralq.blob.core.windows.net Azure blob 448 81ms 430ms 0 0.0%
Show 3 internal / host RPC calls
NameCallsAvgP95
POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream 1 119ms 119ms
POST /Settlement/Complete 337 99ms 336ms
POST /Settlement/Complete 123 88ms 193ms
SUCCESS Function is performing within normal bounds
Key Findings
  • P95 = 3.5s | Average = 0.5s over last 24h
  • Slowest dependency: Invoke — avg 0.4s, P95 0.7s (82 calls)
  • High-failure dependency: Invoke — 24% failure rate (20/82 calls)
  • 20 swallowed exceptions in 24h — all reported success=true, masking real failures
  • 21 error-level trace messages logged
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Fix exception swallowing — let failures propagate so success=false is recorded and the Service Bus message is dead-lettered for reprocessing instead of silently dropped.
  • Investigate 24% failure rate on Invoke: check endpoint health, RBAC permissions, payload size, and schema compatibility.
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.7s Peak P95: 3.5s at hour 10:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
3.5s 4523111e…
3.2s 4523111e…
1.0s 4523111e…
0.8s 4523111e…
0.8s 4523111e…
0.7s 4523111e…
0.6s 4523111e…
0.5s 4523111e…
0.5s 4523111e…
0.5s 4523111e…
All from 1 instance
Warning / Error Traces 84
SevMessageCountLast Seen
Error Message processing error (Action=ProcessMessageCallback, EntityPath=groupingqueue, Endpoint=devsbreferralq.servicebus.windows.net) 20
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 5b3eb979-df06-456f-a2ad-e6de89683cbb. Creating a single packet as a possible supplemental document. 20
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 0e688650-4849-4a55-a128-1f46b6496247. Creating a single packet as a possible supplemental document. 14
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 4b68c167-b4ed-4d1c-902a-a76d49f48e5a. Creating a single packet as a possible supplemental document. 5
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 4f923857-4980-4cf6-82ac-f8255c8a5251. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 6 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 2 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 296490f8-4c64-4f1b-ba9b-e50644f463de. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 5 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 4 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId d4a3c62e-f1a6-4b79-b4cd-4be7d8395224. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 3 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected for patient 'Unidentified' on Multi Orders page 1 for DocumentId 4ed8ba4c-21f8-4143-95b6-322ac3806b32. Creating a single packet as a possible supplemental document. 2
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId c843ac34-3cfb-4806-84b8-92c7e2e6fbac. Creating a single packet as a possible supplemental document. 1
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 1a93bd64-7862-43be-a372-75fb453d1014. Creating a single packet as a possible supplemental document. 1
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId acfb2a8a-2b3e-4ae3-bf2e-b1994753ce9b. Creating a single packet as a possible supplemental document. 1
Error API call failed. DocumentId: 5b3eb979-df06-456f-a2ad-e6de89683cbb, StatusCode: InternalServerError, TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, Service: PatientPacketProcessor. Response: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"6c557943b3b9","details":[]} 1
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 0cd139c8-e634-46bc-959b-0c74ad2ee8cf. Creating a single packet as a possible supplemental document. 1
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 946bd030-514f-48f2-8b1a-588166bb48a4. Creating a single packet as a possible supplemental document. 1
Warning DeerOak: No referral option selected on any Referral Order/Order page for DocumentId 688e9037-a9b0-490f-b4b9-02be42a7ce2f. Creating a single packet as a possible supplemental document. 1
Swallowed Exceptions 20

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"6c557943b3b9","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"6ba05767f233","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"d9f9c44234f9","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"de3e81ab1b4c","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"af2f8870e3e5","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"7f84747b2d49","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"617d3c5b6879","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"7842637b9b7d","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"dc191ea84733","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.

Type: System.Exception
Message: Processing failed: Failed to send payload to API: HTTP 500 InternalServerError: {"success":false,"errorCode":"INTERNAL_ERROR","message":"The patient packet details could not be saved right now. Please try again later.","retryable":false,"correlationId":"3acb1ea698cc","details":[]}
Last seen:

Root Cause: Unexpected exception during function execution
Fix: Review the full stack trace in Application Insights for this exception type.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
Invoke InProc 82 355ms 705ms 20 24.4%
cachesync-api.azurewebsites.net HTTP 266 99ms 275ms 20 7.5%
SUCCESS Function is performing within normal bounds
Key Findings
  • P95 = 1.7s | Average = 0.7s over last 24h
  • Slowest dependency: Invoke — avg 0.6s, P95 1.2s (67 calls)
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.9s Peak P95: 1.7s at hour 21:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
1.7s 4523111e…
1.6s 4523111e…
1.5s 4523111e…
1.2s 4523111e…
1.2s 4523111e…
1.1s 4523111e…
1.0s 4523111e…
0.9s 4523111e…
0.9s 4523111e…
0.9s 4523111e…
All from 1 instance
Warning / Error Traces 97
SevMessageCountLast Seen
Warning Facility validation failed. FacilityName: Garnet Hill Rehab & Skilled Care, FacilityAddress: [not extracted], State: , PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. 28
Warning Facility validation failed. FacilityName: [not extracted], FacilityAddress: [not extracted], State: [not extracted], PacketId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32, DocumentId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32. 12
Warning Facility validation failed. FacilityName: [not extracted], FacilityAddress: [not extracted], State: [not extracted], PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. 10
Warning Facility validation failed. FacilityName: Garnet Hill Rehab & Skilled Care, FacilityAddress: 1420 McCreary Road, State: , PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. 9
Warning Facility validation failed. FacilityName: Garnet Hill Rehab & Skilled Care, FacilityAddress: [not extracted], State: [not extracted], PacketId: 0e688650-4849-4a55-a128-1f46b6496247, DocumentId: 0e688650-4849-4a55-a128-1f46b6496247. 7
Warning Facility validation failed. FacilityName: , FacilityAddress: [not extracted], State: , PacketId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32, DocumentId: 4ed8ba4c-21f8-4143-95b6-322ac3806b32. 6
Warning Facility validation failed. FacilityName: HERITAGE PARK OF KATY NURSING AND REHABILITATION, FacilityAddress: [not extracted], State: [not extracted], PacketId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf, DocumentId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf. 2
Warning Facility validation failed. FacilityName: , FacilityAddress: [not extracted], State: , PacketId: 81ea2e7d-7429-4056-b41d-0efc91c8ea02, DocumentId: 81ea2e7d-7429-4056-b41d-0efc91c8ea02. 2
Warning Patient not found in target system. Group: 1/1, FirstName: PATRICK, LastName: S***, PacketId: 4f923857-4980-4cf6-82ac-f8255c8a5251, DocumentId: 4f923857-4980-4cf6-82ac-f8255c8a5251, PageIds: [24784f84-d7e1-47aa-873d-25e8a6a243f0, bfc72bc8-e406-407a-b318-80805dcdf429, 23c83109-5597-49ab-8553-513239369eb7]. 2
Warning Facility validation failed. FacilityName: , FacilityAddress: [not extracted], State: , PacketId: 19ad5cc4-974e-449f-8a68-568556d13343, DocumentId: 19ad5cc4-974e-449f-8a68-568556d13343. 2
Warning Validation failed. Method: ValidateAsync, Validator: ClassificationValidator, StatusCode: 54, Reason: One or more pages could not be classified., DocumentId: 36430251-e289-4bf0-b6a3-be8dcfc1cb21, TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c 2
Warning Facility validation failed. FacilityName: HERITAGE PARK OF KATY NURSING AND REHABILITATION, FacilityAddress: [not extracted], State: [not extracted], PacketId: 27a397be-d66c-44ac-8624-be4c9cad542b, DocumentId: 27a397be-d66c-44ac-8624-be4c9cad542b. 2
Warning Patient not found in target system. Group: 1/1, FirstName: Annette, LastName: K***, PacketId: c843ac34-3cfb-4806-84b8-92c7e2e6fbac, DocumentId: c843ac34-3cfb-4806-84b8-92c7e2e6fbac, PageIds: [1cba47b5-72a2-438d-9726-e7f66dc3a2a7, c2ca1a8c-d6ba-42d5-80af-6bce2c95f10b, 4f894a11-2c8a-43fa-8f76-290b605a4e33, 314499e9-0c6b-40d4-99cc-e7bbb12e13df]. 2
Warning Patient not found in target system. Group: 1/2, FirstName: JEANLONO, LastName: C***, PacketId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, DocumentId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, PageIds: [c5354f15-1ffd-4907-be1c-e9b19f278dd1, 1b38e4a0-af02-4782-b6e5-4a12fcf394d1, 4814b2b9-4a3e-420a-8d07-6d42c0a87ea6, c67a7a42-60f5-496a-848f-1445b0e5ee37]. 2
Warning Patient not found in target system. Group: 1/2, FirstName: Frank, LastName: C***, PacketId: 5b3eb979-df06-456f-a2ad-e6de89683cbb, DocumentId: 5b3eb979-df06-456f-a2ad-e6de89683cbb, PageIds: [51734c03-b66c-4e72-93ea-192e4af64ec6]. 2
Warning Patient not found in target system. Group: 2/2, FirstName: Joanlene, LastName: C***, PacketId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, DocumentId: d4a3c62e-f1a6-4b79-b4cd-4be7d8395224, PageIds: [dadfc7c4-e529-4e59-b3ea-d72e0b571d60]. 2
Warning Facility validation failed. FacilityName: Rolling Hills Healthcare Center, FacilityAddress: 3625 St. Joseph Road, State: , PacketId: 36430251-e289-4bf0-b6a3-be8dcfc1cb21, DocumentId: 36430251-e289-4bf0-b6a3-be8dcfc1cb21. 2
Warning Validation failed. Method: ValidateAsync, Validator: PatientNameValidator, StatusCode: 54, Reason: One or more pages are missing a valid patient name., DocumentId: f8531cd4-df19-409a-98f6-5a586b2412e8, TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c 1
Warning Patient not found in target system. Group: 3/4, FirstName: FAYE, LastName: I***, PacketId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf, DocumentId: 0cd139c8-e634-46bc-959b-0c74ad2ee8cf, PageIds: [ad7ddca9-0a9c-4093-a64a-12eba64060e9, 6b3bd842-b5df-461f-8fb0-7bd52231c3eb]. 1
Warning Patient not found in target system. Group: 1/1, FirstName: Michael G., LastName: C***, PacketId: 767db6df-7c49-4558-aee6-37ced3338824, DocumentId: 767db6df-7c49-4558-aee6-37ced3338824, PageIds: [22a84300-3945-428f-9ff6-8c628e0d5c69, 411f50cd-cd26-475b-8235-70bae28f88e6, 63a70200-7d48-4f9d-8449-3e9ad6ec2176, 7ad5f7f2-86f8-44c9-989d-0d7a54c486b2]. 1
Swallowed Exceptions

No exceptions in the last 24h.

External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
Invoke InProc 67 616ms 1.2s 0 0.0%
cachesync-api.azurewebsites.net HTTP 985 34ms 279ms 0 0.0%
Show 3 internal / host RPC calls
NameCallsAvgP95
POST /AzureFunctionsRpcMessages.FunctionRpc/EventStream 1 129ms 129ms
POST /Settlement/Complete 57 62ms 115ms
POST /Settlement/Complete 10 60ms 179ms
SUCCESS Function is performing within normal bounds
Key Findings
  • P95 = 0.1s | Average = 0.0s over last 24h
  • Slowest dependency: BlobContainerClient.CreateIfNotExists — avg 0.0s, P95 0.0s (3 calls)
  • High-failure dependency: Azure blob: referralqstorage/deer-oaks — 50% failure rate (3/6 calls)
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Investigate 50% failure rate on Azure blob: referralqstorage/deer-oaks: check endpoint health, RBAC permissions, payload size, and schema compatibility.
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.0s Peak P95: 0.1s at hour 9:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
0.3s 4523111e…
0.1s 4523111e…
0.1s 4523111e…
0.1s 4523111e…
0.0s 4523111e…
0.0s 4523111e…
0.0s 4523111e…
0.0s 4523111e…
0.0s 4523111e…
0.0s 4523111e…
All from 1 instance
Warning / Error Traces

No warning or error traces in the last 24h.

Swallowed Exceptions

No exceptions in the last 24h.

External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
BlobContainerClient.CreateIfNotExists InProc | Microsoft.Storage 3 20ms 31ms 0 0.0%
BlobContainerClient.GetBlobsByHierarchy InProc | Microsoft.Storage 3 19ms 32ms 0 0.0%
referralqstorage.blob.core.windows.net Azure blob 6 13ms 31ms 3 50.0%
SUCCESS Function is performing within normal bounds
Key Findings
  • P95 = 0.4s | Average = 0.0s over last 24h
  • Slowest dependency: cachesync-api.azurewebsites.net — avg 0.6s, P95 7.0s (68 calls)
  • High-failure dependency: PUT devreferralq — 50% failure rate (128/256 calls)
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Investigate 50% failure rate on PUT devreferralq: check endpoint health, RBAC permissions, payload size, and schema compatibility.
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.1s Peak P95: 0.4s at hour 21:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
7.7s wn1mdwk0…
7.2s wn1mdwk0…
7.2s wn1mdwk0…
7.1s wn1mdwk0…
7.1s wn1mdwk0…
3.2s wn1mdwk0…
3.1s wn1mdwk0…
1.1s wn1mdwk0…
0.6s wn1mdwk0…
0.5s wn1mdwk0…
All from 1 instance
Warning / Error Traces 6
SevMessageCountLast Seen
Warning CacheSync attempt 1/3 returned 400. Retrying in 1s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: b1e77eb8-20fd-4819-b17d-e3a5b2f07037 1
Warning CacheSync attempt 2/3 returned 400. Retrying in 2s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: b1e77eb8-20fd-4819-b17d-e3a5b2f07037 1
Warning Failed to notify document update API after 3 attempts for tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c and document b1e77eb8-20fd-4819-b17d-e3a5b2f07037. Final status: BadRequest 1
Warning CacheSync attempt 1/3 returned 400. Retrying in 1s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: 5059c0f3-f65a-464a-a01e-54a41ada693f 1
Warning CacheSync attempt 2/3 returned 400. Retrying in 2s. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c, DocumentId: 5059c0f3-f65a-464a-a01e-54a41ada693f 1
Warning Failed to notify document update API after 3 attempts for tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c and document 5059c0f3-f65a-464a-a01e-54a41ada693f. Final status: BadRequest 1
Swallowed Exceptions

No exceptions in the last 24h.

External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
cachesync-api.azurewebsites.net HTTP 68 642ms 7.0s 6 8.8%
rq-portal-api.azurewebsites.net HTTP 47 385ms 401ms 0 0.0%
BlobClient.Upload InProc | Microsoft.Storage 128 33ms 131ms 0 0.0%
devreferralq.blob.core.windows.net Azure blob 256 22ms 95ms 128 50.0%
BlobContainerClient.CreateIfNotExists InProc | Microsoft.Storage 128 12ms 29ms 0 0.0%
DANGER High exception volume (69 in 24h) — top: PostgresException
Key Findings
  • P95 = 0.7s | Average = 0.2s over last 24h
  • Slowest dependency: devsbreferralq.servicebus.windows.net/extractentitiesqueue — avg 0.1s, P95 0.2s (65 calls)
  • 69 swallowed exceptions in 24h — all reported success=true, masking real failures
  • All slow executions originate from a single instance — possible resource leak or memory pressure on that host
Recommendations
  • Fix exception swallowing — let failures propagate so success=false is recorded and the Service Bus message is dead-lettered for reprocessing instead of silently dropped.
  • Restart or recycle the affected function app instance to clear potential memory leaks. Monitor instance count and enable auto-scale if on Consumption plan.
P95 Duration by Hour (last 24h)
Avg P95: 0.2s Peak P95: 0.7s at hour 22:00
Top 10 Slowest Executions
Time Duration OK? Instance (short)
44.3s wn1mdwk0…
43.8s wn1mdwk0…
43.8s wn1mdwk0…
43.5s wn1mdwk0…
43.5s wn1mdwk0…
42.8s wn1mdwk0…
38.8s wn1mdwk0…
38.3s wn1mdwk0…
36.4s wn1mdwk0…
28.9s wn1mdwk0…
All from 1 instance
Warning / Error Traces 1144
SevMessageCountLast Seen
Warning Authorization header missing. Path: / 498
Warning Tenant claim missing on authenticated identity. Path: /, Method: GET 498
Warning FacilityMasterValidationService: no local or PaceHR match for facility Garnet Hill Rehab & Skilled Care, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. 52
Warning [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Garnet+Hill+Rehab+%26+Skilled+Care 35
Warning [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Garnet+Hill+Rehab+%26+Skilled+Care&AddressLine1=1420+McCreary+Road 13
Warning Authorization header missing. Path: /favicon.ico 11
Warning [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=HERITAGE+PARK+OF+KATY+NURSING+AND+REHABILITATION 4
Warning FacilityMasterValidationService: no local or PaceHR match for facility HERITAGE PARK OF KATY NURSING AND REHABILITATION, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. 4
Warning [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Garnet+Hill+Rehab+%26+Skilled+Care&City=Wylie&AddressLine1=1420+McCreary+Road 3
Warning TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: DANIEL, LastName: DEMISSIE 3
Warning FacilityMasterValidationService: no local or PaceHR match for facility Rolling Hills Healthcare Center, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. 3
Warning [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Rolling+Hills+Healthcare+Center&AddressLine1=3625+St.+Joseph+Road 3
Warning TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: Frank, LastName: Cabido 3
Warning TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: Annette, LastName: Korb 2
Warning TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: JEANLONO, LastName: COLIRON 2
Warning FacilityMasterValidationService: no local or PaceHR match for facility The Cedars SNF, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. 2
Warning FacilityMasterValidationService: no local or PaceHR match for facility Fox Hollow Post Acute, tenant 872385ce-ebaa-498e-97f7-5d33c4f2e89c. 2
Warning TenantDeerOaksAdapter.PatientLookup (Master_validate): FacilityUniqueId is required (facility must be validated first). FirstName: Joanlene, LastName: Collron 2
Warning [FacilityLookup] EMPTY RESULT — API returned 200 but 0 records. Url: https://pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net/v1/Facility/Lookup?AccountId=97&FacilityName=Fox+Hollow+Post+Acute&AddressLine1=310+America+Drive 2
Warning Slow DB connection acquisition: 6881ms (threshold 5000ms). Pool may be under pressure. TenantId: 872385ce-ebaa-498e-97f7-5d33c4f2e89c 2
Swallowed Exceptions 69

Type: Npgsql.PostgresException
Message: 23502: null value in column "patient_dob" of relation "patient_packets" violates not-null constraint DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
Last seen:

Root Cause: Database connectivity or query error
Fix: Check Postgres connection string, firewall rules, and query syntax.

Type: System.TimeoutException
Message: The operation has timed out
Inner: The operation has timed out.
Last seen:

Root Cause: Operation timed out waiting for a resource
Fix: Increase timeout or investigate slow dependencies. Add retry with backoff.
External & Dependency Calls — grouped by Endpoint, sorted by Avg Duration
Endpoint Type Calls Avg P95 Failures Fail %
pacehr-d-referralq-api-app-crbcf2fzbkbmbrff.eastus2-01.azurewebsites.net HTTP 156 38ms 109ms 3 1.9%
devsbreferralq.servicebus.windows.net/extractentitiesqueue Azure Service Bus 130 25ms 229ms 0 0.0%
devsbreferralq.servicebus.windows.net/Extractor Azure Service Bus 124 19ms 75ms 0 0.0%
devsbreferralq.servicebus.windows.net/enrollmentpacketqueue1 Azure Service Bus 12 16ms 46ms 0 0.0%
devsbreferralq.servicebus.windows.net/patientvalidatequeue Azure Service Bus 146 16ms 62ms 0 0.0%
devsbreferralq.servicebus.windows.net/groupingqueue Azure Service Bus 126 15ms 39ms 0 0.0%
devsbreferralq.servicebus.windows.net/enrollmentpacketqueue Azure Service Bus 206 13ms 40ms 0 0.0%
devsbreferralq.servicebus.windows.net/pageclassificationqueue Azure Service Bus 920 12ms 87ms 0 0.0%
devsbreferralq.servicebus.windows.net/missingdocqueue Azure Service Bus 1,544 7ms 32ms 0 0.0%
devsbreferralq.servicebus.windows.net/OCRQueue Azure Service Bus 746 6ms 28ms 0 0.0%
devsbreferralq.servicebus.windows.net/ActionDeterminatorQueue Azure Service Bus 20 6ms 37ms 0 0.0%