trouble in data sync

Certified Associate Developer

hi everyone

we just have a trouble in data sync.

we have created some record types from the database, but one of them failed in data sync.

the status is "failed" in sync history.

the detail shows "Alert: Unable to sync due to a system failure. Issue identifier: [e0bcc3cb-4961-443e-be9d-ced7237ae97c]".

is there some proboble reasons for this trouble?

Thank you all.

  Discussion posts and replies are publicly visible

Parents
  • Do you see any additional information in the stdout log during that time? Often there will be a more detailed error message in those logs. In addition, can you share some more information about this such as:

    • What is the source type (database, web service, salesforce)
    • What kind of sync is it (manual sync, scheduled sync, sync from write to data store entity)
    • Is this the first time it failed or has it failed consistently
  • 0
    Certified Associate Developer
    in reply to Peter Lewis

    It is database.

    Record Type Save and Manual Sync and Scheduled sync are all failed. But other records created from other database have been synced successfully.

    We can see the data at data preview which is in the database, but there`s no data in the list.

  • Ok I think I'd still be interested in understanding what's in the stdout log - usually that log can tell you more information about what has occurred. If you haven't used that log before, you can access it by selecting the navigation menu at the top and selecting "System Logs" (assuming you're on cloud) and finding the log called `tomcat-stdout.log.<date of issue>`.

  • 0
    Certified Associate Developer
    in reply to Peter Lewis

    Thanks for your reply.

    There's part of the log. And the full ver is upload as an attachment.

    it seems there's a uncoverable error but I don't understand what it is......

    _________

    2022-06-24 00:03:11,632 [Appian Timer - 1] WARN com.appiancorp.common.ObjectSynchronizer - 1 services were missing users.
    2022-06-24 00:08:54,117 [ajp-nio-0.0.0.0-8009-exec-33304] ERROR com.appiancorp.sail.ValueUiEvaluator - Unable to find bindings for uiSource /sail-application. Using initial bindings...
    2022-06-24 00:10:55,813 [RecordSyncScheduler_Worker-1] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: 122_e271d6fe-ac69-40b2-8605-8697dca9ea6d] fired for job [group: record_load_group, name: 122]
    2022-06-24 00:11:11,395 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Unrecoverable error in record data sync: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f]
    2022-06-24 00:11:11,479 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Error syncing record type with uuid: 6f76c674-d9af-431e-848a-8f649ea54ebe
    com.appiancorp.suiteapi.common.exceptions.AppianException: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f] (APNX-1-4521-016)
    at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45)
    at com.appiancorp.record.datasync.error.RecordDataSyncException.<init>(RecordDataSyncException.java:27)
    at com.appiancorp.record.datasync.error.UnretriableRecordDataSyncException.<init>(UnretriableRecordDataSyncException.java:15)
    at com.appiancorp.record.datasync.error.GenericDataSyncException.<init>(GenericDataSyncException.java:13)
    at com.appiancorp.record.data.recordloaders.ads.RecordAdsExceptionTranslator.translateException(RecordAdsExceptionTranslator.java:35)
    at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.writeRecordsAndTranslateAdsExceptions(SequentialBulkLoadController.java:381)
    at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$null$1(SequentialBulkLoadController.java:324)
    at com.appiancorp.record.data.sourceloaders.BatchRetryHelper.retryLoop(BatchRetryHelper.java:42)
    at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$adsWriteWithRetriesCallable$2(SequentialBulkLoadController.java:323)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
    Caused by: com.appian.data.client.AdsUserInputException: APNX-3-1100-008: Failed to parse request contents, Content-Type: application/transit+json, Body: ["^ ","options",["^ ","store","a","user",3000000000,"username","Administrator","branchId",0,"groups",[0,1,-2,67,23,27]],"data",[["^ ","0d7125f5-fb2b-4b61-bf4a-7c55d97e1fdc",["~#int",6],"443de25c-e5d2-490c-99b1-3345d8fb8bb3",["^8",20],"c93bb2f8-3021-48fd-8b7f-1f1fbce840f4",["^8",60],"3bfd8798-d918-457d-bf90-cafa651f3d33",true,"fa24adf8-4614-485f-9a0a-284bbfb553c9",true,"30a2f35a-cb9e-4c97-adb5-9cc644ab4353",["~#ts",[1655164800000,["^8",0]]],"70a6ff70-41b7-4000-9661-beea3365713b",true,"090237fd-4b98-44d5-9a12-e892f8273bc8",["^8",3],"546752af-a693-42ba-8094-d477edfa9bfc",true,"a2fabddc-a637-4f04-8110-588c849bfd83",false,"ae11253a-942a-4e22-8e81-7a048cc86b50",0,"id",["^ ","d50c2b73-6565-4898-8d05-8c3d2c614f16","~n1","id",null],"c787c435-2954-4bb3-bc25-8ed3d8d228b9",true,"877e30bc-55cf-4676-ac3a-144a3078c641",["^8",10]]]], Error: unsupported tag n

    _________________

    2022-06-24 00:00:00,083 [RecordSyncScheduler_Worker-1] INFO  com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_replica_reaper] fired for job [group: record_load_group, name: record_replica_reaper]
    2022-06-24 00:00:00,122 [RecordSyncScheduler_Worker-1] INFO  com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_replica_reaper] completed for job [group: record_load_group, name: record_replica_reaper]
    2022-06-24 00:00:00,139 [RecordSyncScheduler_Worker-2] INFO  com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_synchronize_ads_join_indices] fired for job [group: record_load_group, name: record_synchronize_ads_join_indices]
    2022-06-24 00:00:00,627 [RecordSyncScheduler_Worker-2] INFO  com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_synchronize_ads_join_indices] completed for job [group: record_load_group, name: record_synchronize_ads_join_indices]
    2022-06-24 00:03:11,632 [Appian Timer - 1] WARN  com.appiancorp.common.ObjectSynchronizer - 1 services were missing users.
    2022-06-24 00:08:54,117 [ajp-nio-0.0.0.0-8009-exec-33304] ERROR com.appiancorp.sail.ValueUiEvaluator - Unable to find bindings for uiSource /sail-application. Using initial bindings...
    2022-06-24 00:10:55,813 [RecordSyncScheduler_Worker-1] INFO  com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: 122_e271d6fe-ac69-40b2-8605-8697dca9ea6d] fired for job [group: record_load_group, name: 122]
    2022-06-24 00:11:11,395 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Unrecoverable error in record data sync: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f]
    2022-06-24 00:11:11,479 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Error syncing record type with uuid: 6f76c674-d9af-431e-848a-8f649ea54ebe
    com.appiancorp.suiteapi.common.exceptions.AppianException: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f] (APNX-1-4521-016)
    	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45)
    	at com.appiancorp.record.datasync.error.RecordDataSyncException.<init>(RecordDataSyncException.java:27)
    	at com.appiancorp.record.datasync.error.UnretriableRecordDataSyncException.<init>(UnretriableRecordDataSyncException.java:15)
    	at com.appiancorp.record.datasync.error.GenericDataSyncException.<init>(GenericDataSyncException.java:13)
    	at com.appiancorp.record.data.recordloaders.ads.RecordAdsExceptionTranslator.translateException(RecordAdsExceptionTranslator.java:35)
    	at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.writeRecordsAndTranslateAdsExceptions(SequentialBulkLoadController.java:381)
    	at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$null$1(SequentialBulkLoadController.java:324)
    	at com.appiancorp.record.data.sourceloaders.BatchRetryHelper.retryLoop(BatchRetryHelper.java:42)
    	at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$adsWriteWithRetriesCallable$2(SequentialBulkLoadController.java:323)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:750)
    Caused by: com.appian.data.client.AdsUserInputException: APNX-3-1100-008: Failed to parse request contents, Content-Type: application/transit+json, Body: ["^ ","options",["^ ","store","a","user",3000000000,"username","Administrator","branchId",0,"groups",[0,1,-2,67,23,27]],"data",[["^ ","0d7125f5-fb2b-4b61-bf4a-7c55d97e1fdc",["~#int",6],"443de25c-e5d2-490c-99b1-3345d8fb8bb3",["^8",20],"c93bb2f8-3021-48fd-8b7f-1f1fbce840f4",["^8",60],"3bfd8798-d918-457d-bf90-cafa651f3d33",true,"fa24adf8-4614-485f-9a0a-284bbfb553c9",true,"30a2f35a-cb9e-4c97-adb5-9cc644ab4353",["~#ts",[1655164800000,["^8",0]]],"70a6ff70-41b7-4000-9661-beea3365713b",true,"090237fd-4b98-44d5-9a12-e892f8273bc8",["^8",3],"546752af-a693-42ba-8094-d477edfa9bfc",true,"a2fabddc-a637-4f04-8110-588c849bfd83",false,"ae11253a-942a-4e22-8e81-7a048cc86b50",0,"id",["^ ","d50c2b73-6565-4898-8d05-8c3d2c614f16","~n1","id",null],"c787c435-2954-4bb3-bc25-8ed3d8d228b9",true,"877e30bc-55cf-4676-ac3a-144a3078c641",["^8",10]]]], Error: unsupported tag n
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    	at com.appian.data.client.AdsExceptionBuilder.build(AdsExceptionBuilder.java:133)
    	at com.appian.data.client.DataClientImpl.buildWriteError(DataClientImpl.java:1021)
    	at com.appian.data.client.DataClientImpl.writeWithRetry(DataClientImpl.java:567)
    	at com.appian.data.client.DataClientImpl.write00(DataClientImpl.java:516)
    	at com.appian.data.client.DataClientImpl.lambda$write0$9(DataClientImpl.java:500)
    	at com.appian.data.client.DataClientImpl.runWithAdditionalGroupsIfNeeded(DataClientImpl.java:296)
    	at com.appian.data.client.DataClientImpl.write0(DataClientImpl.java:499)
    	at com.appian.data.client.DataClientImpl.write(DataClientImpl.java:477)
    	at com.appiancorp.record.data.recordloaders.ads.RecordAdsReplicaBatchWriter.batchWrite0(RecordAdsReplicaBatchWriter.java:80)
    	at com.appiancorp.record.data.recordloaders.ads.RecordAdsReplicaBatchWriter.batchWrite(RecordAdsReplicaBatchWriter.java:60)
    	at com.appiancorp.record.data.recordloaders.ads.RecordAdsReplicaLoadOperationSupport.writeRecords(RecordAdsReplicaLoadOperationSupport.java:107)
    	at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$timeWriteRecords$4(SequentialBulkLoadController.java:399)
    	at com.appiancorp.tracing.SafeTracerImpl.traceDangerousDebug(SafeTracerImpl.java:117)
    	at com.appiancorp.tracing.TracingHelper.traceDangerousDebug(TracingHelper.java:53)
    	at com.appiancorp.record.metrics.RecordReplicaLoadMetricsLogger.time(RecordReplicaLoadMetricsLogger.java:174)
    	at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.timeWriteRecords(SequentialBulkLoadController.java:397)
    	at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.writeRecordsAndTranslateAdsExceptions(SequentialBulkLoadController.java:379)
    	... 7 more
    2022-06-24 00:11:11,493 [RecordSyncScheduler_Worker-1] INFO  com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: 122_e271d6fe-ac69-40b2-8605-8697dca9ea6d] completed for job [group: record_load_group, name: 122]
    2022-06-24 00:11:32,327 [ajp-nio-0.0.0.0-8009-exec-33333] ERROR com.appiancorp.ap2.listings.ListingsServletFilter - Invalid path requested for log listing. Request URI = /suite/shared-logs/pkutechdemo-1/tomcat/ Dispatch URI = /shared-logs/pkutechdemo-1/tomcat/
    2022-06-24 00:11:46,541 [ajp-nio-0.0.0.0-8009-exec-33347] ERROR com.appiancorp.ap2.listings.ListingsServletFilter - Invalid path requested for log listing. Request URI = /suite/shared-logs/pkutechdemo-1/tomcat/ Dispatch URI = /shared-logs/pkutechdemo-1/tomcat/
    2022-06-24 00:11:58,168 [ajp-nio-0.0.0.0-8009-exec-33332] ERROR com.appiancorp.ap2.listings.ListingsServletFilter - Invalid path requested for log listing. Request URI = /suite/shared-logs/pkutechdemo-1/tomcat/ Dispatch URI = /shared-logs/pkutechdemo-1/tomcat/

  • Ok nothing seems obvious to me, and I haven't see that "Failed to parse request contents error" before. I'd recommend opening a support ticket with Appian to see if they can help you investigate.

Reply Children
  • 0
    Certified Associate Developer
    in reply to Peter Lewis

    OK thanks for you advice

  • 0
    Certified Senior Developer
    in reply to Peter Lewis

    I am facing similar kind of an issue where the record is sync is failing randomly. We are using record sync smart service in a process model. NOTE: when performing manual sync or scheduled sync then it never fails. It is only failing when using smart service and that too randomly. Few times it is success but few times it fails. There is no defined pattern at times it fails around 300s and at ties it fails within 26s.

    We tried checking the stdout logs but there is no clear error message. we have identified few different kind of errors in the logs as below;

    Can you please provide any insight/help of what these errors are?

  • 0
    Certified Lead Developer
    in reply to sindhujaj0001

    Also mentioned here:

    v21.4 Cloud I am getting the below error when updating data of a synced record using a process that has Write to Data Store Entity nodes and Write to Multiple Data Store Entities nodes. [Appian Work…
    11 replies

    Hi all, While Data Sync Enable, Doing related action For Update query in the table by using the Write to data store smart service. I am getting data sync error as below screenshot. I have generated…
    By in Discussions > Data and Records
    20 replies

    Seems like the issue was;

    "In my case, the underlying database table's PK was configured as INT UNSIGNED. When I updated it to be simply INT, the data sync errors on DSE writes went away."

    Did you take a look at the type of the primary key in the database?

    Otherwise, probably a good idea to log a support case with Appian.

  • 0
    Certified Senior Developer
    in reply to Mathieu Drouin

    the source table do not have any integer field. it is all text. Even the primary key is a text value (composite key). The sync is failing when using smart service and that too not always. But it is working in Dev and test env. It is not working in prod env only.

    Can there be some other issue that can cause this error randomly in PROD only.
    again, Manual and scheduled sync is working fine only smart service sync is failing randomly. In dev and test all three types of sync is working absolutely fine

  • 0
    Certified Lead Developer
    in reply to sindhujaj0001

    Are you able to check the type returned by the web service for your key?

    I did have a weird problem once where I had 2 different types returned by the API which was causing an issue.

  • 0
    Certified Senior Developer
    in reply to Mathieu Drouin

    I don't think this is the problem of type because the error is coming randomly. At times through smart service it works successfully and at time it fails and also it doesn't fail in dev or test at all. It is only failing in PROD.

  • 0
    Certified Lead Developer
    in reply to sindhujaj0001

    Guess what I am getting at is that it is most probably a data issue.

    If you are using the smart service, it means you are doing a sync for specific ids.

    I would assume that you don't have the same exact data in all your environments which would explain why it works in certain environments and not others.

  • 0
    Certified Senior Developer
    in reply to Mathieu Drouin

    Just wanted to add one more info, we have done SAP integration through which we are sourcing data in the record.