hi everyone
we just have a trouble in data sync.
we have created some record types from the database, but one of them failed in data sync.
the status is "failed" in sync history.
the detail shows "Alert: Unable to sync due to a system failure. Issue identifier: [e0bcc3cb-4961-443e-be9d-ced7237ae97c]".
is there some proboble reasons for this trouble?
Thank you all.
Discussion posts and replies are publicly visible
Do you see any additional information in the stdout log during that time? Often there will be a more detailed error message in those logs. In addition, can you share some more information about this such as:
It is database.
Record Type Save and Manual Sync and Scheduled sync are all failed. But other records created from other database have been synced successfully.
We can see the data at data preview which is in the database, but there`s no data in the list.
Ok I think I'd still be interested in understanding what's in the stdout log - usually that log can tell you more information about what has occurred. If you haven't used that log before, you can access it by selecting the navigation menu at the top and selecting "System Logs" (assuming you're on cloud) and finding the log called `tomcat-stdout.log.<date of issue>`.
Thanks for your reply.
There's part of the log. And the full ver is upload as an attachment.
it seems there's a uncoverable error but I don't understand what it is......
_________
2022-06-24 00:03:11,632 [Appian Timer - 1] WARN com.appiancorp.common.ObjectSynchronizer - 1 services were missing users.2022-06-24 00:08:54,117 [ajp-nio-0.0.0.0-8009-exec-33304] ERROR com.appiancorp.sail.ValueUiEvaluator - Unable to find bindings for uiSource /sail-application. Using initial bindings...2022-06-24 00:10:55,813 [RecordSyncScheduler_Worker-1] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: 122_e271d6fe-ac69-40b2-8605-8697dca9ea6d] fired for job [group: record_load_group, name: 122]2022-06-24 00:11:11,395 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Unrecoverable error in record data sync: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f]2022-06-24 00:11:11,479 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Error syncing record type with uuid: 6f76c674-d9af-431e-848a-8f649ea54ebecom.appiancorp.suiteapi.common.exceptions.AppianException: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f] (APNX-1-4521-016) at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) at com.appiancorp.record.datasync.error.RecordDataSyncException.<init>(RecordDataSyncException.java:27) at com.appiancorp.record.datasync.error.UnretriableRecordDataSyncException.<init>(UnretriableRecordDataSyncException.java:15) at com.appiancorp.record.datasync.error.GenericDataSyncException.<init>(GenericDataSyncException.java:13) at com.appiancorp.record.data.recordloaders.ads.RecordAdsExceptionTranslator.translateException(RecordAdsExceptionTranslator.java:35) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.writeRecordsAndTranslateAdsExceptions(SequentialBulkLoadController.java:381) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$null$1(SequentialBulkLoadController.java:324) at com.appiancorp.record.data.sourceloaders.BatchRetryHelper.retryLoop(BatchRetryHelper.java:42) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$adsWriteWithRetriesCallable$2(SequentialBulkLoadController.java:323) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750)Caused by: com.appian.data.client.AdsUserInputException: APNX-3-1100-008: Failed to parse request contents, Content-Type: application/transit+json, Body: ["^ ","options",["^ ","store","a","user",3000000000,"username","Administrator","branchId",0,"groups",[0,1,-2,67,23,27]],"data",[["^ ","0d7125f5-fb2b-4b61-bf4a-7c55d97e1fdc",["~#int",6],"443de25c-e5d2-490c-99b1-3345d8fb8bb3",["^8",20],"c93bb2f8-3021-48fd-8b7f-1f1fbce840f4",["^8",60],"3bfd8798-d918-457d-bf90-cafa651f3d33",true,"fa24adf8-4614-485f-9a0a-284bbfb553c9",true,"30a2f35a-cb9e-4c97-adb5-9cc644ab4353",["~#ts",[1655164800000,["^8",0]]],"70a6ff70-41b7-4000-9661-beea3365713b",true,"090237fd-4b98-44d5-9a12-e892f8273bc8",["^8",3],"546752af-a693-42ba-8094-d477edfa9bfc",true,"a2fabddc-a637-4f04-8110-588c849bfd83",false,"ae11253a-942a-4e22-8e81-7a048cc86b50",0,"id",["^ ","d50c2b73-6565-4898-8d05-8c3d2c614f16","~n1","id",null],"c787c435-2954-4bb3-bc25-8ed3d8d228b9",true,"877e30bc-55cf-4676-ac3a-144a3078c641",["^8",10]]]], Error: unsupported tag n
_________________
2022-06-24 00:00:00,083 [RecordSyncScheduler_Worker-1] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_replica_reaper] fired for job [group: record_load_group, name: record_replica_reaper] 2022-06-24 00:00:00,122 [RecordSyncScheduler_Worker-1] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_replica_reaper] completed for job [group: record_load_group, name: record_replica_reaper] 2022-06-24 00:00:00,139 [RecordSyncScheduler_Worker-2] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_synchronize_ads_join_indices] fired for job [group: record_load_group, name: record_synchronize_ads_join_indices] 2022-06-24 00:00:00,627 [RecordSyncScheduler_Worker-2] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: record_synchronize_ads_join_indices] completed for job [group: record_load_group, name: record_synchronize_ads_join_indices] 2022-06-24 00:03:11,632 [Appian Timer - 1] WARN com.appiancorp.common.ObjectSynchronizer - 1 services were missing users. 2022-06-24 00:08:54,117 [ajp-nio-0.0.0.0-8009-exec-33304] ERROR com.appiancorp.sail.ValueUiEvaluator - Unable to find bindings for uiSource /sail-application. Using initial bindings... 2022-06-24 00:10:55,813 [RecordSyncScheduler_Worker-1] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: 122_e271d6fe-ac69-40b2-8605-8697dca9ea6d] fired for job [group: record_load_group, name: 122] 2022-06-24 00:11:11,395 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Unrecoverable error in record data sync: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f] 2022-06-24 00:11:11,479 [RecordSyncScheduler_Worker-1] ERROR com.appiancorp.record.service.BulkLoadServiceImpl - Error syncing record type with uuid: 6f76c674-d9af-431e-848a-8f649ea54ebe com.appiancorp.suiteapi.common.exceptions.AppianException: Unable to sync due to a system failure. Issue identifier: [ee3aa307-15e0-45f4-bf51-36d8c27da87f] (APNX-1-4521-016) at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) at com.appiancorp.record.datasync.error.RecordDataSyncException.<init>(RecordDataSyncException.java:27) at com.appiancorp.record.datasync.error.UnretriableRecordDataSyncException.<init>(UnretriableRecordDataSyncException.java:15) at com.appiancorp.record.datasync.error.GenericDataSyncException.<init>(GenericDataSyncException.java:13) at com.appiancorp.record.data.recordloaders.ads.RecordAdsExceptionTranslator.translateException(RecordAdsExceptionTranslator.java:35) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.writeRecordsAndTranslateAdsExceptions(SequentialBulkLoadController.java:381) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$null$1(SequentialBulkLoadController.java:324) at com.appiancorp.record.data.sourceloaders.BatchRetryHelper.retryLoop(BatchRetryHelper.java:42) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$adsWriteWithRetriesCallable$2(SequentialBulkLoadController.java:323) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.appian.data.client.AdsUserInputException: APNX-3-1100-008: Failed to parse request contents, Content-Type: application/transit+json, Body: ["^ ","options",["^ ","store","a","user",3000000000,"username","Administrator","branchId",0,"groups",[0,1,-2,67,23,27]],"data",[["^ ","0d7125f5-fb2b-4b61-bf4a-7c55d97e1fdc",["~#int",6],"443de25c-e5d2-490c-99b1-3345d8fb8bb3",["^8",20],"c93bb2f8-3021-48fd-8b7f-1f1fbce840f4",["^8",60],"3bfd8798-d918-457d-bf90-cafa651f3d33",true,"fa24adf8-4614-485f-9a0a-284bbfb553c9",true,"30a2f35a-cb9e-4c97-adb5-9cc644ab4353",["~#ts",[1655164800000,["^8",0]]],"70a6ff70-41b7-4000-9661-beea3365713b",true,"090237fd-4b98-44d5-9a12-e892f8273bc8",["^8",3],"546752af-a693-42ba-8094-d477edfa9bfc",true,"a2fabddc-a637-4f04-8110-588c849bfd83",false,"ae11253a-942a-4e22-8e81-7a048cc86b50",0,"id",["^ ","d50c2b73-6565-4898-8d05-8c3d2c614f16","~n1","id",null],"c787c435-2954-4bb3-bc25-8ed3d8d228b9",true,"877e30bc-55cf-4676-ac3a-144a3078c641",["^8",10]]]], Error: unsupported tag n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.appian.data.client.AdsExceptionBuilder.build(AdsExceptionBuilder.java:133) at com.appian.data.client.DataClientImpl.buildWriteError(DataClientImpl.java:1021) at com.appian.data.client.DataClientImpl.writeWithRetry(DataClientImpl.java:567) at com.appian.data.client.DataClientImpl.write00(DataClientImpl.java:516) at com.appian.data.client.DataClientImpl.lambda$write0$9(DataClientImpl.java:500) at com.appian.data.client.DataClientImpl.runWithAdditionalGroupsIfNeeded(DataClientImpl.java:296) at com.appian.data.client.DataClientImpl.write0(DataClientImpl.java:499) at com.appian.data.client.DataClientImpl.write(DataClientImpl.java:477) at com.appiancorp.record.data.recordloaders.ads.RecordAdsReplicaBatchWriter.batchWrite0(RecordAdsReplicaBatchWriter.java:80) at com.appiancorp.record.data.recordloaders.ads.RecordAdsReplicaBatchWriter.batchWrite(RecordAdsReplicaBatchWriter.java:60) at com.appiancorp.record.data.recordloaders.ads.RecordAdsReplicaLoadOperationSupport.writeRecords(RecordAdsReplicaLoadOperationSupport.java:107) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.lambda$timeWriteRecords$4(SequentialBulkLoadController.java:399) at com.appiancorp.tracing.SafeTracerImpl.traceDangerousDebug(SafeTracerImpl.java:117) at com.appiancorp.tracing.TracingHelper.traceDangerousDebug(TracingHelper.java:53) at com.appiancorp.record.metrics.RecordReplicaLoadMetricsLogger.time(RecordReplicaLoadMetricsLogger.java:174) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.timeWriteRecords(SequentialBulkLoadController.java:397) at com.appiancorp.record.data.sourceloaders.SequentialBulkLoadController.writeRecordsAndTranslateAdsExceptions(SequentialBulkLoadController.java:379) ... 7 more 2022-06-24 00:11:11,493 [RecordSyncScheduler_Worker-1] INFO com.appiancorp.record.service.quartz.LoggingTriggerListener - Trigger [group: record_load_group, name: 122_e271d6fe-ac69-40b2-8605-8697dca9ea6d] completed for job [group: record_load_group, name: 122] 2022-06-24 00:11:32,327 [ajp-nio-0.0.0.0-8009-exec-33333] ERROR com.appiancorp.ap2.listings.ListingsServletFilter - Invalid path requested for log listing. Request URI = /suite/shared-logs/pkutechdemo-1/tomcat/ Dispatch URI = /shared-logs/pkutechdemo-1/tomcat/ 2022-06-24 00:11:46,541 [ajp-nio-0.0.0.0-8009-exec-33347] ERROR com.appiancorp.ap2.listings.ListingsServletFilter - Invalid path requested for log listing. Request URI = /suite/shared-logs/pkutechdemo-1/tomcat/ Dispatch URI = /shared-logs/pkutechdemo-1/tomcat/ 2022-06-24 00:11:58,168 [ajp-nio-0.0.0.0-8009-exec-33332] ERROR com.appiancorp.ap2.listings.ListingsServletFilter - Invalid path requested for log listing. Request URI = /suite/shared-logs/pkutechdemo-1/tomcat/ Dispatch URI = /shared-logs/pkutechdemo-1/tomcat/
Ok nothing seems obvious to me, and I haven't see that "Failed to parse request contents error" before. I'd recommend opening a support ticket with Appian to see if they can help you investigate.
OK thanks for you advice
Peter Lewis I am facing similar kind of an issue where the record is sync is failing randomly. We are using record sync smart service in a process model. NOTE: when performing manual sync or scheduled sync then it never fails. It is only failing when using smart service and that too randomly. Few times it is success but few times it fails. There is no defined pattern at times it fails around 300s and at ties it fails within 26s.
We tried checking the stdout logs but there is no clear error message. we have identified few different kind of errors in the logs as below;
Can you please provide any insight/help of what these errors are?
Also mentioned here:
Seems like the issue was;
"In my case, the underlying database table's PK was configured as INT UNSIGNED. When I updated it to be simply INT, the data sync errors on DSE writes went away."
Did you take a look at the type of the primary key in the database?
Otherwise, probably a good idea to log a support case with Appian.
Mathieu Drouin the source table do not have any integer field. it is all text. Even the primary key is a text value (composite key). The sync is failing when using smart service and that too not always. But it is working in Dev and test env. It is not working in prod env only.
Can there be some other issue that can cause this error randomly in PROD only.again, Manual and scheduled sync is working fine only smart service sync is failing randomly. In dev and test all three types of sync is working absolutely fine
Are you able to check the type returned by the web service for your key?
I did have a weird problem once where I had 2 different types returned by the API which was causing an issue.
I don't think this is the problem of type because the error is coming randomly. At times through smart service it works successfully and at time it fails and also it doesn't fail in dev or test at all. It is only failing in PROD.
Guess what I am getting at is that it is most probably a data issue.
If you are using the smart service, it means you are doing a sync for specific ids.
I would assume that you don't have the same exact data in all your environments which would explain why it works in certain environments and not others.