KB-2038 Issues writing to MySQL database when emojis or unicode characters are used

Parmida Borhani — Tue, 10 Dec 2019 02:03:41 GMT

Current Revision posted to Appian Knowledge Base by Parmida Borhani on 12/10/2019 2:03:41 AM

Symptoms

When attempting to write text containing emojis and certain other unicode characters to the MySQL business database from a Write to Data Store Entity smart service, the following error is logged:

ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: '<EXAMPLE_STRING>' for column '<EXAMPLE_COLUMN>' at row 1
ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

Work with a DBA to update the default collation on the database and on existing tables and columns. The following steps show one possible solution to this for MySQL databases:

Take a backup of the database, including all database configurations and settings.
Check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands:
```
select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';
```
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will now inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

To identify which tables have columns that are using a character set, run the following query:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = '<schema_name>'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. Make sure that the intended character set change will not cause new problems.

The following is an example of an incompatibility that could result from changing the character set. The character set latin1 only takes one byte per character. The maximum index key length in MySQL is 767 bytes. So a column with a latin1 character set can have an index as long as the column limits to less than 767 characters. But utf8mb4 can take up to 4 bytes per character, which means that an index can only be set on a field that is a maximum of 767/4=191 characters. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three possible options are to:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index, change the character set, then recreate the index.
2. Keep the column length as-is, but recreate the index with a maximum key length of 191 characters only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For more information, see the MySQL documentation for column character sets and collation.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: database, emoji, mysql, application design, collation, infrastructure, unicode

DRAFT - KB-XXXX - Issues writing to MySQL database when emojis or unicode characters are used

Parmida Borhani — Mon, 09 Dec 2019 05:56:59 GMT

Revision 20 posted to Appian Knowledge Base by Parmida Borhani on 12/9/2019 5:56:59 AM

Symptoms

When attempting to write text containing emojis and certain other unicode characters to the MySQL business database from a Write to Data Store Entity smart service, the following error is logged:

ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: '<EXAMPLE_STRING>' for column '<EXAMPLE_COLUMN>' at row 1
ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

Work with a DBA to update the default collation on the database and on existing tables and columns. The following steps show one possible solution to this for MySQL databases:

Take a backup of the database, including all database configurations and settings.
Check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands:
```
select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';
```
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will now inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

To identify which tables have columns that are using a character set, run the following query:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = '<schema_name>'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. Make sure that the intended character set change will not cause new problems.

The following is an example of an incompatibility that could result from changing the character set. The character set latin1 only takes one byte per character. The maximum index key length in MySQL is 767 bytes. So a column with a latin1 character set can have an index as long as the column limits to less than 767 characters. But utf8mb4 can take up to 4 bytes per character, which means that an index can only be set on a field that is a maximum of 767/4=191 characters. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three possible options are to:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index, change the character set, then recreate the index.
2. Keep the column length as-is, but recreate the index with a maximum key length of 191 characters only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For more information, see the MySQL documentation for column character sets and collation.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: database, emoji, mysql, collation, Cloud

DRAFT - KB-XXXX - Issues writing to MySQL database when emojis or unicode characters are used

Parmida Borhani — Mon, 09 Dec 2019 05:56:08 GMT

Revision 19 posted to Appian Knowledge Base by Parmida Borhani on 12/9/2019 5:56:08 AM

Symptoms

When attempting to write text containing emojis and certain other unicode characters to the MySQL business database from a Write to Data Store Entity smart service, the following error is logged:

ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

Work with a DBA to update the default collation on the database and on existing tables and columns. The following steps show one possible solution to this for MySQL databases:

Take a backup of the database, including all database configurations and settings.
Check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands:
```
select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';
```
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will now inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

To identify which tables have columns that are using a character set, run the following query:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = '<schema_name>'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. Make sure that the intended character set change will not cause new problems.

The following is an example of an incompatibility that could result from changing the character set. The character set latin1 only takes one byte per character. The maximum index key length in MySQL is 767 bytes. So a column with a latin1 character set can have an index as long as the column limits to less than 767 characters. But utf8mb4 can take up to 4 bytes per character, which means that an index can only be set on a field that is a maximum of 767/4=191 characters. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three possible options are to:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index, change the character set, then recreate the index.
2. Keep the column length as-is, but recreate the index with a maximum key length of 191 characters only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For more information, see the MySQL documentation for column character sets and collation.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: database, emoji, mysql, collation, Cloud

DRAFT - KB-XXXX - Update MySQL collation to support emojis

Tom Ryan — Fri, 06 Dec 2019 02:33:04 GMT

Revision 18 posted to Appian Knowledge Base by Tom Ryan on 12/6/2019 2:33:04 AM

Symptoms

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

Work with your DBA to update the default collation on the database and on existing tables and columns. The following steps show one possible solution to this for MySQL databases:

Take a backup of the database, including all database configurations and settings.
Check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands:
```
select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';
```
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will now inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

To identify which tables have columns that are using a character set, run the following query:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = '<schema_name>'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. Make sure that the intended character set change will not cause new problems.

The following is an example of an incompatibility that could result from changing the character set. The character set latin1 only takes one byte per character. The maximum index key length in MySQL is 767 bytes. So a column with a latin1 character set can have an index as long as the column limits to less than 767 characters. But utf8mb4 can take up to 4 bytes per character, which means that an index can only be set on a field that is a maximum of 767/4=191 characters. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three possible options are to:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
2. Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For more information, see the MySQL documentation for column character sets and collation.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: database, mysql, Cloud

DRAFT - KB-XXXX - Update MySQL collation to support emojis

Michael Haines — Thu, 05 Dec 2019 19:54:51 GMT

Revision 17 posted to Appian Knowledge Base by Michael Haines on 12/5/2019 7:54:51 PM

Symptoms

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

Work with your DBA to update the default collation on the database and on existing tables and columns:

Take a backup of the database
First check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands:
```
select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';
```
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Once this is confirmed, take a snapshot of the current settings before applying new changes in case the current settings need to restored later on.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

Run this query to help identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. Make sure that the intended character set change will not cause new problems.

The following is an example of an incompatibility that could result from changing the character set. The character set latin1 only takes one byte per character. The maximum index key length in mysql is 767 bytes. So a column with a latin1 character set can have an index as long as the column limits to less than 767 characters. But utf8mb4 can take up to 4 bytes per character, which means that an index can only be set on a field that is a maximum of 767/4=191 characters. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three possible options are to:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
2. Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For more information, see the MySQL documentation for column character sets and collation.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: database, mysql, Cloud

DRAFT - KB-XXXX - Update MySQL collation to support emojis

Parmida Borhani — Thu, 31 Oct 2019 00:13:16 GMT

Revision 16 posted to Appian Knowledge Base by Parmida Borhani on 10/31/2019 12:13:16 AM

Symptoms

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

First check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands from phpMyAdmin:select * from performance_schema.global_variables where variable_name like '%character%' or variable_name like '%collation%';
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Once this is confirmed, take a snapshot of the current settings before applying new changes in case the current settings need to restored later on.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

The remaining steps should be performed by the customer, not by Support. Run this query to help identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. For example, latin1 characters only take one byte per character. The maximum index key length in mysql is 767 bytes. So they can have an index on a character field that is 767 bytes. But utf8mb4 can take up to 4 bytes per character. Which means that an index can only be set on a varchar field that is 767/4=191 characters only. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three options could be:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
2. Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - KB-XXXX - Update MySQL collation to support emojis

Parmida Borhani — Thu, 31 Oct 2019 00:12:09 GMT

Revision 15 posted to Appian Knowledge Base by Parmida Borhani on 10/31/2019 12:12:09 AM

Symptoms

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

First check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands from phpMyAdmin:
```
select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';
```
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Once this is confirmed, take a snapshot of the current settings before applying new changes in case the current settings need to restored later on.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

The remaining steps should be performed by the customer, not by Support. Run this query to help identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. For example, latin1 characters only take one byte per character. The maximum index key length in mysql is 767 bytes. So they can have an index on a character field that is 767 bytes. But utf8mb4 can take up to 4 bytes per character. Which means that an index can only be set on a varchar field that is 767/4=191 characters only. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three options could be:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
2. Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - KB-XXXX - Update MySQL collation to support emojis

Michael Haines — Wed, 30 Oct 2019 22:38:57 GMT

Revision 14 posted to Appian Knowledge Base by Michael Haines on 10/30/2019 10:38:57 PM

Symptoms

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

First check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands from phpMyAdmin:
```
select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';
```
Confirm that the default collation should in fact be changed, and that changing it will not break any intended functionality.
Once this is confirmed, take a snapshot of the current settings before applying new changes in case the current settings need to restored later on.
Run the following commands to manually set the default server level character set and collation:
```
set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';
```
Any new tables created will inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one.

The remaining steps should be performed by the customer, not by Support. Run this query to help identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

When changing a character set, it is possible to have incompatibilities between the old and the new set. For example, latin1 characters only take one byte per character. The maximum index key length in mysql is 767 bytes. So they can have an index on a character field that is 767 bytes. But utf8mb4 can take up to 4 bytes per character. Which means that an index can only be set on a varchar field that is 767/4=191 characters only. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. To work around this, three options could be:
1. Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
2. Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.
3. Do nothing about the problem if changing the collation is more costly than enabling storage of emojis.
For each <table_name> that should have the collation changed, run:
```
alter table <table_name> convert to character set utf8mb4;
```

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - KB-XXXX - How to update MySQL collation to support emojis

Michael Haines — Mon, 30 Sep 2019 13:31:56 GMT

Revision 13 posted to Appian Knowledge Base by Michael Haines on 9/30/2019 1:31:56 PM

Symptom

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

First check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands from phpMyAdmin:

select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';

Confirm that you would like to change the default collation, and that it changing it will not break any intended functionality. Once you have confirmed, take a snapshot of the current settings before applying new changes in case you need to restore them later on. Next, run the following commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

Any new tables created will inherit the default character set and collation. However, any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one. The customer will need to do that explicitly themselves. For each <table_name>, the customer should run:

alter table <table_name> convert to character set utf8mb4;

It is possible that there might be an incompatibility between two old and new character sets. In this case, you will need to decide for yourself what is the appropriate course of action. For example, latin1 characters only take one byte per character. The maximum index key length in mysql is 767 bytes. So they can have an index on a character field that is 767 bytes. But utf8mb4 can take up to 4 bytes per character. Which means that you can have only have an index on a varchar field that is 767/4=191 characters only. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. Two alternatives could be:

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, you could choose to do nothing about the problem if changing the collation is more costly than enabling storage of emojis. Run this query to help identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Mon, 30 Sep 2019 13:31:20 GMT

Revision 12 posted to Appian Knowledge Base by Michael Haines on 9/30/2019 1:31:20 PM

Symptom

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

First check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands from phpMyAdmin:

select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Mon, 30 Sep 2019 13:30:04 GMT

Revision 11 posted to Appian Knowledge Base by Michael Haines on 9/30/2019 1:30:04 PM

Symptom

When attempting to write text containing emojis and certain other Unicode characters to the MySQL business database from a Write to Data Store node, the following error is logged:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

First check the current default character set at different levels as well as the current default collation setting for the MySQL database by running the following commands from phpMyAdmin:

select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, you could choose to do nothing about the problem, that it is more important to retain status quo than to support storing emojis. Run this query to help identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Fri, 27 Sep 2019 20:37:04 GMT

Revision 10 posted to Appian Knowledge Base by Michael Haines on 9/27/2019 8:37:04 PM

Symptom

When attempting to write text containing emojis and certain other Unicode characters to the customer's MySQL business database from a Write to Data Store node, the customer encounters the following errors:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

On the customer's MySQL database, first check the current default character set at different levels as well as the current default collation setting by running the following commands:

select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';

If the default collation has already been set intentionally by the customer, then follow up with them to make sure that they would actually like the collation to be changed. Once you have confirmed, take a snapshot of the current settings before applying new changes in case you need to restore them later on. Next, run the following commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

They need to do this themselves because something may not be compatible between two character sets. In this case, they will need to decide for themselves what is the appropriate course of action. For example, latin1 characters only take one byte per character. The maximum index key length in mysql is 767 bytes. So they can have an index on a character field that is 767 bytes. But utf8mb4 can take up to 4 bytes per character. Which means that you can have only have an index on a varchar field that is 767/4=191 characters only. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. Two alternatives could be:

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, the customer could choose to do nothing about the problem. This query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Fri, 27 Sep 2019 20:36:19 GMT

Revision 9 posted to Appian Knowledge Base by Michael Haines on 9/27/2019 8:36:19 PM

Symptom

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

On the customer's MySQL database, first check the current default character set at different levels as well as the current default collation setting by running the following commands:

select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, the customer could choose to do nothing about the problem. This query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Fri, 27 Sep 2019 20:35:58 GMT

Revision 8 posted to Appian Knowledge Base by Michael Haines on 9/27/2019 8:35:58 PM

Symptom

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

On the customer's MySQL database, first check the current default character set at different levels as well as the current default collation setting by running the following commands:

select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, the customer could choose to do nothing about the problem. This query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Fri, 27 Sep 2019 20:35:12 GMT

Revision 7 posted to Appian Knowledge Base by Michael Haines on 9/27/2019 8:35:12 PM

Symptom

When attempting to store text containing emojis and certain other Unicode characters in the customer's MySQL business database from a Write to Data Store node, the customer encounters the following error:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default character set configuration of the MySQL database not supporting the full set of unicode characters.

Action

On the customer's MySQL database, first check the current default character set at different levels as well as the current default collation setting by running the following commands:

select *
from performance_schema.global_variables
where variable_name like '%character%'
or variable_name like '%collation%';

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, the customer could choose to do nothing about the problem. This query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Fri, 27 Sep 2019 18:22:18 GMT

Revision 6 posted to Appian Knowledge Base by Michael Haines on 9/27/2019 6:22:18 PM

Symptom

When attempting to store text containing emojis and certain other Unicode characters in the customer's MySQL business database, the customer encounters the following error:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default configuration of the MySQL database not supporting the full set of unicode characters.

Action

On the customer's MySQL database, run the following commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, the customer could choose to do nothing about the problem. This query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Thu, 26 Sep 2019 17:26:52 GMT

Revision 5 posted to Appian Knowledge Base by Michael Haines on 9/26/2019 5:26:52 PM

Symptom

When attempting to store text containing emojis and certain other Unicode characters in the customer's MySQL business database, the customer encounters the following error:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the default configuration of the MySQL database not supporting the full set of unicode characters.

Action

On the customer's MySQL database, run the following commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

Or, the customer could choose to do nothing about the problem. This query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed:September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Thu, 26 Sep 2019 17:23:19 GMT

Revision 4 posted to Appian Knowledge Base by Michael Haines on 9/26/2019 5:23:19 PM

Symptom

When attempting to store text containing emojis and certain other Unicode characters in the customer's MySQL business database, they encounter the following error:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the MySQL database by default not being configured to support the full set of unicode characters.

Action

On the customer's MySQL database, run the following SQL commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

Any new tables created will inherit the default charater set and collation. But any tables that are already created with the previous character set and collation will not automatically switch their character set to the new one. The customer will need to do that explicitly themselves. For each <table_name>, the customer should run:

alter table <table_name> convert to character set utf8mb4;

They need to do it themselves because something may not be compatible between two character sets in which case they will need to decide what to do. For example, latin1 characters only take one byte per character. The maximum index key length in mysql is 767 bytes. So they can have an index on a character field that is 767 bytes. But utf8mb4 can take up to 4 bytes per character. Which means that you can have only have an index on a varchar field that is 767/4=191 characters only. The column length can be more than 191 characters but the index key length cannot be more than 191 characters. They will need to decide what to do in that case:

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

The following query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from information_schema.columns
where character_set_name is not null and table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed:September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Thu, 26 Sep 2019 17:22:38 GMT

Revision 3 posted to Appian Knowledge Base by Michael Haines on 9/26/2019 5:22:38 PM

Symptom

When attempting to store text containing emojis and certain other Unicode characters in the customer's MySQL business database, they encounter the following error:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the MySQL database by default not being configured to support the full set of unicode characters.

Action

On the customer's MySQL database, run the following SQL commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

The following query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type from information_schema.columns where character_set_name is not null and table_schema = 'Appian' order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed:September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Thu, 26 Sep 2019 17:21:29 GMT

Revision 2 posted to Appian Knowledge Base by Michael Haines on 9/26/2019 5:21:29 PM

Symptom

When attempting to store text containing emojis and certain other Unicode characters in the customer's MySQL business database, they encounter the following error:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1
2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the MySQL database by default not being configured to support the full set of unicode characters.

Action

On the customer's MySQL database, run the following SQL commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4'; set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

The following query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type from information_schema.columns where character_set_name is not null and table_schema = 'Appian' order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed:September 2019

Tags: database, mysql, Cloud

DRAFT - INT-XXXX - How to update MySQL collation to support emojis

Michael Haines — Thu, 26 Sep 2019 17:10:56 GMT

Revision 1 posted to Appian Knowledge Base by Michael Haines on 9/26/2019 5:10:56 PM

Symptom

When attempting to store text containing emojis and certain other Unicode characters in the customer's MySQL business database, they encounter the following error:

2019-08-22 06:56:39,213 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: 'blah' for column 'blah' at row 1 2019-08-22 06:56:39,214 [Appian Work Item - 75 - execution00 : UnattendedJavaActivityRequest] ERROR com.appiancorp.type.external.writetodatastore.WriteToDataStoreCore - Error when trying to write to the data store. com.appiancorp.suiteapi.process.exceptions.SmartServiceException: userMsg[error.duringWrite.userMsg=An error occurred while trying to write to the entity <entity> [id=<uuid>, type=<type> (id=<id>)] (data store: <data store>). Details: org.hibernate.exception.GenericJDBCException: could not insert: [<entity>]: java.sql.SQLException: Incorrect string value: '<value>' for column '<column>' at row <rownum> Data: TypedValue[it=<id>,v={<value>}

Cause

This is caused by the MySQL database by default not being configured to support the full set of unicode characters.

Action

On the customer's MySQL database, run the following SQL commands to manually set the default server level character set and collation:

set global character_set_server = 'utf8mb4';
set global collation_server = 'utf8mb4_unicode_ci';

alter table <table_name> convert to character set utf8mb4;

Reduce the length of the column on which there is an index to 191 characters only. Drop the index. Change the character set. Then recreate the index.
Keep the column length as it is but just recreate the index with a maximum key length of 191 chars only. This will work if the index is not unique. It will not work if the index is unique.

The following query will help them identify which tables have columns that are using a character set:

select table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_set_name, collation_name, column_type
from   information_schema.columns
where  character_set_name is not null
and    table_schema = 'Appian'
order by table_name, ordinal_position;

Affected Versions

This article applies to all versions of Appian.

Last Reviewed:September 2019

Tags: database, mysql, Cloud