Charset conversion to utf8 and ENUM fields

[expired user #9902]'s profile image [expired user #9902] posted 8 years ago in General Permalink

Hi, I am performing database charset conversions (Tool > Table Tools > Bulk Table Editor > Convert to charset) from latin1 to utf8mb4. The tool provided by HeidiSQL is really great, but I have issues with ENUM fields which have special chars (french accents!) in their values.

My initial table structure is:

CREATE TABLE IF NOT EXISTS `catalogues_clients` (
  `ref` int(11) NOT NULL AUTO_INCREMENT,
  `pseudo` varchar(20) NOT NULL DEFAULT '',
  `etat_donnees` enum('Ok','Problème') NOT NULL DEFAULT 'Problème',
  PRIMARY KEY (`ref`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

First issue, etat_donnees has a default value that contains an accent ("Problème"), and HeidiSQL fails doing the conversion:

SQL Error (1067): Invalid default value for 'etat_donnees'.

If I do the following change on the structure to remove the default value , then HeidiSQL agrees to do the conversion:

CREATE TABLE IF NOT EXISTS `catalogues_clients` (
  `ref` int(11) NOT NULL AUTO_INCREMENT,
  `pseudo` varchar(20) NOT NULL DEFAULT '',
  `etat_donnees` enum('Ok','Problème') NOT NULL,
  PRIMARY KEY (`ref`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

But second issue, after conversion, my table now looks like that:

CREATE TABLE IF NOT EXISTS `catalogues_clients` (
  `ref` int(11) NOT NULL AUTO_INCREMENT,
  `pseudo` varchar(20) NOT NULL DEFAULT '',
  `etat_donnees` enum('Ok','Probl?me') NOT NULL,
  PRIMARY KEY (`ref`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;

And of course, all the previous values "Problème" now display as "Probl?me".

For sure it would be quite easy to write a small script that will correct this, but as I have several databases with many tables with many fields (!), I would prefer that the HeidiSQL conversion works at once...

Do you know if it is a known bug of HeidiSQL and if there is a workaround? Thanks for your help;-)

ansgar's profile image ansgar posted 8 years ago Permalink

I get this when I try to convert that latin1-table to utf-8:

ALTER TABLE `test`.`catalogues_clients` CONVERT TO CHARSET utf8;
/* SQL Fehler (1067): Invalid default value for 'etat_donnees' */

I guess you were on a server which does not complain about that but simply does a wrong conversion. Means you have to manually fix that ENUM field I'm afraid.

[expired user #9902]'s profile image [expired user #9902] posted 8 years ago Permalink

Thank you Ansgar, you are right, the problem does not come prom HeidiSQL but from MySQL itself that does perform a wrong conversion with ENUM values. I am afraid that what I have to do before converting a table is:

  1. Creating new ENUM values without accents
  2. Update the records to the values without accents
  3. Delete the ENUM values with accents
  4. Perform the UTF8 conversion
  5. Create new ENUM values with accents
  6. Update the records to the values with accents
  7. Delete the ENUM values without accents. :-( So next step is to avoid to use ENUM...

Please login to leave a reply, or register at first.