Automatically import .csv file to database

Eisram's profile image Eisram posted 3 years ago in General Permalink

Hello,

I need to import a .csv file into HeidiSQL automatically, preferrebly via batch file or something task scheduler compatible.

I know there already is a topic about this question but it was asked 6 years ago. It would be awesome if someone could help since I tried to realise a solution for the past week or so...

ansgar's profile image ansgar posted 3 years ago Permalink

You cannot automate CSV imports in HeidiSQL that way.

I recommend to invoke a mysql command line instead, executing such a command per task scheduler:

mysql -hlocalhost -uroot --execute="LOAD DATA LOCAL INFILE 'C:\\temp\\whatever.csv'
   INTO TABLE mydb.mytable
   CHARACTER SET utf8 FIELDS TERMINATED BY ';' optionally ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '\n'
   (col1, col2, col3, ...);"
Eisram's profile image Eisram posted 3 years ago Permalink

Thanks so much :) I will try it as soon as I can!

flo44's profile image flo44 posted 2 years ago Permalink

Hey Ansgar, im trying to do the same thing as Eisram.

this is my command line so far but it doesnt work. There is no data imported into the db. What am I doing wrong? would be awesome if u can help me.

mysql -h 127.0.0.1 -u root --execute="LOAD DATA LOW_PRIORITY LOCAL INFILE 'C:\Users\flosc\Desktop\UNI\KOOP\ALS\20220407.CSV' REPLACE INTO TABLE c-factory.daten_als FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '\r\n' IGNORE 20 LINES (Toleranz, @ColVar1, @ColVar2, Uhrzeit, @ColVar4, @ColVar5, @ColVar6, @ColVar7, @ColVar8, @ColVar9) SET Zähler = REPLACE(REPLACE(@ColVar1, '.', ''), ',', '.'), Gutteile = REPLACE(REPLACE(@ColVar2, '.', ''), ',', '.'), Dosierzeit = REPLACE(REPLACE(@ColVar4, '.', ''), ',', '.'), Zykluszeit = REPLACE(REPLACE(@ColVar5, '.', ''), ',', '.'), Einspritzzeit = REPLACE(REPLACE(@ColVar6, '.', ''), ',', '.'), Spritzdruck = REPLACE(REPLACE(@ColVar7, '.', ''), ',', '.'), Umschaltvolumen = REPLACE(REPLACE(@ColVar8, '.', ''), ',', '.'), Massepolster = REPLACE(REPLACE(@ColVar9, '.', ''), ',', '.');"

ansgar's profile image ansgar posted 2 years ago Permalink

@flo44 Probably the file does not fit to your LOAD command.

Please attach that 20220407.CSV here. Be sure it does not contain sensitive information!

flo44's profile image flo44 posted 2 years ago Permalink

thank you for your fast reply!

this is the csv file:

1 attachment(s):
ansgar's profile image ansgar posted 2 years ago Permalink

Forgot to say, please also post the CREATE code of your "daten_als" table. You can copy it from the "Table" tab in HeidiSQL.

flo44's profile image flo44 posted 2 years ago Permalink

sorry for my late reply, thats the create code:

CREATE TABLE `daten_als` (
    `Toleranz` TEXT NULL DEFAULT NULL COLLATE 'utf8mb3_german2_ci',
    `Zähler` FLOAT NULL DEFAULT NULL,
    `Gutteile` FLOAT NULL DEFAULT NULL,
    `Uhrzeit` TEXT NULL DEFAULT NULL COLLATE 'utf8mb3_german2_ci',
    `Dosierzeit` FLOAT NULL DEFAULT NULL,
    `Zykluszeit` FLOAT NULL DEFAULT NULL,
    `Einspritzzeit` FLOAT NULL DEFAULT NULL,
    `Spritzdruck` FLOAT NULL DEFAULT NULL,
    `Umschaltvolumen` FLOAT NULL DEFAULT NULL,
    `Massepolster` FLOAT NULL DEFAULT NULL
)
COLLATE='utf8mb3_german2_ci'
ENGINE=InnoDB
;
ansgar's profile image ansgar posted 2 years ago Permalink

@flo44: the table and the LOAD command looks quite ok, and works fine here. The only issue with the LOAD command is that the file path needed escaped backslashes:

... LOCAL INFILE 'C:\\Users\\flosc\\Deskt...' REPLACE ...

But you should have seen an error message like "cannot find file" or similar, didn't you?

flo44's profile image flo44 posted 2 years ago Permalink

@ansgar: there were no error messages after i tried the command. there are escaped backslashes but they are not shown in the preview when i post something here.

this is what the cmd tab is showing me when i type in the LOAD command:

C:\WINDOWS\system32>mysql -h 127.0.0.1 -u root --execute="LOAD DATA LOW_PRIORITY LOCAL INFILE 'C:\Users\flosc\Desktop\UNI\KOOP\ALS\20220407.CSV' REPLACE INTO TABLE c-factory.daten_als FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '\r\n' IGNORE 20 LINES (Toleranz, @ColVar1, @ColVar2, Uhrzeit, @ColVar4, @ColVar5, @ColVar6, @ColVar7, @ColVar8, @ColVar9) SET Zähler = REPLACE(REPLACE(@ColVar1, '.', ''), ',', '.'), Gutteile = REPLACE(REPLACE(@ColVar2, '.', ''), ',', '.'), Dosierzeit = REPLACE(REPLACE(@ColVar4, '.', ''), ',', '.'), Zykluszeit = REPLACE(REPLACE(@ColVar5, '.', ''), ',', '.'), Einspritzzeit = REPLACE(REPLACE(@ColVar6, '.', ''), ',', '.'), Spritzdruck = REPLACE(REPLACE(@ColVar7, '.', ''), ',', '.'), Umschaltvolumen = REPLACE(REPLACE(@ColVar8, '.', ''), ',', '.'), Massepolster = REPLACE(REPLACE(@ColVar9, '.', ''), ',', '.');" mysql Ver 15.1 Distrib 10.7.3-MariaDB, for Win64 (AMD64), source revision 65f602310c4ea086488bfaa50f36952162541a8c Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Usage: mysql [OPTIONS] [database]

Default options are read from the following files in the given order: C:\WINDOWS\my.ini C:\WINDOWS\my.cnf C:\my.ini C:\my.cnf C:\Program Files\MariaDB 10.7\my.ini C:\Program Files\MariaDB 10.7\my.cnf C:\Program Files\MariaDB 10.7\data\my.ini C:\Program Files\MariaDB 10.7\data\my.cnf The following groups are read: mysql mariadb-client client client-server client-mariadb The following options may be given as the first argument: --print-defaults Print the program argument list and exit. --no-defaults Don't read default options from any option file. The following specify which files/extra groups are read (specified before remaining options): --defaults-file=# Only read default options from the given file #. --defaults-extra-file=# Read this file after the global files are read. --defaults-group-suffix=# Additionally read default groups with # appended as a suffix.

-?, --help Display this help and exit. -I, --help Synonym for -? --abort-source-on-error Abort 'source filename' operations in case of errors --auto-rehash Enable automatic rehashing. One doesn't need to use 'rehash' to get table and field completion, but startup and reconnecting may take a longer time. Disable with --disable-auto-rehash. (Defaults to on; use --skip-auto-rehash to disable.) -A, --no-auto-rehash No automatic rehashing. One has to use 'rehash' to get table and field completion. This gives a quicker start of mysql and disables rehashing on reconnect. --auto-vertical-output Automatically switch to vertical output mode if the result is wider than the terminal width. -B, --batch Don't use history file. Disable interactive behavior. (Enables --silent.) --binary-as-hex Print binary data as hex --character-sets-dir=name Directory for character set files. --column-type-info Display column type information. -c, --comments Preserve comments. Send comments to the server. The default is --skip-comments (discard comments), enable with --comments. -C, --compress Use compression in server/client protocol. -#, --debug[=#] This is a non-debug version. Catch this and exit. --debug-check Check memory and open file usage at exit. -T, --debug-info Print some debug info at exit. -D, --database=name Database to use. --default-character-set=name Set the default character set. --delimiter=name Delimiter to be used. -e, --execute=name Execute command and quit. (Disables --force and history file.) -E, --vertical Print the output of a query (rows) vertically. -f, --force Continue even if we get an SQL error. Sets abort-source-on-error to 0 -G, --named-commands Enable named commands. Named commands mean this program's internal commands; see mysql> help . When enabled, the named commands can be used from any line of the query, otherwise only from the first line, before an enter. Disable with --disable-named-commands. This option is disabled by default. -i, --ignore-spaces Ignore space after function names. --init-command=name SQL Command to execute when connecting to MariaDB server. Will automatically be re-executed when reconnecting. --local-infile Enable/disable LOAD DATA LOCAL INFILE. -b, --no-beep Turn off beep on error. -h, --host=name Connect to host. -H, --html Produce HTML output. -X, --xml Produce XML output. --line-numbers Write line numbers for errors. (Defaults to on; use --skip-line-numbers to disable.) -L, --skip-line-numbers Don't write line number for errors. -n, --unbuffered Flush buffer after each query. --column-names Write column names in results. (Defaults to on; use --skip-column-names to disable.) -N, --skip-column-names Don't write column names in results. --sigint-ignore Ignore SIGINT (CTRL-C). -o, --one-database Ignore statements except those that occur while the default database is the one named at the command line. --pager[=name] Pager to use to display results. If you don't supply an option, the default pager is taken from your ENV variable PAGER. Valid pagers are less, more, cat [> filename], etc. See interactive help (\h) also. This option does not work in batch mode. Disable with --disable-pager. This option is disabled by default. -p, --password[=name] Password to use when connecting to server. If password is not given it's asked from the tty. -W, --pipe Use named pipes to connect to server. -P, --port=# Port number to use for connection or 0 for default to, in order of preference, my.cnf, $MYSQL_TCP_PORT, /etc/services, built-in default (3306). --progress-reports Get progress reports for long running commands (like ALTER TABLE) (Defaults to on; use --skip-progress-reports to disable.) --prompt=name Set the command line prompt to this value. --protocol=name The protocol to use for connection (tcp, socket, pipe). -q, --quick Don't cache result, print it row by row. This may slow down the server if the output is suspended. Doesn't use history file. -r, --raw Write fields without conversion. Used with --batch. --reconnect Reconnect if the connection is lost. Disable with --disable-reconnect. This option is enabled by default. (Defaults to on; use --skip-reconnect to disable.) -s, --silent Be more silent. Print results with a tab as separator, each row on new line. -S, --socket=name The socket file to use for connection. --ssl Enable SSL for connection (automatically enabled with other flags). --ssl-ca=name CA file in PEM format (check OpenSSL docs, implies --ssl). --ssl-capath=name CA directory (check OpenSSL docs, implies --ssl). --ssl-cert=name X509 cert in PEM format (implies --ssl). --ssl-cipher=name SSL cipher to use (implies --ssl). --ssl-key=name X509 key in PEM format (implies --ssl). --ssl-crl=name Certificate revocation list (implies --ssl). --ssl-crlpath=name Certificate revocation list path (implies --ssl). --tls-version=name TLS protocol version for secure connection. --ssl-verify-server-cert Verify server's "Common Name" in its cert against hostname used when connecting. This option is disabled by default. -t, --table Output in table format. --tee=name Append everything into outfile. See interactive help (\h) also. Does not work in batch mode. Disable with --disable-tee. This option is disabled by default. -u, --user=name User for login if not current user. -U, --safe-updates Only allow UPDATE and DELETE that uses keys. -U, --i-am-a-dummy Synonym for option --safe-updates, -U. -v, --verbose Write more. (-v -v -v gives the table output format). -V, --version Output version information and exit. -w, --wait Wait and retry if connection is down. --connect-timeout=# Number of seconds before connection timeout. --max-allowed-packet=# The maximum packet length to send to or receive from server. --net-buffer-length=# The buffer size for TCP/IP and socket communication. --select-limit=# Automatic limit for SELECT when using --safe-updates. --max-join-size=# Automatic limit for rows in a join when using --safe-updates. --secure-auth Refuse client connecting to server if it uses old (pre-4.1.1) protocol. --server-arg=name Send embedded server this as a parameter. --show-warnings Show warnings after every statement. --plugin-dir=name Directory for client-side plugins. --default-auth=name Default authentication client-side plugin to use. --binary-mode Binary mode allows certain character sequences to be processed as data that would otherwise be treated with a special meaning by the parser. Specifically, this switch turns off parsing of all client commands except \C and DELIMITER in non-interactive mode (i.e., when binary mode is combined with either 1) piped input, 2) the --batch mysql option, or 3) the 'source' command). Also, in binary mode, occurrences of '\r\n' and ASCII '\0' are preserved within strings, whereas by default, '\r\n' is translated to '\n' and '\0' is disallowed in user input. --connect-expired-password Notify the server that this client is prepared to handle expired password sandbox mode even if --batch was specified.

Variables (--variable-name=value) and boolean options {FALSE|TRUE} Value (after reading options)


abort-source-on-error FALSE auto-rehash TRUE auto-vertical-output FALSE binary-as-hex FALSE character-sets-dir (No default value) column-type-info FALSE comments FALSE compress FALSE debug-check FALSE debug-info FALSE database (No default value) default-character-set auto delimiter ; vertical FALSE force FALSE named-commands FALSE ignore-spaces FALSE init-command (No default value) local-infile FALSE no-beep FALSE host 127.0.0.1 html FALSE xml FALSE line-numbers TRUE unbuffered FALSE column-names TRUE sigint-ignore FALSE port 3306 progress-reports FALSE prompt \N [\d]> protocol quick FALSE raw FALSE reconnect FALSE socket (No default value) ssl FALSE ssl-ca (No default value) ssl-capath (No default value) ssl-cert (No default value) ssl-cipher (No default value) ssl-key (No default value) ssl-crl (No default value) ssl-crlpath (No default value) tls-version (No default value) ssl-verify-server-cert FALSE table FALSE user root safe-updates FALSE i-am-a-dummy FALSE connect-timeout 0 max-allowed-packet 16777216 net-buffer-length 16384 select-limit 1000 max-join-size 1000000 secure-auth FALSE show-warnings FALSE plugin-dir C:\Program Files\MariaDB 10.7/lib/plugin default-auth (No default value) binary-mode FALSE connect-expired-password FALSE

Please login to leave a reply, or register at first.