Processing CSV files using biterscripting

I often use biterscripting for processing CSV files. It has no limits on number of columns/fields or number of rows.

It seems to be pretty simple to learn - I am not a programmer - but was able to pick it up after seeing a few sample codes on the internet.

I will provide free sample code for you here.

You said:

I have an export from a survey SPSS file in CSV format. It's unusual because it has about 2,500 fields. I don't want to manipulate it to be smaller, and Access and SQL Server clearly have limitations (256 & 1024 fields respectively). I'll have very simple queries to run and only a max of 20k rows; and no time issues.

I will assume

- The CSV (Comma Separated Values) file is stored at C:/SPSS.csv .
- You want to get the totals of column 2437, which is a real number. (This, of course, is arbitrary - just wanted to show you how things can be done.

Here is the script.

# Set field separator to comma
set $wsep = ","
# Store total here
var real total
# Read data in
var str data ; cat "C:/SPSS.csv" > $data
# Process rows one at a time
while ($data <> "")
do
# Get next row
var str row ; lex -e "1" $data > $row
# Extract column 2437, convert to real.
var str column2437 ; wex "2437" $row > $column2437
var real value ; set $value = makereal(str($column2437))
# Add to total
set $total = $total + $value
done
# Show total
echo $total

Download biterscripting at http://www.biterscripting.com . I thinnk it is free. They have some other sample scripts related to CSV and data files.

Jenni