I want to save csv files that I download from a server in a GitLab repo. The data is updated weekly. I would like to write a c shell script, which should be executed using .gitlab-ci.yml.
It should be checked, which file has already been downloaded and only the missing csv files should download.
How can I write an array in c shell? I have data for many years 2000, 2001, 2002 … 2018
set Year=[2000, 2001, 2002]
set Week=[1..53]
The if loops are not correct. Which loop is better or what I do made wrong?
How can I check if the contents of a csv file are damaged? is
gzip -t $Year/$Week.csv.zip the right choice?
How is the commit message updated?
- git commit -m 'Add "$Symbol"-"$Year"-"$Week".csv.zip'
Here is the .gitlab-ci.yml:
image: alpine:latest
before_script:
- apk update
- apk upgrade
- apk add git zip
# Identification
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS@gitlab.com/$CI_PROJECT_PATH.git
- git config --global user.email "user.email@example.com"
- git config --global user.name "Max"
job1:
script:
- URL="https://stars.example.com"
- Symbol="alpha"
- Year="2015"
- Week="3"
# Check if Directory Year exists
- (if [ -d $Year ]; then echo $Year found; else mkdir $Year; fi);
# Download the cvs files
- (if [ -s $Year/$Symbol-$Year-$Week.csv.zip ]; then echo $Symbol-$Year-$Week.csv.zip found; else wget -c $URL/$Symbol/$Year/$Week.csv.gz -P $Year; fi);
# Test the validity of a compressed File
- gzip -t $Year/$Week.csv.gz && echo The file is okay || echo The file is corrupted
# Extract the cvs file
- gunzip $Year/$Week.csv.gz
# Compress csv file to Stars-Year-Week.csv.zip
- zip $Year/$Symbol-$Year-$Week.csv.zip $Year/$Week.csv
# Delete .csv file
- rm $Year/$Week.csv
# Push to gitlab
- git checkout master
- git add .
- git commit -m 'Add "$Symbol"-"$Year"-"$Week".csv.zip'
- git push origin master