Semi-Automated Batch Download and Update of GitLab Projects Using Bash and API

When joining a new team or needing to clone many GitLab repositories, copying each URL manually is inefficient; this article presents a semi‑automated approach that combines a few lightweight tools instead of a full script.

Step 1: Retrieve the project list via the GitLab API

Use the API endpoint

https://jihulab.com/api/v4/groups/{group_id}/projects?per_page=100&page=1

(replace {group_id} with the actual group ID) to obtain all projects in a group. Remember that pagination defaults to 20 items per page and can be increased up to 100.

curl --location --request GET 'https://jihulab.com/api/v4/groups/845/projects?per_page=100&page=1' \
--header 'Content-Type: application/json' \
--header 'Cookie: xxxx'

Step 2: Extract the SSH URLs

Feed the JSON response to jq (online at https://jqplay.org/ or locally) and use the filter .[] .ssh_url_to_repo to obtain a list of repository URLs.

"[email protected]:gitlab-cn/let-me-create-gitlab-board-for-you.git"
"[email protected]:gitlab-cn/gitlab-svgs.git"
"[email protected]:gitlab-cn/developer-guide.git"
... (additional URLs)

Step 3: Bash script for bulk cloning

codeArray=(
"[email protected]:gitlab-cn/let-me-create-gitlab-board-for-you.git"
"[email protected]:gitlab-cn/gitlab-svgs.git"
"[email protected]:gitlab-cn/developer-guide.git"
... (other URLs)
)
for code in ${codeArray[@]}
do
    echo $code
    git clone $code
done

Make the script executable and run it:

chmod a+x gitClone.sh
./gitClone.sh

Step 4: Bulk update of existing clones

Use a similar traversal script to iterate over directories, pull the latest changes, and optionally run Maven clean:

#! /bin/bash
function getdir(){
    echo "Scanning directory $1"
    for e in `ls $1`
    do
        dir_or_file="$1/$e"
        if [ "$e" = . ] || [ "$e" = .. ]; then
            continue
        fi
        if [ -d $dir_or_file ]
        then
            cd $dir_or_file
            if [[ -e ".git" ]]; then
                echo "git pulling $dir_or_file"
                git checkout -f master
                git pull
                mvn clean
            fi
            cd ..
        fi
    done
}
function main(){
    getdir `pwd`
}
# start
main

Ensure you have the necessary permissions to clone or pull each repository.

Summary

The semi‑automated workflow consists of (1) using the GitLab API to list projects, (2) extracting SSH URLs with jq, (3) looping over the URLs in a Bash script to clone them, and (4) optionally traversing the cloned directories to pull updates.