Repo Management

From Manjaro-Arm Building
Jump to: navigation, search

Repo Management is a script written by Strit to be used to help automate the maintaining of Manjaro-Arm repos. The script is functional but can be improved.

Tasks

The tasks of this script is simple.

  • Grab ArchlinuxArm (alarm) packages
  • Remove the packages in the alarm directory that correspond to the same packages that manjaro-arm maintains
  • Rebuild the database

Adapting it to new repo

Since this script was hardcoded for the mirror.strits.dk repo, it needs to be changed to sync with the server hosting your repo. The script is run on the repo server itself. You just need to adjust the variables.

Usage

Usage is pretty straight forward. Just run the script and/or add it to a cron job.

Script

Repo Mananagement
#!/bin/bash

# Define path
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin                                                                   

# Define variables
UPLOAD='/mnt/Upload/manjaro-arm/staging'
ARM='/mnt/Upload/jenkins'
REPO='/mnt/mirrors/manjaro-arm/stable'
GPGKEY='3C7489C71368F5B4A3E66D3C70FBB189B338D5DF'
SRC='/mnt/mirrors/manjaro-arm/stable'


# Sign all the packages
echo "Signing not-signed packages..."
ls -1 $UPLOAD/armv6h/core/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/armv6h/extra/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/armv6h/community/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/armv7h/core/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/armv7h/extra/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/armv7h/community/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/aarch64/core/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/aarch64/extra/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/aarch64/community/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/any/core/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/any/extra/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file

ls -1 $UPLOAD/any/community/*.pkg.tar.xz > list.file 2> /dev/null
while read line2; do
    if [ -f "$line2.sig" ]
    then
    echo "" > /dev/null
    else
    gpg --detach-sign --default-key $GPGKEY "$line2" 2> /dev/null
    fi
done < list.file


# Run wget commands from alarm servers
echo "Fetching new packages from Arch Linux ARM repository... This will take a while!"
echo "aarch64 community..."
	cd $REPO/aarch64/community/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/community/
	wget -q -r -np -nH -nd -nc -A "dtc-overlay*.xz","dtc-overlay*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "uboot-tools*.xz","uboot-tools*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "vboot-utils*.xz","vboot-utils*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "aml-libs-c2*.xz","aml-libs-c2*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "xinput_calibrator*.xz","xinput_calibrator*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "libcrossguid-git*.xz","libcrossguid-git*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/aur/
	wget -q -r -np -nH -nd -nc -A "kodi-c2*.xz","kodi-c2*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/

echo "aarch64 core..."
	cd $REPO/aarch64/core/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/core/
	wget -q -r -np -nH -nd -nc -A "xf86-video-fbturbo-git*.xz","xf86-video-fbturbo-git*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "xf86-video-odroid-c2*.xz","xf86-video-odroid-c2*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "odroid-c2-libgl*.xz","odroid-c2-libgl*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/
	wget -q -r -np -nH -nd -nc -A "uboot-odroid-c2*.xz","uboot-odroid-c2*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/alarm/

echo "aarch64 extra..."
	cd $REPO/aarch64/extra/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/aarch64/extra/
	rm ffmpeg-1\:2.7*

echo "armv7h community..."
	cd $REPO/armv7h/community/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/community/
	wget -q -r -np -nH -nd -nc -A "kodi*.xz","kodi*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "libcec-rpi*.xz","libcec-rpi*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "omxplayer-git*.xz","omxplayer-git*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "dtc-overlay*.xz","dtc-overlay*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "wiringpi*.xz","wiringpi*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "uboot-tools*.xz","uboot-tools*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "vboot-utils*.xz","vboot-utils*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "xinput_calibrator*.xz","xinput_calibrator*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "libcrossguid-git*.xz","libcrossguid-git*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/aur/

echo "armv7h core..."
	cd $REPO/armv7h/core/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/core/
	wget -q -r -np -nH -nd -nc -A "xf86-video-fbturbo-git*.xz","xf86-video-fbturbo-git*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "uboot-beagle*.xz","uboot-beagle*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "uboot-odroid-c1*.xz","uboot-odroid-c1*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "uboot-odroid-xu3*.xz","uboot-odroid-xu3*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "uboot-raspberrypi*.xz","uboot-raspberrypi*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "xf86-video-odroid-c1*.xz","xf86-video-odroid-c1*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/

echo "armv7h extra..."
	cd $REPO/armv7h/extra/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/extra/

echo "armv6h community..."
	cd $REPO/armv6h/community/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/community/
	wget -q -r -np -nH -nd -nc -A "kodi*.xz","kodi*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/alarm/
	wget -q -r -np -nH -nd -nc -A "libcec-rpi*.xz","libcec-rpi*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/alarm/
	wget -q -r -np -nH -nd -nc -A "omxplayer-git*.xz","omxplayer-git*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/alarm/
	wget -q -r -np -nH -nd -nc -A "dtc-overlay*.xz","dtc-overlay*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/alarm/
	wget -q -r -np -nH -nd -nc -A "wiringpi*.xz","wiringpi*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/alarm/
	wget -q -r -np -nH -nd -nc -A "xinput_calibrator*.xz","xinput_calibrator*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/alarm/
	wget -q -r -np -nH -nd -nc -A "libcrossguid-git*.xz","libcrossguid-git*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/aur/

echo "armv6h core..."
	cd $REPO/armv6h/core/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/core/
	wget -q -r -np -nH -nd -nc -A "xf86-video-fbturbo-git*.xz","xf86-video-fbturbo-git*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/alarm/

echo "armv6h extra..."
	cd $REPO/armv6h/extra/
	wget -q -r -np -nH -nd -nc -A "*.xz","*.sig*" http://de4.mirror.archlinuxarm.org/armv6h/extra/

echo "any core..."
	cd $REPO/any/core/
	wget -q -r -np -nH -nd -nc -A "raspberrypi-firmware*.xz","raspberrypi-firmware*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/
	wget -q -r -np -nH -nd -nc -A "raspberrypi-bootloader*.xz","raspberrypi-bootloader*.sig*" http://de4.mirror.archlinuxarm.org/armv7h/alarm/



#Running the merge commands
echo "Removing alarm packages, we maintain ourselves..." 
while read line; do
	rm -r $REPO/armv7h/core/$line
done < $ARM/armv7h-core-list

while read line; do
	rm -r $REPO/armv7h/extra/$line
done < $ARM/armv7h-extra-list

while read line; do
	rm -r $REPO/armv6h/core/$line
done < $ARM/armv6h-core-list

while read line; do
	rm -r $REPO/armv6h/extra/$line
done < $ARM/armv6h-extra-list

while read line; do
	rm -r $REPO/aarch64/core/$line
done < $ARM/aarch64-core-list

while read line; do
	rm -r $REPO/aarch64/extra/$line
done < $ARM/aarch64-extra-list



# Move the signed packages and the signatures to the manjaro-arm repo
echo "Moving signed packages and signatures to the Manjaro-ARM repo..."
if [ ! -z 'ls $UPLOAD/armv6h/core/*' ]
then cp -p $UPLOAD/armv6h/core/* $REPO/armv6h/core/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/armv6h/extra/*' ]
then cp -p $UPLOAD/armv6h/extra/* $REPO/armv6h/extra/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/armv6h/community/*' ]
then cp -p $UPLOAD/armv6h/community/* $REPO/armv6h/community/ 2> /dev/null
fi

if [ ! -z 'ls $UPLOAD/armv7h/core/*' ]
then cp -p $UPLOAD/armv7h/core/* $REPO/armv7h/core/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/armv7h/extra/*' ]
then cp -p $UPLOAD/armv7h/extra/* $REPO/armv7h/extra/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/armv7h/community/*' ]
then cp -p $UPLOAD/armv7h/community/* $REPO/armv7h/community/ 2> /dev/null
fi

if [ ! -z 'ls $UPLOAD/aarch64/core/*' ]
then cp -p $UPLOAD/aarch64/core/* $REPO/aarch64/core/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/aarch64/extra/*' ]
then cp -p $UPLOAD/aarch64/extra/* $REPO/aarch64/extra/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/aarch64/community/*' ]
then cp -p $UPLOAD/aarch64/community/* $REPO/aarch64/community/ 2> /dev/null
fi

if [ ! -z 'ls $UPLOAD/any/core/*' ]
then cp -p $UPLOAD/any/core/* $REPO/any/core/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/any/extra/*' ]
then cp -p $UPLOAD/any/extra/* $REPO/any/extra/ 2> /dev/null
fi
if [ ! -z 'ls $UPLOAD/any/community/*' ]
then cp -p $UPLOAD/any/community/* $REPO/any/community/ 2> /dev/null
fi


#Do some symlinking
echo "Symlinking packages and signatures from the any folder..."
cd $SRC/any/core &&
for f in *
do
ln -s ../../any/core/"$f" ../../armv6h/core/"$f"
ln -s ../../any/core/"$f" ../../armv7h/core/"$f" 
ln -s ../../any/core/"$f" ../../aarch64/core/"$f"
done
cd $SRC/any/extra &&
for f in *
do
ln -s ../../any/extra/"$f" ../../armv6h/extra/"$f"
ln -s ../../any/extra/"$f" ../../armv7h/extra/"$f" 
ln -s ../../any/extra/"$f" ../../aarch64/extra/"$f"
done
cd $SRC/any/community &&
for f in *
do
ln -s ../../any/community/"$f" ../../armv6h/community/"$f"
ln -s ../../any/community/"$f" ../../armv7h/community/"$f" 
ln -s ../../any/community/"$f" ../../aarch64/community/"$f"
done

#then we remove broken symlinks
echo "Removing broken symlinks..."
cd $SRC/armv6h/core &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done
cd $SRC/armv6h/extra &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done
cd $SRC/armv6h/community &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done

cd $SRC/armv7h/core &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done 
cd $SRC/armv7h/extra &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done 
cd $SRC/armv7h/community &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done 

cd $SRC/aarch64/core &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done 
cd $SRC/aarch64/extra &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done
cd $SRC/aarch64/community &&
for x in * .[!.]* ..?*; do if [ -L "$x" ] && ! [ -e "$x" ]; then rm -- "$x"; fi; done


#removing potential stopping files
rm $SRC/armv6h/core/core.db.tar.gz.lck
rm $SRC/armv6h/extra/extra.db.tar.gz.lck
rm $SRC/armv6h/community/community.db.tar.gz.lck
rm $SRC/armv7h/core/core.db.tar.gz.lck
rm $SRC/armv7h/extra/extra.db.tar.gz.lck
rm $SRC/armv7h/community/community.db.tar.gz.lck
rm $SRC/aarch64/core/core.db.tar.gz.lck
rm $SRC/aarch64/extra/extra.db.tar.gz.lck
rm $SRC/aarch64/community/community.db.tar.gz.lck


#then we do the repo-adds
echo "Running repo-add commands... This will take a while!"
cd $SRC/armv6h/core
repo-add -q -n -R core.db.tar.gz *.pkg.tar.xz &&
cd $SRC/armv6h/extra
repo-add -q -n -R extra.db.tar.gz *.pkg.tar.xz &&
cd $SRC/armv6h/community
repo-add -q -n -R community.db.tar.gz *.pkg.tar.xz &&

cd $SRC/armv7h/core
repo-add -q -n -R core.db.tar.gz *.pkg.tar.xz &&
cd $SRC/armv7h/extra
repo-add -q -n -R extra.db.tar.gz *.pkg.tar.xz &&
cd $SRC/armv7h/community
repo-add -q -n -R community.db.tar.gz *.pkg.tar.xz &&

cd $SRC/aarch64/core
repo-add -q -n -R core.db.tar.gz *.pkg.tar.xz &&
cd $SRC/aarch64/extra
repo-add -q -n -R extra.db.tar.gz *.pkg.tar.xz &&
cd $SRC/aarch64/community
repo-add -q -n -R community.db.tar.gz *.pkg.tar.xz &&