New Entity AI™ - disk_ops

dugout-> disk_ops_id.dugout.

Build generic disk operations and routines. These include backups of standardized operation system components that utilize generic archive programs.

#!/bin/sh ############################################################################### ## COPYRIGHT (C) 2022-2024 NEW ENTITY OPERATIONS INC. ALL RIGHTS RESERVED ## CREATED: 2022/02/13 ## INSTANCE: disk_ops_id.dugout ## MODIFIED: 2024/02/07 ## OVERVIEW: Build generic disk operations and routines. These include backups ## of standardized operation system components that utilize generic archive ## programs. Archive programs may also be implemented using an implementation ## specific backup schema or program according to your backup needs. ## HISTORY: VERSION 1.6 ## -> 2022/02/13 (VERSION 1.0) Development ## -> 2022/10/16 (VERSION 1.1) ## -> 2023/11/20 (VERSION 1.2) ## -> 2023/12/17 (VERSION 1.3) ## -> 2023/12/20 (VERSION 1.4) ## -> 2024/01/03 (VERSION 1.5) ## -> 2024/02/07 (VERSION 1.6) Production ############################################################################### ## Backup Operations to Disk are complicated and this script does not help keep ## you safe if you misconfigure something or are being reckless with how you ## apply flags in the underlying backup program of your choice. ## Data loss can easily occur, and it does occur often - that's why the point ## of disk ops is to harden each operation with designated id-specific macros ## ## Please note*: All custom locations or non-standard ops are located in the ## disk_ops_id_custom.dugout file. The routines listed here are just generic ## backup routines implemented on many modern systems. Even if a location on ## your machine is considered standard to you, if it is not listed here, it ## is not thought to be standard by New Entity Operations Inc. ## ## [X] Last Verified 2024/01/03 ## ## ** NOTE *** ## BE CAREFUL! ## ## When in doubt, run a trial run first. Always catch all variables before ## utilizing them in a backup process. ## ## Consider the following points when running automated backup routines: ## 1.) The -z option typically is used to catch empty variables ## 2.) When transferring to/from a remote machine, or utilizing a network, use ## another .dugout template such as connection_mainframe.dugout, ## network_ops_id.dugout, or even run_program.dugout ## 3.) Comment out all backup scripts until they're TESTED with a dry run. ## 4.) If something fails, uncomment it and rework the flow until it's correct ## 5.) Don't allow non-working scripts to even be active within the system. ## 6.) You must use a wrapper to provide alerts for both verbose debugging and ## logging. ## ## The 2024 Backup Super-construct details ## ## The default backup user is root. This can be modified, but it is not ## recommended unless you are an advanced operator that knows how to do this ## properly. ## ## Under normal conditions, you'll need root access or have to provide it with ## something like sudo or similar. ## ## Other user automation account properties can be added when necessary in the ## system_blueprints_id.dugout file ## ## Some of these including: ## wineuser, dj, snoopy, octavian, and realmleader ## -> When an account is added, exclude the a directories ## -> Like this: ## --exclude=/home/$USER/* ## -> or use: ## --exlude-from=FILE.ds ## ## New Entity Linux: v.1.0 and New Entity OS (Reimplemented Kernel) ## -> New Entity Linux: Kernel 6.5.9 2023/11/01 ## -> New Entity OS: Kernel 0.0.4 2024/01/01 ## ## Either Kernels are valid for running automated disk operations. ## ## Features of the Linux version that are not available in the standard OS ## 1. apparmor enhancements ## 2. clamav/maldet enhancements ## 3. proxy-stack enabled and forwarding schemas and dns hardening ## 4. cron-enabled ## 5. visual-ques (sensory-enhancements) through conky/i3 status bars ## -> are enabled in both, but only in a limited capacity on NEOS ## 6. Traditional DBs, along with a generic and active light db: sqlite3 ## ## *NOTE-> See all of the features in your IPDVC documentation that aren't ## ## Features available on both that have to do with automated backups ## 1. .CORE-> Codenamed TRINE (Cognitive Operations Resource Enclave) ## 2. Multi-array thread-limited backups ## 3. virtual-machine Narray backup exporting through FILTER ## -> Narray is the Alpha version of the "RING" framework that ## can speak directly to CORE.HOST ## 4. Icon-based operations pack enabled (NEW ENTITY 38 ICONS INCLUDED) ## 5. /srv instance for custom http I/O proxies and HTML interfaces ## 6. NOVA Framework API through /NOVASTORE ## 7. /EXTRA custom-stack for additional binary system-enhancements ## 8. GATEWAY custom network runtime ## 9. IPDVC documentation and storage stack with EntityScript ## 10. .dugout enhancements and wrappers ## ## When excluding folders from the main backup, consider that these carry ## essential code/files/scripts for a functioning system. ## ## If you don't need these, you may choose to exclude them. If you do ## need them, skip this suggestion: ## --exclude=*ESSENTIAL_PART_1* ## --exclude=*ESSENTIAL_PART_N* ## ## Each backup script is bound by a run_backup_* function that is executed ## ideally through an alias in run_program.dugout ## ## The following labels are valid by default: ## 1.) A* - Main System storage, including full primary system backup and full ## primary backup cloning ## 2.) B* - Secondary conditional backups from the main primary backup ## 3.) C* - Partial primary system backups, represented as select instances ## 4.) X* - External System backups, primarily for cloning the primary system ## into another instance, such as a flash drive or anther HDD/SSD ## 5.) Y* - Secondary backups, branching what's excluded from a B* instance ## 6.) Z* - Additional Media Archives by category ## ## Any other label can be added with a unique purpose, such as D, E, F, etc. ## ## Increasing a number after the key A0... to A1 indicates a device format ## alteration, or a different realm of backup under 1 instance rule set ## ## For safety, ARCHIVE_MOCK=1 is standard. To remove it, set the environment ## variable to 0 manually, or in a linker-script ## ## Types of archiving ## archive_simple_no_rules ## Use-> archive_simple_no_rules ## ## archive_generic_with_rules ## Use-> archive_generic_with_rules ## ## archive_simple_no_rules ## Use-> archive_simple_no_rules ## ## archive_simple_with_rules ## Use-> archive_simple_with_rules ## ############################################################################### ## Select Media: Follow the run_backup_* convention where * is the alias ############################################################################### ## (A001) - Backup the default system to 8TB HDD /SLUG_DISTRO: NOVA (TESTED 2023-12-20) run_backup_NOVA() { if [ -z "${BACKUP_NOVA_TO}" ]; then alert_backup_to_location_not_valid "${BACKUP_NOVA_TO}" else ## Define slugs: All 3 must exists except BACKUP_*_IGNORE_SLU, which ## defaults to a blank slot when empty SLUG_BACKUP_NOVA_FROM="${BACKUP_NOVA_FROM}" SLUG_BACKUP_NOVA_TO="${BACKUP_NOVA_TO}" SLUG_BACKUP_NOVA_IGNORE="${BACKUP_NOVA_IGNORE}" ## requires super-user privs to see the locked nested directory alert_disk_ops_full_archive_now ## Check slugs if sudo [ -d "${SLUG_BACKUP_NOVA_FROM}" ]; then # requires super-user privs to see the locked nested directory alert_disk_ops_full_archive_started alert_disk_ops_view_root_directory confirm_view_directory "/" if sudo [ -d "${SLUG_BACKUP_NOVA_TO}" ]; then alert_disk_ops_directory_located "${SLUG_BACKUP_NOVA_TO}" alert_disk_ops_locate_rules if [ -f "${SLUG_BACKUP_NOVA_IGNORE}" ]; then alert_backup_ruleset_defined "${SLUG_BACKUP_NOVA_IGNORE}" while read -r SLUG_BACKUP_LOGIC do alert_backup_rule_instance "${SLUG_BACKUP_LOGIC}" done < "${SLUG_BACKUP_NOVA_IGNORE}" alert_backup_system_starting "${SLUG_BACKUP_NOVA_FROM}" "${SLUG_BACKUP_NOVA_TO}" alert_disk_ops_initialized archive_generic_with_rules "${SLUG_BACKUP_NOVA_FROM}" "${SLUG_BACKUP_NOVA_TO}" "${SLUG_BACKUP_NOVA_IGNORE}" else alert_backup_ruleset_not_defined alert_backup_standard_starting "${SLUG_BACKUP_NOVA_FROM}" "${SLUG_BACKUP_NOVA_TO}" SLUG_BACKUP_NOVA_IGNORE="NONE" alert_disk_ops_initialized if [ "${SLUG_BACKUP_NOVA_IGNORE}" = "NONE" ]; then archive_simple_no_rules "${SLUG_BACKUP_NOVA_FROM}" "${SLUG_BACKUP_NOVA_TO}" else archive_generic_with_rules "${SLUG_BACKUP_NOVA_FROM}" "${SLUG_BACKUP_NOVA_TO}" "${SLUG_BACKUP_NOVA_IGNORE}" fi fi else alert_backup_location_not_valid "${SLUG_BACKUP_NOVA_TO}" fi else alert_backup_from_location_not_valid "${SLUG_BACKUP_NOVA_FROM}" fi fi } ############################################################################### ## Single instance main system slugs [Working: 2023/12/20] ############################################################################### ### (C001) - Backup of /etc (TESTED: 2023/11/17) run_backup_etc() { if [ -z "${BACKUP_ETC_TO}" ]; then alert_backup_to_location_not_valid "${PATH_ETC_TO}" else ## Define slugs: Assuming no exclusions SLUG_BACKUP_ETC_FROM="${BACKUP_ETC_FROM}" SLUG_BACKUP_ETC_TO="${BACKUP_ETC_TO}" SLUG_BACKUP_ETC_IGNORE="NONE" ## Check slugs if [ -d "${SLUG_BACKUP_ETC_FROM}" ]; then alert_disk_ops_directory_generic_from "${SLUG_BACKUP_ETC_FROM}" if [ -d "${SLUG_BACKUP_ETC_TO}" ]; then alert_disk_ops_directory_slug_located "${SLUG_BACKUP_ETC_TO}" alert_disk_ops_locate_rules if [ "${SLUG_BACKUP_ETC_IGNORE}" = "NONE" ]; then alert_backup_ruleset_not_defined alert_backup_custom_starting "${SLUG_BACKUP_ETC_FROM}" "${SLUG_BACKUP_ETC_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_ETC_FROM}" archive_generic_no_rules "${SLUG_BACKUP_ETC_FROM}" "${SLUG_BACKUP_ETC_TO}" else alert_backup_ruleset_defined "${SLUG_BACKUP_ETC_IGNORE}" alert_backup_standard_starting "${SLUG_BACKUP_ETC_FROM}" "${SLUG_BACKUP_ETC_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_ETC_FROM}" archive_generic_with_rules "${SLUG_BACKUP_ETC_FROM}" "${SLUG_BACKUP_ETC_TO}" "${SLUG_BACKUP_ETC_IGNORE}" fi else alert_backup_location_not_valid "${SLUG_BACKUP_ETC_TO}" fi else alert_backup_from_location_not_valid "${SLUG_BACKUP_ETC_FROM}" fi fi } ## (C002) - Backup of /home (TESTED: 2023/12/20) run_backup_home() { if [ -z "${BACKUP_HOME_TO}" ]; then alert_backup_to_location_not_valid "${PATH_HOME_TO}" else ## Define slugs: Assuming no exclusions SLUG_BACKUP_HOME_FROM="${BACKUP_HOME_FROM}" SLUG_BACKUP_HOME_TO="${BACKUP_HOME_TO}" SLUG_BACKUP_HOME_IGNORE="${BACKUP_HOME_IGNORE}" ## Check slugs if [ -d "${SLUG_BACKUP_HOME_FROM}" ]; then alert_disk_ops_directory_generic_from "${SLUG_BACKUP_HOME_FROM}" if [ -d "${SLUG_BACKUP_HOME_TO}" ]; then alert_disk_ops_directory_slug_located "${SLUG_BACKUP_HOME_TO}" alert_disk_ops_locate_rules if [ "${SLUG_BACKUP_HOME_IGNORE}" = "NONE" ]; then alert_backup_ruleset_not_defined alert_backup_custom_starting "${SLUG_BACKUP_HOME_FROM}" "${SLUG_BACKUP_HOME_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_HOME_FROM}" archive_generic_no_rules "${SLUG_BACKUP_HOME_FROM}" "${SLUG_BACKUP_HOME_TO}" else alert_backup_ruleset_defined "${SLUG_BACKUP_HOME_IGNORE}" alert_backup_standard_starting "${SLUG_BACKUP_HOME_FROM}" "${SLUG_BACKUP_HOME_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_HOME_FROM}" archive_generic_with_rules "${SLUG_BACKUP_HOME_FROM}" "${SLUG_BACKUP_HOME_TO}" "${SLUG_BACKUP_HOME_IGNORE}" fi else alert_backup_location_not_valid "${SLUG_BACKUP_HOME_TO}" fi else alert_backup_from_location_not_valid "${SLUG_BACKUP_HOME_FROM}" fi fi } ## (C003) - Backup of /srv (TESTED: 2023/12/20) ## Dual backup schema: primary, secondary run_backup_srv() { ## Define slugs: All 3 must exists except BACKUP_*_IGNORE_SLU, which ## defaults to a blank slot when empty SLUG_BACKUP_SRV_FROM="${BACKUP_SRV_FROM}" SLUG_BACKUP_SRV_TO_PRIMARY="${BACKUP_SRV_TO_PRIMARY}" SLUG_BACKUP_SRV_TO_SECONDARY="${BACKUP_SRV_TO_SECONDARY}" SLUG_BACKUP_SRV_IGNORE="${BACKUP_SRV_IGNORE}" ## Check slugs if [ -d "${SLUG_BACKUP_SRV_FROM}" ]; then alert_disk_ops_directory_generic_from "${SLUG_BACKUP_SRV_FROM}" alert_disk_ops_redundant_archive_initialized alert_backup_redundant_location "_TO_PRIMARY" "${SLUG_BACKUP_SRV_TO_PRIMARY}" alert_backup_redundant_location "_TO_SECONDARY" "${SLUG_BACKUP_SRV_TO_SECONDARY}" ## _TO_PRIMARY if [ -d "${SLUG_BACKUP_SRV_TO_PRIMARY}" ]; then alert_disk_ops_directory_slug_located "${SLUG_BACKUP_SRV_TO_PRIMARY}" alert_disk_ops_locate_rules if [ -f "${SLUG_BACKUP_SRV_IGNORE}" ]; then alert_backup_ruleset_defined "${SLUG_BACKUP_SRV_IGNORE}" while read -r SLUG_BACKUP_LOGIC do alert_backup_rule_instance "${SLUG_BACKUP_LOGIC}" done < "$SLUG_BACKUP_SRV_IGNORE" alert_backup_custom_starting "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_PRIMARY}" alert_disk_ops_redundant_archive_starting "srv" "1" archive_generic_with_rules "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_PRIMARY}" "${SLUG_BACKUP_SRV_IGNORE}" else alert_backup_ruleset_not_defined alert_backup_standard_starting "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_PRIMARY}" SLUG_BACKUP_SRV_IGNORE="NONE" alert_disk_ops_redundant_archive_starting "srv" "1" archive_generic_no_rules "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_PRIMARY}" fi else alert_backup_location_not_valid "${SLUG_BACKUP_SRV_TO_PRIMARY}" fi ## _TO_SECONDARY if [ -d "${SLUG_BACKUP_SRV_TO_SECONDARY}" ]; then alert_disk_ops_directory_slug_located "${SLUG_BACKUP_SRV_TO_SECONDARY}" alert_disk_ops_locate_rules if [ -f "${SLUG_BACKUP_SRV_IGNORE}" ]; then alert_backup_ruleset_defined "${SLUG_BACKUP_SRV_IGNORE}" while read -r SLUG_BACKUP_LOGIC do alert_backup_rule_instance "${SLUG_BACKUP_LOGIC}" done < "$SLUG_BACKUP_SRV_IGNORE" alert_backup_custom_starting "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_SECONDARY}" alert_disk_ops_redundant_archive_starting "srv" "2" archive_generic_with_rules "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_SECONDARY}" "${SLUG_BACKUP_SRV_IGNORE}" else alert_backup_ruleset_not_defined alert_backup_standard_starting "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_SECONDARY}" SLUG_BACKUP_SRV_IGNORE="NONE" alert_disk_ops_redundant_archive_starting "srv" "2" archive_generic_no_rules "${SLUG_BACKUP_SRV_FROM}" "${SLUG_BACKUP_SRV_TO_SECONDARY}" fi else alert_backup_location_not_valid "${SLUG_BACKUP_SRV_TO_SECONDARY}" fi else alert_backup_from_location_not_valid "${SLUG_BACKUP_SRV_FROM}" fi } ## (C004) - Backup of /usr (WORKING: 2023/12/20) run_backup_usr() { ## look at /etc/fstab: Anything that's linked to the webserver and not ## needed/that's already backed up elsewhere should be excluded. ## This includes GATEWAY and SYSTEM. if [ -z "${BACKUP_USR_TO}" ]; then alert_backup_to_location_not_valid "${PATH_USR_TO}" else ## Define slugs: Assuming no exclusions SLUG_BACKUP_USR_FROM="${BACKUP_USR_FROM}" SLUG_BACKUP_USR_TO="${BACKUP_USR_TO}" SLUG_BACKUP_USR_IGNORE="NONE" ## Check slugs if [ -d "${SLUG_BACKUP_USR_FROM}" ]; then alert_disk_ops_directory_generic_from "${SLUG_BACKUP_USR_FROM}" if [ -d "${SLUG_BACKUP_USR_TO}" ]; then alert_disk_ops_directory_slug_located "${SLUG_BACKUP_USR_TO}" alert_disk_ops_locate_rules if [ "${SLUG_BACKUP_USR_IGNORE}" = "NONE" ]; then alert_backup_ruleset_not_defined alert_backup_custom_starting "${SLUG_BACKUP_USR_FROM}" "${SLUG_BACKUP_USR_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_USR_FROM}" archive_generic_no_rules "${SLUG_BACKUP_USR_FROM}" "${SLUG_BACKUP_USR_TO}" "${SLUG_BACKUP_USR_IGNORE}" else alert_backup_ruleset_defined "${SLUG_BACKUP_USR_IGNORE}" alert_backup_custom_starting "${SLUG_BACKUP_USR_FROM}" "${SLUG_BACKUP_USR_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_USR_FROM}" archive_generic_with_rules "${SLUG_BACKUP_USR_FROM}" "${SLUG_BACKUP_USR_TO}" "${SLUG_BACKUP_USR_IGNORE}" fi else alert_backup_location_not_valid "${SLUG_BACKUP_USR_TO}" fi else alert_backup_from_location_not_valid "${SLUG_BACKUP_USR_FROM}" fi fi } ## (C005) - Backup of /var (WORKING: 2023/12/20) run_backup_var() { if [ -z "${BACKUP_VAR_TO}" ]; then alert_backup_to_location_not_valid "${PATH_VAR_TO}" else ## Define slugs: Assuming no exclusions SLUG_BACKUP_VAR_FROM="${BACKUP_VAR_FROM}" SLUG_BACKUP_VAR_TO="${BACKUP_VAR_TO}" SLUG_BACKUP_VAR_IGNORE="${BACKUP_VAR_IGNORE}" ## Check slugs if [ -d "${SLUG_BACKUP_VAR_FROM}" ]; then alert_disk_ops_directory_generic_from "${SLUG_BACKUP_VAR_FROM}" if [ -d "${SLUG_BACKUP_VAR_TO}" ]; then alert_disk_ops_directory_slug_located "${SLUG_BACKUP_VAR_TO}" alert_disk_ops_locate_rules if [ "${SLUG_BACKUP_VAR_IGNORE}" = "NONE" ]; then alert_backup_ruleset_not_defined alert_backup_custom_starting "${SLUG_BACKUP_VAR_FROM}" "${SLUG_BACKUP_VAR_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_VAR_FROM}" archive_generic_no_rules "${SLUG_BACKUP_VAR_FROM}" "${SLUG_BACKUP_VAR_TO}" else alert_backup_ruleset_defined "${SLUG_BACKUP_VAR_IGNORE}" alert_backup_standard_starting "${SLUG_BACKUP_VAR_FROM}" "${SLUG_BACKUP_VAR_TO}" alert_disk_ops_initialized_partial "${SLUG_BACKUP_VAR_FROM}" archive_generic_with_rules "${SLUG_BACKUP_VAR_FROM}" "${SLUG_BACKUP_VAR_TO}" "${SLUG_BACKUP_VAR_IGNORE}" fi else alert_backup_location_not_valid "${SLUG_BACKUP_VAR_TO}" fi else alert_backup_from_location_not_valid "${SLUG_BACKUP_VAR_FROM}" fi fi } ############################################################################### ## FAILURES ############################################################################### ## ... list any failures here ############################################################################### ## DEPRECATED: ############################################################################### ## ... list any devices that are set for removal here