Container Assets

Container Assets is the name we have given to a set of bash scripts that can be run in a container in order to provide that container with certain attributes

A container can include multiple container assets

Some assets depend on other assets

What Goes In a Container Asset?

Largely the assets combine:

  • Installation scripts
  • Configuration templates
  • Optional extra configuration/installation scripts
  • Required functionality

Listing and Instructions for Container Assets

Note

the listing is generated with this command:

1
2
cd /opt/Projects/snippets-edmondscommerce/Cluster/containerAssets
for f in $(find . -maxdepth 1 -mindepth 1 -type d ); do echo "~~~ $(basename $f)"; echo '```'; tree $f; echo '```'; echo ""; done | sed 's/~~~/###/g'

Pushing container assets to staging containers

To push one container asset to one container:

1
2
3
ssh {cluster1-or-2}
cd /opt/Projects/snippets-edmondscommerce/Cluster/shellscripts/cluster
./pushContainerAssetsToContainer.bash {asset-folder} {container-id} # For example: magento1 154
To push one container asset to all containers:

1
2
3
ssh {cluster1-or-2}
cd /opt/Projects/snippets-edmondscommerce/Cluster/shellscripts/cluster
for containerID in $(pct list | grep jira | cut -d ' ' -f 1); do ./pushContainerAssetsToContainer.bash {asset-folder} $containerID; done

Pushing container assets to local containers

Do the below commands from your host machine

1
2
3
4
cd /opt/Projects/snippets-edmondscommerce/
cd Cluster/shellscripts/desktop/lxc/
ll # you should see pushContainerAssetsToContainer.bash 
bash pushContainerAssetsToContainer.bash {asset-folder} {container-id} # Run As Root!
Now go back to your local container to see if the container asset is now there.

Note

If you are already inside to folder just cd out and back in again if you do not see the asset straight away

Available Container Assets

PHP-FPM_Nginx

This one will install the full web development/staging stack

01 MySQL

This actually depends on the MySQL container asset and simply asserts that mysql has already been installed.

02 FPM

Your choice of PHP version along with PHP-FPM

Nginx

Composer plus some nice tab completion stuff

PHPMy Admin

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
./PHP-FPM_Nginx
└── home
    └── ec
        └── PHP-FPM_Nginx_Install
            ├── _01_mysql.bash
            ├── _02_fpm.bash
            ├── _02_fpm-old-php-versions.bash
            ├── _03_nginx.bash
            ├── _04_config.bash
            ├── _05_composer.bash
            ├── _06_phpMyAdmin.bash
            ├── configTemplates
            │   ├── etc
            │   │   ├── nginx
            │   │   │   ├── conf.d
            │   │   │   │   ├── default.conf
            │   │   │   │   ├── standard.conf.template
            │   │   │   │   └── varnish.conf.template
            │   │   │   ├── includes
            │   │   │   │   ├── gzip.conf
            │   │   │   │   ├── header.conf
            │   │   │   │   ├── maintenance.conf
            │   │   │   │   ├── php-fpm.conf
            │   │   │   │   └── robots_noindex.conf
            │   │   │   └── nginx.conf
            │   │   ├── php.d
            │   │   │   └── opcache.ini
            │   │   └── php-fpm.d
            │   │       └── php-fpm-backend.conf
            │   ├── usr
            │   │   └── lib
            │   │       └── systemd
            │   │           └── system
            │   │               ├── mysqld.service
            │   │               ├── nginx.service
            │   │               └── php-fpm.service
            │   └── var
            │       └── www
            │           └── vhosts
            │               └── default
            │                   └── index.php
            ├── fpm-old
            │   ├── _01_install-phpbrew.bash
            │   ├── _02_get-phpbrew-version.bash
            │   ├── _03_install-php-version.bash
            │   └── README.md
            ├── run.bash
            ├── runOptional_display_errors.bash
            ├── runOptional_installXdebug.bash
            ├── runOptional_setupVhost.bash
            ├── runOptional_toggleXdebug.bash
            └── _top.bash

19 directories, 32 files

bytemarkMigration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
./bytemarkMigration
└── home
    └── ec
        └── bytemarkMigration
            ├── 01_bytemarkSSHAccess.bash
            ├── 02_cloneRepo.bash
            ├── 03_createRemoteOnCluster.bash
            ├── 04_createDBDumpAndImportLocally.bash
            ├── run.bash
            └── _top.inc.bash

3 directories, 6 files

chromeHeadless

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
./chromeHeadless
└── home
    └── ec
        └── ChromeHeadless
            ├── edmondscommerce.pem
            ├── google-chrome.repo
            ├── run.bash
            └── _top.inc.bash

3 directories, 4 files

general

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
./general
├── home
│   └── ec
│       ├── cron
│       │   ├── cronRunner.bash
│       │   └── cronRunner.php
│       ├── updateThisContainer.bash
│       └── yum
│           └── needy-restart.bash
└── root

5 directories, 4 files

gitAccess

1
2
3
4
5
6
7
8
./gitAccess
└── home
    └── ec
        └── gitAccess
            ├── catPubKey.bash
            └── run.bash

3 directories, 2 files

gitReposBackup

1
2
3
4
5
6
7
8
./gitReposBackup
└── home
    └── ec
        └── cron
            ├── _crontab_temp
            └── pullAllRepos.bash

3 directories, 2 files

jira

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
./jira
└── home
    └── ec
        └── jiraInstall
            ├── _01_setupMysql.bash
            ├── _02_downloadInstallJira.bash
            ├── _03_setPublicNginxConfig.bash
            └── installJira.bash

3 directories, 4 files

keyMaster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
./keyMaster
└── home
    └── ec
        ├── createClientCertificates
        │   ├── _create_certificate_authority.bash
        │   ├── _create_client_ssl_certificate.bash
        │   ├── create_edmondscommerce_skeleton_certificate.bash
        │   ├── _create_passfile.bash
        │   ├── _create_root_certificate_authority.bash
        │   ├── run.bash
        │   ├── template_intermediate_openssl.conf
        │   ├── template_root_openssl.conf
        │   ├── _top.inc.bash
        │   └── _variables.bash
        └── createSSLCertificates
            ├── _create_root_ca.bash
            ├── _create_ssl_cert.bash
            ├── run.bash
            └── _variables.bash

4 directories, 14 files

Order of operation

  1. run.bash clientname
  2. _create_client_ssl_certificate.bash
    1. If /etc/ssl/intermediate/{clientname} doesn't exist: _create_certificate_authority.bash
      1. If /etc/ssl/root/ca doesn't exist: _create_root_certificate_authority.bash
        1. If /etc/ssl/root/ca/ exists, abort everything
        2. Creates /etc/ssl/root/ca/
        3. Populates the above with openssl.cnf, certs/, crl/, csr/, newcerts/, private/
        4. Creates private/ca.key and certs/ca.cert
      2. If /etc/ssl/intermediate/{clientname} exists, abort everything
      3. Creates /etc/ssl/intermediate/{clientname}
      4. Populates the above with openssl.cnf, certs/, crl/, csr/, newcerts/, private/
      5. Creates private/intermediate.key, csr/intermediate.csr and certs/intermediate.cert
      6. Saves chain to certs/ca-chain.cert
    2. If /etc/ssl/intermediate/{clientname}/_secret doesn't exist: _create_passfile.bash
      • Creates /etc/ssl/intermediate/{clientname}/_secret/passfile
    3. Creates directory /etc/ssl/intermediate/{clientname}/certs/users/{clientname} and populates with{clientname}.key, {clientname}.csr, {clientname}.crt and {clientname}.p12
  3. If no EC certs exist: create_edmondscommerce_skeleton_certificate.bash

Files touched by the script

  • /etc/ssl/
    • intermediate/{clientname}
      • _secret/passfile
      • certs/
        • intermediate.cert
      • crl/
      • csr/
        • intermediate.csr
      • newcerts/
      • private/
        • intermediate.key
      • openssl.cnf
    • root/ca/
      • certs/
        • ca.cert
      • crl/
      • csr/
      • newcerts/
      • private/
        • ca.key
      • openssl.cnf

Replacing the keyMaster's generated certificates

  1. ssh cluster1
  2. Backup the keyMaster container in case of problems
  3. pct enter the keyMaster
  4. vim /etc/ssl/root/ca/index.txt
  5. Search for your client, looking for a CommonName like ...O=Edmonds Commerce Limited/CN=clientname...
  6. Remove the line with the CommonName matching your client
  7. rm -rf /etc/ssl/intermediate/{clientname}
  8. exit back to cluster1
  9. rm -rf /etc/ssl/intermediate/{clientname} on cluster1 too

You should then be clear to recreate your certificates using the _clientCerts.bash clientname

magento2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
./magento2
└── home
    └── ec
        └── magento2
            ├── _01_NginxVarnishSetup.bash
            ├── _02_Magento2InstallAsUser_ec.bash
            ├── _03_Magento2PostInstallAsUser_ec.bash
            ├── _04_NginxVarnishStart.bash
            ├── _05_CodingStandards.bash
            ├── configTemplates
            │   └── etc
            │       ├── nginx
            │       │   ├── conf.d
            │       │   │   └── magento2.conf.template
            │       │   └── includes
            │       │       └── magento2.conf
            │       └── varnish
            │           └── default.vcl
            ├── dataMigrationToolAndProcess
            │   ├── _00_parseLogAndUpdateConfigXml.php
            │   ├── _01_parseLogAndUpdateMapXml.php
            │   ├── _02_parseLogAndUpdateClassMapXml.php
            │   ├── _03_postImportTasks.bash
            │   ├── _04_parseMoveXmlAndUpdateMapXml.php
            │   ├── preRun
            │   │   ├── _01_getM1ReferenceDbAndGenWhiteList.bash
            │   │   ├── _02_downloadDatabase.bash
            │   │   ├── _03_importDatabase.bash
            │   │   ├── _04_fixKnownIssues.bash
            │   │   ├── _05_downloadMedia.bash
            │   │   └── run.bash
            │   ├── run.bash
            │   └── _top.inc.php
            ├── optional_dropAndReinstallDb.bash
            ├── optional_elasticSearch.bash
            ├── optional_GitRepo.bash
            ├── optional_installSampleData.bash
            ├── run.bash
            └── _top.inc.bash

11 directories, 27 files

mysql

1
2
3
4
5
6
7
8
9
./mysql
└── home
    └── ec
        └── mysqlInstall
            ├── 01_Install.bash
            ├── 02_Secure.bash
            └── installMysql.bash

3 directories, 3 files

mysqlBackups

1
2
3
4
5
6
7
8
9
./mysqlBackups
└── home
    └── ec
        └── mysqlBackups
            ├── daily.bash
            ├── hourly.bash
            └── _top.bash

3 directories, 3 files

publicNginx

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
./publicNginx
├── etc
│   └── nginx
│       ├── conf.d
│       │   └── default.conf
│       ├── includes
│       │   ├── jira_server.conf
│       │   ├── proxy_pass.conf
│       │   ├── robots_noindex.conf
│       │   └── ssl_hardening.conf
│       └── nginx.conf
└── root
    ├── createVhost
    │   ├── _letsencrypt.bash
    │   ├── _setup_client_vhost.bash
    │   ├── setup_docs_vhost.bash
    │   ├── setup_jira_vhost.bash
    │   ├── setup_staging_vhost.bash
    │   ├── template_client_index.html
    │   ├── template_client_vhost.conf
    │   ├── template_docs_vhost.conf
    │   ├── template_jira_vhost.conf
    │   ├── template_port80_redirect_vhost.conf
    │   ├── template_staging_vhost.conf
    │   └── _top.inc.bash
    └── nginxInstall
        ├── _installLetsEncrypt.bash
        ├── installNginx.bash
        └── installNginxWithPagespeed.bash

7 directories, 21 files

sphinxSearch

Sphinx is a full-text search engine.

Docs

sphinxSearch/ └── home └── ec └── sphinxSearch └── run.bash

3 directories, 1 file

elasticSearch

1
2
3
4
5
6
7
./elasticSearch
└── home
    └── ec
        └── elasticSearch
            └── run.bash

3 directories, 1 file

jiraShell

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
./jiraShell
└── home
    └── ec
        └── jiraShell
            ├── env
            ├── jiraShell.php
            ├── jiraShellQueueFlush.php
            ├── jiraShellTest.php
            └── run.bash

3 directories, 5 files

documentation

1
2
3
4
5
6
7
./documentation
└── home
    └── ec
        └── documentation
            └── run.bash

3 directories, 1 file

magento1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
./magento1
└── home
    └── ec
        └── magento1
            ├── _magerunAddons.bash
            ├── _magerun.bash
            ├── _modman.bash
            ├── moduleDevInit
            │   ├── _01_installMagento.bash
            │   ├── _02_createModule.bash
            │   ├── _03_gitRepo.bash
            │   └── run.bash
            └── run.bash

4 directories, 8 files

otp

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
./otp
└── root
    ├── catEncryptedSecret
    ├── createEncryptedSecret
    ├── getCode
    ├── info
    ├── projects
    └── updateEncryptionPass

1 directory, 6 files

openVPN

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
./openVPN
└── root
    └── openVPN
        ├── _01_installOpenVpn.bash
        ├── _02_generateKeys.bash
        ├── _03_configureIptables.bash
        ├── _04_routing.bash
        ├── _05_generateConfigFile.bash
        ├── configToCopy
        │   ├── etc
        │   │   └── systemd
        │   │       └── system
        │   │           └── openvpn@server.service
        │   └── sbin
        │       └── ifup-local
        ├── _createTunnel.bash
        ├── README.md
        ├── run.bash
        └── _top.bash

7 directories, 11 files

mysqlBeast

1
2
3
4
5
6
7
8
9
./mysqlBeast
└── home
    └── ec
        └── mysqlBeast
            ├── optional_createDatabase.bash
            ├── run.bash
            └── _top.inc.bash

3 directories, 3 files

mkDocs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
./mkDocs
└── home
    └── ec
        └── mkDocs
            ├── _01_installPython_Pip_MkDocs.bash
            ├── _02_createProject.bash
            ├── _03_gitRepo.bash
            ├── _04_nginx.bash
            ├── _05_bashrc.bash
            ├── bashrc.inc.bash
            ├── configTemplates
            │   ├── etc
            │   │   └── nginx
            │   │       ├── conf.d
            │   │       │   └── mkdocs.conf
            │   │       ├── includes
            │   │       │   ├── gzip.conf
            │   │       │   ├── header.conf
            │   │       │   ├── maintenance.conf
            │   │       │   └── robots_noindex.conf
            │   │       └── nginx.conf
            │   └── usr
            │       └── lib
            │           └── systemd
            │               └── system
            │                   └── nginx.service
            ├── run.bash
            ├── templateDocs
            │   ├── docs
            │   │   ├── Client-Infrastructure
            │   │   │   ├── Hosting
            │   │   │   │   └── Live-Server.md
            │   │   │   └── People.md
            │   │   ├── Client-Software
            │   │   │   ├── Magento.md
            │   │   │   ├── Third-Party-Integrations.md
            │   │   │   └── Wordpress.md
            │   │   ├── Edmonds-Infrastructure
            │   │   │   ├── Containers.md
            │   │   │   └── Git-Repositories.md
            │   │   ├── images
            │   │   │   ├── eclogo.png
            │   │   │   ├── ec-square150.png
            │   │   │   └── logo.svg
            │   │   ├── index.md
            │   │   ├── Jira
            │   │   │   ├── Access.md
            │   │   │   ├── Creating-Tickets.md
            │   │   │   ├── Home-Screen.md
            │   │   │   ├── Kanban.md
            │   │   │   └── Projects.md
            │   │   └── stylesheets
            │   │       └── extra.css
            │   ├── images
            │   │   └── logo.svg
            │   ├── mkdocs.yml
            │   └── themeOverrides
            │       └── partials
            │           └── footer.html
            └── _top.inc.bash

24 directories, 35 files

containerStartBroker

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
./containerStartBroker
└── home
    └── ec
        └── containerStartBroker
            ├── _01_installPhp.bash
            ├── _02_installNginx.bash
            ├── _03_sshConfig.bash
            ├── _04_config.bash
            ├── bin
            │   └── startContainer.bash
            ├── catPubKey.bash
            ├── configTemplates
            │   ├── etc
            │   │   ├── nginx
            │   │   │   ├── conf.d
            │   │   │   │   └── startupBroker.conf
            │   │   │   ├── includes
            │   │   │   │   ├── gzip.conf
            │   │   │   │   ├── header.conf
            │   │   │   │   ├── maintenance.conf
            │   │   │   │   ├── php-fpm.conf
            │   │   │   │   └── robots_noindex.conf
            │   │   │   └── nginx.conf
            │   │   ├── php.d
            │   │   │   └── opcache.ini
            │   │   └── php-fpm.d
            │   │       └── php-fpm-backend.conf
            │   ├── usr
            │   │   └── lib
            │   │       └── systemd
            │   │           └── system
            │   │               ├── nginx.service
            │   │               └── php-fpm.service
            │   └── var
            │       └── www
            │           └── vhosts
            │               └── startupBroker
            │                   └── index.php
            ├── run.bash
            └── _top.inc.bash

19 directories, 20 files

ciTools

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
./ciTools
└── home
    └── ec
        └── ciTools
            ├── _01_InstallPhing.bash
            ├── _02_CreateAVhost.bash
            ├── _03_CreateConfigFiles.bash
            ├── _10_UsageInstructions.bash
            ├── configTemplates
            │   └── etc
            │       └── nginx
            │           ├── conf.d
            │           │   ├── magento2.conf.template
            │           │   └── phpci.conf.template
            │           └── includes
            │               └── magento2.conf
            ├── phing
            │   └── build.xml
            ├── run.bash
            └── _top.inc.bash

9 directories, 10 files