Installation steps

Prepare local environment

Linux

Install following tools and packages:

Windows and macOS

Instruction for macOS and Windows is available in OpenStack installation steps.

Download packages

/aws/latest/cb-bosh-deployment-latest.tgz
/aws/latest/cb-cf-deployment-latest.tgz
/aws/latest/cb-concourse-deployment-latest.tgz
/aws/latest/cb-dns-deployment-latest.tgz
/aws/latest/cb-docker-images-latest.tgz
/aws/latest/cb-elk-deployment-latest.tgz
/aws/latest/cb-env-latest.tgz
/aws/latest/cb-installer-latest.tgz
/aws/latest/cb-k8s-deployment-latest.tgz
/aws/latest/cb-opscontrol-latest.tgz
/aws/latest/cb-prometheus-deployment-latest.tgz
/aws/latest/cb-utils-latest.tgz

Unpack packages

tar -zxvf cb-installer-latest.tgz
mkdir cloudboostr
./unpackage.sh . cloudboostr latest

Navigate to cb-opscontrol/terraform/aws.

Prepare SSH keys

First step is to prepare SSH keys that will be used for jumpbox and bosh connection. Also GIT key is required for terraform and scripts to download the required repositories.

If SSH keys were prepared before hand or are being created by external tool like Vault skip this and go to the next step.

# Create SSH keys

export EMAIL=[YOUR_EMAIL_HERE]
mkdir keys
ssh-keygen -t rsa -b 4096 -C "$EMAIL" -f ./keys/jumpbox_devops -N '' -m pem
ssh-keygen -t rsa -b 4096 -C "$EMAIL" -f ./keys/bosh_devops -N '' -m pem
ssh-keygen -t rsa -b 4096 -C "$EMAIL" -f ./keys/git_private_key -N '' -m pem

Prepare resources

Upload keys

Then keys should be uploaded to the sensitive-data container. Filenames and container name can be changed in configuration if required.

aws s3 cp ./keys/bosh_devops         s3://sensitive-data/bosh_devops
aws s3 cp ./keys/bosh_devops.pub     s3://sensitive-data/bosh_devops.pub
aws s3 cp ./keys/jumpbox_devops      s3://sensitive-data/jumpbox_devops
aws s3 cp ./keys/jumpbox_devops.pub  s3://sensitive-data/jumpbox_devops.pub
aws s3 cp ./keys/git_private_key     s3://sensitive-data/git_private_key
aws s3 cp ./keys/git_private_key.pub s3://sensitive-data/git_private_key.pub

Configure terraform

Create file terraform.tfvars based on terraform.tfvars.example.

Example

### AWS PROVIDER ########################################################
# AWS access_key for the account                 [REQUIRED]
aws_access_key = ((aws_access_key))

# AWS secret for the account                     [REQUIRED]
aws_secret_key = ((aws_secret_key))

# AWS region                                     [REQUIRED]
aws_region = ((aws_region))

# List of Availability Zones                     [REQUIRED]
azs = ((azs))

# AWS account ID                                 [REQUIRED]
aws_account_id = ((aws_account_id))

### LOAD BALANCERS ######################################################
concourse_certificate_arn = ((concourse_certificate_arn))
grafana_certificate_arn = ((grafana_certificate_arn))
control_plane_certificate_arn = ((control_plane_certificate_arn))
uaa_certificate_arn = ((uaa_certificate_arn))

### SECURITY GROUPS #####################################################
# IP address or range allowed to access jumpbox 
jumpbox_whitelist_ssh_in = ((jumpbox_whitelist))

### DNS #################################################################
# DNS domain for externaly accessible services   [REQUIRED]
opscontrol_base_domain = ((opscontrol_base_domain))

# Floating IP created manually for the DNS       [REQUIRED]
dns_instance_public_ip = ((dns_instance_public_ip))

### CUSTOM SCRIPTS ######################################################
# Jumpbox VM type                                [REQUIRED]
jumpbox_instance_type = ((jumpbox_instance_type))

# BOSH Director VM type                          [REQUIRED]
bosh_instance_type = ((bosh_instance_type))

# AZ for bosh VMs                                [REQUIRED]
bosh_vm_az = ((bosh_vm_az))

### DEPLOYMENTS CONFIG ##################################################
# Git repository address for the config files    [REQUIRED]
config_repository_url = ((config_repository_url))

# Git repository branch for the config files     [REQUIRED]
config_repository_branch = ((config_repository_branch))

### BACKUP AND RESTORE ##################################################
# Bucket for Prometheus/Grafana backup
prometheus_backup_bucket_name = ((prometheus_backup_bucket_name))

# Bucket for ELK backup (can be shared with Prometheus)
elk_backup_bucket_name = ((elk_backup_bucket_name))

sensitive_data_storage_container_name = ((sensitive_data_storage_container_name))

### PACKAGES ############################################################
# Packages bucket configuration
cb_deployments_package_bucket = ((cb_deployments_package_bucket))
cb_deployments_package_target_cloud = ((cb_deployments_package_target_cloud))
cb_deployments_package_version = ((cb_deployments_package_version))

The full list of available configuration parameters is presented below:

# AWS provider:
- aws_access_key - AWS access_key for the account
- aws_secret_key - AWS secret for the account
- aws_region - AWS region
- azs - List of availability zones. This should be 2 element list!
- aws_account_id - AWS account ID

- public_dns_ip - default = "8.8.8.8"

# LOAD BALANCERS
- concourse_certificate_arn - ARN for concourse certificate uploaded to AWS
- grafana_certificate_arn - ARN for grafana certificate uploaded to AWS
- control_plane_certificate_arn - ARN for control plane certificate uploaded to AWS
- uaa_certificate_arn - ARN for uaa certificate uploaded to AWS

# Existing resources:
- sensitive_data_storage_container_name - Container name for keys
- sensitive_data_git_private_key_filename - GIT SSH private key file name uploaded to the sensitive data container
- sensitive_data_bosh_public_key_filename - BOSH public key file name uploaded to the sensitive data container
- sensitive_data_bosh_private_key_filename - BOSH private key file name uploaded to the sensitive data container
- sensitive_data_jumpbox_public_key_filename - Jumpbox public key file name uploaded to the sensitive data container
- sensitive_data_jumpbox_private_key_filename - Jumpbox private key file name uploaded to the sensitive data container
- prometheus_backup_bucket_name - Prometheus backup bucket
- elk_backup_bucket_name - Elk backup bucket

# Jumpbox:
- jumpbox_whitelist_ssh_in - List of CIDRs from which SSH to jumpbox is allowed
- opscontrol_whitelist_out - List of CIDRs to which (except env) traffic from opscontrol is allowed (e.g. proxy)
- env_cidrs - List of CIDRs of envs to and from which traffic is allowed
- jumpbox_instance_type - AWS VM type that should be used for jumpbox

# Others:
- bosh_instance_type - VM instance type that should be used for BOSH Director
- bosh_vm_az - BOSH Director deployment Availabilty Zone
- env_name - Prefix appended to the resources names
- opscontrol_base_domain - Base domain name to all services in opscontrol
- dns_instance_public_ip - Floating IP created manually for the DNS
- python_alias - Alias for python3 installed in place where `terraform apply` is applied

# Networks:
- network_cidr - Whole network CIDR

- mgmt_subnet_cidr - Management subnet CIDR
- mgmt_gateway_ip - IP for management subnet gateway
- mgmt_reserved_ips - Management reserved IP range

- dmz_subnet_cidrs - DMZ subnet CIDRs per az
- dmz_gateway_ips - IPs for dmz subnet gateway per az
- dmz_reserved_ips - DMZ reserved IP ranges per az

- dns_subnet_cidr - DNS subnet CIDR
- dns_gateway_ip - IP for dns subnet gateway
- dns_reserved_ips - DNS reserved IP range

- telemetry_subnet_cidrs - Telemetry subnet CIDRs per az
- telemetry_gateway_ips - IPs for telemetry subnet gateway per az
- telemetry_reserved_ips - Telemetry reserved IP ranges per az
- telemetry_static_ips - Telemetry static IP ranges per az

- bosh_private_ip - BOSH Director address in mgmt reserved IP range
- consul_private_ip - Consul address in telemetry static IP range


# Repositories:
- config_repository_url - Git URI to the config repository
- config_repository_branch - Git branch name in the config repository

- cb_deployments_package_bucket - Bucket with deployments packages
- cb_deployments_package_target_cloud - Target cloud of deployments packages
- cb_deployments_package_version - Version of deployments packages

# EXTENSIONS:
- extensions_bucket_name - Bucket name for extension ops files
- extensions_bosh_directory - Directory name for bosh extension ops files in extensions bucket
- extensions_bosh_properties - Properties filename for extension ops

- extensions_dns_directory - Directory name for dns extension ops files in extensions bucket
- extensions_dns_properties - Properties filename for extension ops

- extensions_concourse_directory - Directory name for concourse extension ops files in extensions bucket
- extensions_concourse_properties - Properties filename for extension ops

- extensions_elk_directory - Directory name for elk extension ops files in extensions bucket
- extensions_elk_properties - Properties filename for extension ops

- extensions_prometheus_directory - Directory name for prometheus extension ops files in extensions bucket
- extensions_prometheus_properties - Properties filename for extension ops

- extensions_control_plane_directory - Directory name for kubernetes control plane extension ops files in extensions bucket
- extensions_control_plane_properties - Properties filename for extension ops

# UAA identity providers

- oauth_type - Type of identity provider. Should be oauth2.0 or oidc1.0 or empty string
- oauth_idp_alias - OAuth identity provider alias. Used by uaa to identify users authenticated by this provider.
- oauth_idp_name - Human readable name of OAuth identity provider.
- oauth_discovery_url - The OpenID Connect Discovery URL, typically ends with /.well-known/openid-configuration
- oauth_auth_url - The OAuth/OIDC authorization endpoint URL
- oauth_token_url - The OAuth/OIDC token endpoint URL
- oauth_token_key_url - The URL of the token key endpoint which renders a verification key for validating token signatures
- oauth_issuer - The OAuth/OIDC token issuer.
- oauth_user_info_url - The URL which provides user info (this can be blank)
- oauth_link_text - Text to use for the login link to the provider
- oauth_client_id - The client ID which is registered with the external OAuth provider for use by the UAA
- oauth_client_secret - The client secret of the relying party at the external OAuth provider
- oauth_group_name - Name of external provider group that should be mapped in UAA
- oauth_attribute_mappings - An object that contains the mapping fields from external provider to UAA field names (see example configuration)

- saml_idp_alias - A unique alias for the saml provider
- saml_idp_name - Human-readable name for this saml provider
- saml_metadata_location - SAML Metadata - either an XML string or a URL that will deliver XML content
- saml_name_id - The name ID to use for the username, default is urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified
- saml_link_text - Text to use for the login link to the provider
- saml_group_name - Name of external saml provider group that should be mapped in UAA
- saml_attribute_mappings - An object that contains the mapping fields from external provider to UAA field names (see example configuration)

- ldap_idp_name - Human-readable name for this LDAP provider
- ldap_base_url - A URL pointing to the LDAP server, must start with ldap:// or ldaps://
- ldap_mail_attribute_name - The name of the attribute that contains the user's email address, default value is mail
- ldap_user_dn_pattern - One or more patterns used to construct DN. If used then simple bind authentication method is used.
- ldap_user_dn_pattern_delimeter - The delimiter character to break up multiple patterns. Default is semicolon ;
- ldap_search_user_dn - The DN for the LDAP credentials used to search the directory. Required for search authentication methods.
- ldap_search_user_password - Password credentials for the above DN to search the directory. Required for search authentication methods.
- ldap_search_base - Specify only if a part of the directory should be searched, for example dc=test,dc=com. Required for search authentication methods.
- ldap_search_filter - The search filter used for the query. Required for search authentication methods.
- ldap_password_attribute_name - The name of the LDAP attribute that holds the password. If provided then search and compare method is used. Otherwise search and bind will be used.
- ldap_local_password_compare - Set to true if the comparison should be done locally. Required for search and compare method. Default is true
- ldap_password_encoder - A fully qualified Java classname to a password encoder
- ldap_group_search_base - The search base for the group search.
- ldap_group_search_filter - Similar to a user filter, most common is member={0}
- ldap_group_name - Name of LDAP group that should be mapped in UAA

- ssh_allowed_hosts - List of hostnames, separated by space, which has StrictHostKeyChecking set to no

- http_proxy_url - Http proxy url in format http://<user>:<password>@<domain>:<port>
- https_proxy_url - Https proxy url in format http://<user>:<password>@<domain>:<port>
- no_proxy - No proxy commaseparated urls/ips

Configure environments

Configure environments for your purposes. To use default configuration clone the cb-config repository and change mandatory fields for infrastructure, Kubernetes, Cloud Foundry and common for all components.

Address and branch of that repository should be correctly filled in the "Fill the OpsControl configuration" step. Also git private key in sensitive-data container must work for that repository.

Run terraform

Start the deployment operation in directory "terraform" in the opscontrol repository:

terraform init
terraform apply

Terraform backend

Terraform can be configured to use external backend instead of local files. To use that feature follow the guide in the official documentation.

Example

{
    "source_type": "package",
    "package_bucket": "((package_bucket))",
    "package_target_cloud": "aws",
    "package_version": "latest",

    "pipelines": [
        {
            "name": "deploy_bosh",
            "file": "ci/pipelines/aws/deploy-pipeline.yml",
            "vars": []
        },
        {
            "name": "destroy_env",
            "file": "ci/pipelines/aws/destroy-pipeline.yml",
            "vars": []
        },
        {
            "name": "backup_bosh",
            "file": "ci/pipelines/backup-pipeline.yml",
            "vars": [
                {"name": "timer_interval", "value": "24h"}
            ]
        },
        {
            "name": "restore_latest_bosh_backup",
            "file": "ci/pipelines/restore-latest-pipeline.yml",
            "vars": []
        },
        {
            "name": "restore_custom_bosh_backup",
            "file": "ci/pipelines/restore-custom-pipeline.yml",
            "vars": []
        }
    ],

    "vars": [
        {"name": "vpc_id", "value": ""},

        {"name": "aws_region",     "opscontrol_var": "aws_region"},
        {"name": "aws_account_id", "opscontrol_var": "aws_account_id"},
        {"name": "azs",            "value": "((azs))"},

        {"name": "opscontrol_cidr",   "opscontrol_var": "opscontrol_cidr"},
        {"name": "opscontrol_vpc_id", "opscontrol_var": "opscontrol_vpc_id"},

        {"name": "opscontrol_telemetry_route_table_id", "opscontrol_var": "opscontrol_telemetry_route_table_id"},
        {"name": "opscontrol_dmz_route_table_id",       "opscontrol_var": "opscontrol_dmz_route_table_id"},

        {"name": "network_cidr", "value": "10.90.0.0/16"},  

        {"name": "mgmt_subnet_cidr",  "value": "10.90.1.0/26"},
        {"name": "mgmt_gateway_ip",   "value": "10.90.1.1"},
        {"name": "mgmt_reserved_ips", "value": "10.90.1.2-10.90.1.10"},

        {"name": "dmz_subnet_cidr",  "value": "10.90.2.0/26"},
        {"name": "dmz_gateway_ip",   "value": "10.90.2.1"},
        {"name": "dmz_reserved_ips", "value": "10.90.2.2-10.90.2.20"},

        {"name": "services_subnet_cidr",  "value": "10.90.4.0/22"},
        {"name": "services_gateway_ip",   "value": "10.90.4.1"},
        {"name": "services_reserved_ips", "value": "10.90.4.2-10.90.4.20"},

        {"name": "dns_subnet_cidr", "value": "10.90.2.128/26"},
        {"name": "dns_gateway_ip", "value": "10.90.2.129"},
        {"name": "dns_reserved_ips", "value": "10.90.2.130-10.90.2.140"},

        {"name": "cf_subnet_cidr",  "value": "10.90.16.0/22"},
        {"name": "cf_gateway_ip",   "value": "10.90.16.1"},
        {"name": "cf_reserved_ips", "value": "10.90.16.2-10.90.16.20"},

        {"name": "k8s_gateway_ip",   "value": "10.90.32.1"},
        {"name": "k8s_reserved_ips", "value": "10.90.32.2-10.90.32.20"},

        {"name": "k8s_public_subnet_cidr", "value": "10.90.48.0/22"},

        {"name": "jumpbox_instance_type",    "value": "t2.small"},
        {"name": "jumpbox_whitelist_ssh_in", "value": "[0.0.0.0/0]"},
        {"name": "env_whitelist_out",        "value": "[0.0.0.0/0]"},

        {"name": "bosh_private_ip",    "value": "10.90.1.6"},
        {"name": "bosh_instance_type", "value": "t2.medium"},
        {"name": "bosh_director_name", "value": "bosh"},

        {"name": "cf_certificate_arn",  "value": "((cf_certificate_arn))"},
        {"name": "k8s_certificate_arn", "value": "((k8s_certificate_arn))"},

        {"name": "opscontrol_dns_public_ip", "opscontrol_var": "dns_instance_public_ip"},
        {"name": "dns_instance_public_ip", "value": "((dns_instance_public_ip))"},
        {"name": "env_base_domain",        "value": "((env_base_domain))"},

        {"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},
        {"name": "bosh_public_key",    "opscontrol_var": "bosh_public_key"},

        {"name": "extensions_bucket_name",           "value": ""},
        {"name": "extensions_bosh_directory",        "value": "aws/env/cb-bosh-deployment"},
        {"name": "extensions_bosh_properties",       "value": "bosh.properties"},
        {"name": "extensions_dns_directory",         "value": "aws/env/cb-dns-deployment"},
        {"name": "extensions_dns_properties",        "value": "dns.properties"},
        {"name": "extensions_prometheus_directory",  "value": "aws/env/cb-prometheus-deployment"},
        {"name": "extensions_prometheus_properties", "value": "prometheus.properties"}
    ]
}