This mono-repo houses all of the kolla-ansible configuration files for multi-cluster Openstack.

cloud
│
└───us-east-1
│   │   multinode
│   │   ceph-spec.yml
│   └───kolla-config
│       │   globals.yml
│       │   passwords.yml
│       │   ...
│
└───us-west-1
│   │   multinode
│   │   ceph-spec.yml
│   └───kolla-config
│       │   globals.yml
│       │   passwords.yml
│       │   ...
│
└───ceph
    │   bootstrap scripts, keys, etc.

Each directory has a unique ssh key pair that only works for that environment. Reference the gitlab ci for more info (yes it's kind of janky).

Before deploying, you'll need to populate secrets that aren't stored in git:

1. passwords.yml - Run kolla-genpwd or manually add all passwords marked with # TODO
2. ml2_conf.ini - Replace secret = REPLACEME with actual switch secrets for Cumulus devices
3. ceph.conf - Replace rgw_keystone_admin_password = REPLACEME with the actual password
4. certificates/ - Add your TLS certs to star_cloud_paraglide_com.pem
5. openrc files - Add the admin password to any *-openrc.sh files you need

Go to pipelines, choose which step you want to deploy.

You'll be prompted to add variables to kick off the job. DATACENTER defines which folder is selected when the pipeline runner kicks off (somewhat similar to Terraform workspaces).

Example job run: DATACENTER: us-east-1. This will trigger a job for the us-east-1 config.

Below outlines the common CI variables that you'll see. The job names in the pipeline tell you which ones you can use. Ex: "Reconfigure servers with limit" means you can use the LIMIT variable.

Each config folder contains pre-built grafana dashboards. You can see an example of the ceph dashboard here.

Ensure the dashboards reference the correct uids. Node-exporter dashboard needs to ref the correct UID specifically.

To add new dashboards, drop the json file into kolla-config/config/grafana/dashboards.

copy the format-drives.sh and make sure the nvme drives correspond to the ones on the host.

chmod +x format-drives.sh && ./format-drives.sh

To bootstrap a new ceph cluster via cephadm, copy the file in ceph/bootstrap to the target host.

Once ready, run:

sudo ./cephadm_bootstrap.bash -m 172.17.4.4 -n 172.17.150.0/24 -r Squid

Fill out the network details accordingly.

Copy the cat /etc/ceph/ceph.pub from the initial host and add it to each other host.

Run the below command, swap details accordingly. For this to work, you need to make sure the keys are on the right hosts.

ceph orch host add se-c1-5-cpu 172.17.4.5 --labels _admin

To add osds, fill out the ceph-spec.yml in the region folder with appropriate hosts.

Once ready, run:

sudo ceph orch apply -i ceph-spec.yml

This will start OSDs on each host. You can view this with ceph status

Each Openstack services depends on pools. Run this once ceph is healthy:

ceph osd pool create vms 128 128
ceph osd pool create volumes 128 128
ceph osd pool create images 128 128
ceph osd pool create backups 128 128

ceph osd pool application enable vms rbd
ceph osd pool application enable volumes rbd
ceph osd pool application enable images rbd
ceph osd pool application enable backups rbd

Run the following command to create Openstack clients:

./keys.sh

This will save the keyrings to a keys folder. Copy them to the cloud repo of your target.

Ensure the ceph-spec.yml for your region includes rgw with appropriate hosts:

---
service_type: rgw
service_id: openstack.us-east-1.paraglide.com
placement:
  hosts:
    - host1
    - host2
    - host3
spec:
  rgw_frontend_port: 10080

The rgw_frontend_port is important (but arbitrary). We tell Swift/Kolla to map Haproxy to these ports. See ceph_rgw_hosts in globals.yml.

Once set, apply the the ceph update ceph orch apply -i ceph-spec.yml.

Once rgw is enabled, run the following config changes against rgw to ensure swift + keystone are integrated:

ceph config set client.rgw rgw_keystone_url "http://<internal_vip_address>:5000"
ceph config set client.rgw rgw_keystone_api_version "3"
ceph config set client.rgw rgw_keystone_admin_user "ceph_rgw"
ceph config set client.rgw rgw_keystone_admin_password <pass from passwords.yml>
ceph config set client.rgw rgw_keystone_admin_project "admin"
ceph config set client.rgw rgw_keystone_admin_domain "default"
ceph config set client.rgw rgw_enable_apis "s3, swift, admin"
ceph config set client.rgw rgw_swift_account_in_url "true"
ceph config set client.rgw rgw_swift_auth_entry "/auth"
ceph config set client.rgw rgw_swift_url "false"
ceph config set client.rgw rgw_swift_url_prefix "swift"
ceph config set client.rgw rgw_swift_versioning_enabled "true"
ceph config set client.rgw.openstack.<region>.paraglide.com rgw_enable_apis "swift, s3"

Swap out the <> with correct values above. Once set, run:

ceph orch restart rgw.openstack.<region>.paraglide.com

Ensure the ceph_rgw user in openstack has the correct role assignments.

openstack role add --user ceph_rgw --project admin admin

You can ensure the connection works by editing and running the file in rgw.py to test the keystone/swift connections work. You'll need to add credentials to that file first.

python3 rgw.py

CAUTION only do this if needed pls.

ceph mgr module disable cephadm
ceph fsid
cephadm rm-cluster --force --zap-osds --fsid <your-fsid-here>

An outline of of the necessary configurations needed to provision a full cloud:

1. Dedicated bond-data vlans for Ceph monitor daemon + replication, ex -m 172.19.6.3 -n 172.19.8.3/23
   • Ex: vlan3030@bond-data, vlan3030@bond-data
2. Dedicated bond-data vlan for Openstack management traffic.
   • Ex: network_interface: "vlan3050", 172.19.10.3/23
   • Openstack will install managemnt VIPs onto this interface, ex 172.19.10.252/32 172.19.10.253/32 (internal + external)
   • This traffic must be routable from the edge so that we can configure our FQDN (ex: us-east-1.cloud.paraglide.com can have an alias which points to above .253 IP).
3. Dual-physnet configurations.
   • ex: neutron_external_interface: "bond-data,bond-mgmt"
   • The underlying bonds cannot hold static IPs, OVS will take control of these interfaces.
4. Dedicated Ironic dnsmasq range
   • ex below:

   ironic_dnsmasq_dhcp_ranges:
     - range: "172.19.12.100,172.19.12.253,255.255.254.0"
       routers: "172.19.13.254"
   

   • The routers above must be configured appropriately.
5. Dedicated VLAN for ironic dnsmasq
   • ex: ironic_dnsmasq_interface: "vlan3060", matches above dnsmasq range (172.19.12.3/23)
6. Dedicated Ironic Cleaning network
   • ex: ironic_cleaning_network: "US-East-1 OpenStack Cleaning"
   • This must be configured in netbox (or wherever) AND routed properly.
7. Dedicated Octavia (load balancer) network.
   • ex below:

   octavia_amp_network:
     name: lb-mgmt-net
     provider_network_type: vlan
     provider_segmentation_id: 3051
     provider_physical_network: physnet1
     external: false
     shared: false
     subnet:
       name: lb-mgmt-subnet
       cidr: "172.19.16.0/23"
       allocation_pool_start: "172.19.17.222"
       allocation_pool_end: "172.19.17.251"
       gateway_ip: "172.19.17.254"
       enable_dhcp: yes
   

8. Dedicated network vlan ranges for Openstack tenants. You must configure this range for both mgmt + data networks.
   • ex: network_vlan_ranges = physnet1:3020:3099,physnet2:3100:3199
   • physnet1 = bond-data, physnet2 = bond-mgmt
9. Neutron SSH key copied to each switch.
10. Switch information which platform can plug into ml2_conf.ini.
    • ex below:

    [genericswitch:switch-name]
    device_type = netmiko_arista_eos
    ngs_mac_address = fc:bd:67:f3:61:4f
    ip = 172.19.1.21
    username = neutron
    key_file = /var/lib/neutron/.ssh/id_rsa
    ngs_manage_vlans = False
    ngs_max_connections = 5
    

    • ngs_mac_address = management mac address
    • ip = ip of the switch
    • username = username for netmiko login
    • For Cumulus switches, add secret = <switch-secret> (replace REPLACEME in the file)

1. Each host must have /etc/hosts record which points to gitlab (unless we have proper dns set up)
   • ex: 172.17.0.136 gitlab.b52.paraglide.internal
2. Copy registry auth into cat ~/.docker/config.json
3. Ensure inspector IPs match the mgmt interface. ironic_inspector_kernel_cmdline_extras
4. Create a new ssh key for the openstack hosts and add it to CI/CD args.
   • cat ~/.ssh/<region> | base64 -w 0
   • copy the above output into CI/CD as a file arg. This way the runners can communicate with the OS hosts.
5. Copy the ceph keyrings + configs to each respective folder in the cloud. Example.