{"id":87,"date":"2023-11-28T14:03:43","date_gmt":"2023-11-28T19:03:43","guid":{"rendered":"https:\/\/chrisj.cloud\/?p=87"},"modified":"2023-11-28T14:03:43","modified_gmt":"2023-11-28T19:03:43","slug":"stretching-kubernetes-architecture-across-distributed-private-cloud","status":"publish","type":"post","link":"https:\/\/chrisj.cloud\/index.php\/2023\/11\/28\/stretching-kubernetes-architecture-across-distributed-private-cloud\/","title":{"rendered":"Stretching Kubernetes Architecture across Distributed Private Cloud"},"content":{"rendered":"\n

As a Solutions Architect and after all the years helping my clients select the right architectures for their use cases or requirements I have come up with a certain realization. My primary job is to identify and present a list of trade-offs for anything that I build (for my clients) and make sure I help them select the best option.  If you have heard the expression – \u201dYou can’t have your cake and eat it (too)\u201d, this blog is going to put you in a similar dilemma. I\u2019ll describe four distributed Kubernetes (OpenShift) Architectures and their trade-offs to help you decide which one is right for you. Spoiler alert, there is no single option that is better than the rest.<\/p>\n\n\n\n

Let\u2019s start with some assumptions.<\/p>\n\n\n\n

    \n
  1. My private cloud of choice is OpenStack. Not all of the considerations will translate to other datacenter technologies or public clouds<\/li>\n\n\n\n
  2. Distribution of the resources has its limitations. As a matter of fact etcd that is used by Kubernetes has a single digit to low double digit latency requirement. Other words, in order for us to consider stretching Kubernetes across availability zones, we need to ensure those are not too far apart geographically.<\/li>\n\n\n\n
  3. Some of the characteristics might not be available in the upstream versions of the components used in this analysis and at least have not been tested nor verified.<\/li>\n<\/ol>\n\n\n\n

    Here is the overall architecture:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Characteristics:<\/p>\n\n\n\n

      \n
    • Multiple availability Zones <\/li>\n\n\n\n
    • Same geographical location<\/li>\n\n\n\n
    • L2 stretched Openstack control plane (for now)<\/li>\n\n\n\n
    • Kubernetes cluster per AZ or stretched across AZs<\/li>\n\n\n\n
    • Routed Networking between AZs<\/li>\n\n\n\n
    • Availability of all services across 3 AZs (any one AZ can be killed at any time without disruption of the services)<\/li>\n<\/ul>\n\n\n\n

      Now, let me show you four viable openshift deployment architectures that can stretch over multiple availability zones (to enhance SLAs). Especially the first 2 are my strong recommendation to consider, since they use most automation for its lifecycle.<\/p>\n\n\n\n

      \n\n\n\n

      Architecture #1  – IPI – RPN with small stretched L2 for OCP control<\/strong><\/p>\n\n\n\n

      Implementation:<\/p>\n\n\n\n

      Day 1:<\/strong>
      Install minimal OpenShift (Kubernetes) using automation (IPI) over stretched L2 Provider network (1 master and 1 worker per AZ).<\/p>\n\n\n\n

      Day2:<\/strong>
      Create Dedicated MachineSets per AZ for new workers. Scale-out as needed<\/p>\n\n\n\n

      \n\n\n\n

      Pros:<\/strong><\/p>\n\n\n\n

        \n
      • Full use of Openshift (Kubernetes) installer automation<\/li>\n\n\n\n
      • Worker nodes use Routed Provided Networks (RPN) for workload high performance & availability<\/li>\n\n\n\n
      • Stretched network can be done one-per-cluster or combined as desired<\/li>\n<\/ul>\n\n\n\n

        Cons:<\/strong><\/p>\n\n\n\n

          \n
        • Uses Administrator managed VLAN based networks (operational overhead)<\/li>\n\n\n\n
        • Openshift control plane availability dependant on high availability of the stretched network\u2019s gateways<\/li>\n\n\n\n
        • Openshift Master\/Infra nodes do not use RPNs<\/li>\n\n\n\n
        • For multiple clusters a potential need for additional stretched L2 due to VRRP collision
           <\/li>\n<\/ul>\n\n\n\n
          \"\"<\/figure>\n\n\n\n
          \n\n\n\n

          Architecture #2  – IPI – Overlay network with no-DVR on FIPs<\/strong><\/p>\n\n\n\n

          Implementation:<\/p>\n\n\n\n

          Day 1:<\/strong>
          Install fully blown OpenShift (Kubernetes) using automation (IPI) over an overlay tenant network. The ingress and API handles by a floating IP with disabled DVR<\/p>\n\n\n\n

          Day2:<\/strong>
          No action required. Scale-out as needed<\/p>\n\n\n\n

          \n\n\n\n

          Pros:<\/strong><\/p>\n\n\n\n

            \n
          • Simplest architecture<\/li>\n\n\n\n
          • Only 3 Routable IPs required (min) per Openshift cluster<\/li>\n\n\n\n
          • optional day 2 operation to scale out per AZ workers<\/li>\n\n\n\n
          • No admin work required with every new cluster (no operational overhead)<\/li>\n<\/ul>\n\n\n\n

            Cons:<\/strong><\/p>\n\n\n\n

              \n
            • SNAT for workers without a Floating IP<\/li>\n\n\n\n
            • No external access to most workers<\/li>\n\n\n\n
            • Performance can be affected by double encapsulation 
               <\/li>\n<\/ul>\n\n\n\n
              \"\"<\/figure>\n\n\n\n
              \n\n\n\n

              Architecture #3  – UPI – RPN with external load balancer for control plane<\/strong><\/p>\n\n\n\n

              Implementation:<\/p>\n\n\n\n

              Day 1:<\/strong>
              Install fully blown OpenShift (Kubernetes) using manual configuration (UPI) over Routed Provider Networks with 3rd party load balancer<\/p>\n\n\n\n

              Day2:<\/strong>
              Scaling out can require additional manual configuration<\/p>\n\n\n\n

              \n\n\n\n

              Pros:<\/strong><\/p>\n\n\n\n

                \n
              • No stretched L2 Provider network required<\/li>\n\n\n\n
              • OCP Master\/Infra & Worker nodes use RPNs for workload performance & availability<\/li>\n\n\n\n
              • Easy integration with some enterprise load balancers (example – F5 operator available if desired)<\/li>\n<\/ul>\n\n\n\n

                Cons:<\/strong><\/p>\n\n\n\n

                  \n
                • Significant lifecycle overhead<\/li>\n\n\n\n
                • Inability to create new clusters with automation<\/li>\n\n\n\n
                • 3rd party needs to handle infrastructure load balancing<\/li>\n\n\n\n
                • Load balancing configuration is manual (unless automation is created)
                   <\/li>\n<\/ul>\n\n\n\n
                  \"\"<\/figure>\n\n\n\n
                  \n\n\n\n

                  Architecture #4  – Overlay with 3rd party HAProxy controllers<\/strong><\/p>\n\n\n\n

                  Implementation:<\/p>\n\n\n\n

                  Day 1:<\/strong>
                  Openshift installation is partially automated (IPI) with a load balancing component deployed manually and attached to appropriate overlay and provider networks<\/p>\n\n\n\n

                  Day2:<\/strong>
                  After the most laborious Day1+, the scale-out should have the least friction, but operational overhead of HAProxy VMs is there.<\/p>\n\n\n\n

                  \n\n\n\n

                  Pros:<\/strong><\/p>\n\n\n\n

                    \n
                  • No stretched L2 Provider network Requirement<\/li>\n\n\n\n
                  • DVR available for high availability and performance<\/li>\n<\/ul>\n\n\n\n

                    Cons:<\/strong><\/p>\n\n\n\n

                      \n
                    • Significant lifecycle overhead<\/li>\n\n\n\n
                    • 3rd party integration (HAProxy), outside of vendor support<\/li>\n\n\n\n
                    • Worker nodes use SNAT<\/li>\n\n\n\n
                    • No external access to OCP nodes<\/li>\n\n\n\n
                    • Would require testing
                       <\/li>\n<\/ul>\n\n\n\n
                      \"\"<\/figure>\n\n\n\n
                      \n\n\n\n

                      In summary these are 4 viable architectures that could be used for stretching Kubernetes across multiple availability zones in distributed architecture.
                      You have been warned, there is no one architecture to rule them all. The most tradeoffs are around operational overhead, performance and availability, but today you can\u2019t have it all.<\/p>\n\n\n\n

                      There is however light in a tunnel. The future architectures might be able to take advantage of BGP to automatically distribute OpenShift nodes over RPNs in multiple AZs Day1, without a need of external load balancer. In such architecture the operational overhead could be reduced without sacrificing performance or availability. I\u2019ll leave this topic for another time (when the feature is actually implemented) and another blog post.<\/p>\n","protected":false},"excerpt":{"rendered":"

                      As a Solutions Architect and after all the years helping my clients select the right architectures for their use cases or requirements I have come up with a certain realization. My primary job is t","protected":false},"author":1,"featured_media":88,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[1],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/posts\/87"}],"collection":[{"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/comments?post=87"}],"version-history":[{"count":1,"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/posts\/87\/revisions"}],"predecessor-version":[{"id":94,"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/posts\/87\/revisions\/94"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/media\/88"}],"wp:attachment":[{"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/media?parent=87"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/categories?post=87"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chrisj.cloud\/index.php\/wp-json\/wp\/v2\/tags?post=87"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}