K8S | The shortest summary every

A quick overview Kacidi is a tool which integrates with your Infrastructure-as-code and protects your infrastructure from human mistakes and security breaches. Kacidi detects issues in your infrastructure-as-code automatically before the deployment . What is Infrastructure as code ? Infrastructure as Code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model, using the same versioning as DevOps team uses for source code. Like the principle that the same source code generates the same binary, an IaC model generates the same environment every time it is applied. IaC is a key DevOps practice and is used in conjunction with  continuous delivery . Infrastructure as Code evolved to solve the problem of  environment drift  in the release pipeline. Without IaC, teams must maintain the settings of individual deployment environments. Over time, each environment becomes a  snowflake , that is, a unique configuration that cannot be reproduced automatically. Inconsistency among environments leads to issues during deployments. With snowflakes, administration and maintenance of infrastructure involves manual processes which were hard to track and contributed to errors. IaC is great, but can brings some issues to your organization … As the pace of changes and requirements for organisational agility just increases, and as DevOps practices like IaC and Immutable Infrastructure are being the new norm, it is crucial that we perform some critical thinking about our existing processes and how to better align security and DevOps. As more people get involved in maintaining and creating infrastructure, we need to keep malicious actors and simple human errors out of the cloud infrastructure somehow, automatically and enforce best practices in the infrastructure write from the beginning - before a breach or a disruption to service happens. Kacidi solves these issues and make IaC safest and fastest Which IaC platform supported by Kacidi ? Terraform CloudFormation K8S GCP Deployment Manager Azure Resource Manager Kacidi Key Features: Best-practices checks - perform best practices and security checks . Drift detection check - prevent conflicts in infrastructure by performing a drift detection before the Merge . Change-set check - summarize the infrastructure change that will be applied as a of result merging the branch. Personalized policy - set a policy per user or group in your Github organisation, Examples : - “Restrict developer to create instance t2.small only”.“ - "Only Admin can edit VPC details”. Automated workflow - create, update and destroy environments automatically according GitOps (Deploy on Merge). 🔹 Getting Started (5 min setu p ) 🔹 Sign-up : https://kacidi.com/sign-up . Integrate Kacidi with one of your DevOps tools (Github, Gitlab, Jenkins, CircleCI, etc). After the integration setup, Kacidi will notify inside your Pull-requests about best-practices issues and conflicts in your infrastructure. If you want to set personalised-policy, open Kacidi-editor and set policies per user / group . Using the personalized-policy you can avoid human errors that add up to unnecessary costs, security breaches and even achitecture changes. Start protect your cloud infrastructure today ⬇ https://kacidi.com/sign-up

Author : Yevgeniy Brikma Update: we took this blog post series, expanded it, and turned it into a book called Terraform: Up & Running ! This is Part 1 of the Comprehensive Guide to Terraform series . In the intro to the series, we discussed why every company should be using infrastructure-as-code (IAC) . In this post, we’re going to discuss why we picked Terraform as our IAC tool of choice. If you search the Internet for “infrastructure-as-code”, it’s pretty easy to come up with a list of the most popular tools: Chef Puppet Ansible SaltStack CloudFormation Terraform What’s not easy is figuring out which one of these you should use. All of these tools can be used to manage infrastructure as code. All of them are open source, backed by large communities of contributors, and work with many different cloud providers (with the notable exception of CloudFormation, which is closed source and AWS-only). All of them offer enterprise support. All of them are well documented, both in terms of official documentation and community resources such as blog posts and StackOverflow questions. So how do you decide? What makes this even harder is that most of the comparisons you find online between these tools do little more than list the general properties of each tool and make it sound like you could be equally successful with any of them. And while that’s technically true, it’s not helpful. It’s a bit like telling a programming newbie that you could be equally successful building a website with PHP, C, or Assembly — a statement that’s technically true, but one that omits a huge amount of information that would be incredibly useful in making a good decision. In this post, we’re going to dive into some very specific reasons for why we picked Terraform over the other IAC tools. As with all technology decisions, it’s a question of trade-offs and priorities, and while your particular priorities may be different than ours, we hope that sharing our thought process will help you make your own decision. Here are the main trade-offs we considered: Configuration Management vs OrchestrationMutable Infrastructure vs Immutable InfrastructureProcedural vs DeclarativeClient/Server Architecture vs Client-Only Architecture Configuration Management vs Orchestration Chef, Puppet, Ansible, and SaltStack are all “configuration management” tools, which means they are designed to install and manage software on existing servers. CloudFormation and Terraform are “orchestration tools”, which means they are designed to provision the servers themselves, leaving the job of configuring those servers to other tools. These two categories are not mutually exclusive, as most configuration management tools can do some degree of provisioning and most orchestration tools can do some degree of configuration management. But the focus on configuration management or orchestration means that some of the tools are going to be a better fit for certain types of tasks. In particular, we’ve found that if you use Docker or Packer , the vast majority of your configuration management needs are already taken care of. With Docker and Packer, you can create images (such as containers or virtual machine images) that have all the software your server needs already installed and configured (for more info on what makes Docker great, see here ). Once you have such an image, all you need is a server to run it. And if all you need to do is provision a bunch of servers, then an orchestration tool like Terraform is typically going to be a better fit than a configuration management tool (here’s an example of how to use Terraform to deploy Docker on AWS ). Mutable Infrastructure vs Immutable Infrastructure Configuration management tools such as Chef, Puppet, Ansible, and SaltStack typically default to a mutable infrastructure paradigm. For example, if you tell Chef to install a new version of OpenSSL, it’ll run the software update on your existing servers and the changes will happen in-place. Over time, as you apply more and more updates, each server builds up a unique history of changes. This often leads to a phenomenon known as configuration drift , where each server becomes slightly different than all the others, leading to subtle configuration bugs that are difficult to diagnose and nearly impossible to reproduce. If you’re using an orchestration tool such as Terraform to deploy machine images created by Docker or Packer, then every “change” is actually a deployment of a new server (just like every “change” to a variable in functional programming actually returns a new variable). For example, to deploy a new version of OpenSSL, you would create a new image using Packer or Docker with the new version of OpenSSL already installed, deploy that image across a set of totally new servers, and then undeploy the old servers. This approach reduces the likelihood of configuration drift bugs, makes it easier to know exactly what software is running on a server, and allows you to trivially deploy any previous version of the software at any time. Of course, it’s possible to force configuration management tools to do immutable deployments too, but it’s not the idiomatic approach for those tools, whereas it’s a natural way to use orchestration tools. Procedural vs Declarative Chef and Ansible encourage a procedural style where you write code that specifies, step-by-step, how to to achieve some desired end state. Terraform, CloudFormation, SaltStack, and Puppet all encourage a more declarative style where you write code that specifies your desired end state, and the IAC tool itself is responsible for figuring out how to achieve that state. For example, let’s say you wanted to deploy 10 servers (“EC2 Instances” in AWS lingo) to run v1 of an app. Here is a simplified example of an Ansible template that does this with a procedural approach: - ec2: count: 10 image: ami-v1 instance_type: t2.micro And here is a simplified example of a Terraform template that does the same thing using a declarative approach: resource "aws_instance" "example" { count = 10 ami = "ami-v1" instance_type = "t2.micro" } Now at the surface, these two approaches may look similar, and when you initially execute them with Ansible or Terraform, they will produce similar results. The interesting thing is what happens when you want to make a change. For example, imagine traffic has gone up and you want to increase the number of servers to 15. With Ansible, the procedural code you wrote earlier is no longer useful; if you just updated the number of servers to 15 and reran that code, it would deploy 15 new servers, giving you 25 total! So instead, you have to be aware of what is already deployed and write a totally new procedural script to add the 5 new servers: - ec2: count: 5 image: ami-v1 instance_type: t2.micro With declarative code, since all you do is declare the end state you want, and Terraform figures out how to get to that end state, Terraform will also be aware of any state it created in the past. Therefore, to deploy 5 more servers, all you have to do is go back to the same Terraform template and update the count from 10 to 15: resource "aws_instance" "example" { count = 15 ami = "ami-v1" instance_type = "t2.micro" } If you executed this template, Terraform would realize it had already created 10 servers and therefore that all it needed to do was create 5 new servers. In fact, before running this template, you can use Terraform’s “plan” command to preview what changes it would make: > terraform plan + aws_instance.example.11 ami: "ami-v1" instance_type: "t2.micro" + aws_instance.example.12 ami: "ami-v1" instance_type: "t2.micro" + aws_instance.example.13 ami: "ami-v1" instance_type: "t2.micro" + aws_instance.example.14 ami: "ami-v1" instance_type: "t2.micro" + aws_instance.example.15 ami: "ami-v1" instance_type: "t2.micro" Plan: 5 to add, 0 to change, 0 to destroy. Now what happens when you want to deploy v2 the service? With the procedural approach, both of your previous Ansible templates are again not useful, so you have to write yet another template to track down the 10 servers you deployed previous (or was it 15 now?) and carefully update each one to the new version. With the declarative approach of Terraform, you go back to the exact same template once again and simply change the ami version number to v2: resource "aws_instance" "example" { count = 15 ami = "ami-v2" instance_type = "t2.micro" } Obviously, the above examples are simplified. Ansible does allow you to use tags to search for existing EC2 instances before deploying new ones (e.g. using the instance_tags and count_tag parameters), but having to manually figure out this sort of logic for every single resource you manage with Ansible, based on each resource’s past history, can be surprisingly complicated (e.g. finding existing instances not only by tag, but also image version, availability zone, etc). This highlights two major problems with procedural IAC tools: When dealing with procedural code, the state of the infrastructure is not fully captured in the code. Reading through the three Ansible templates we created above is not enough to know what’s deployed. You’d also have to know the order in which we applied those templates. Had we applied them in a different order, we might end up with different infrastructure, and that’s not something you can see in the code base itself. In other words, to reason about an Ansible or Chef codebase, you have to know the full history of every change that has ever happened.The reusability of procedural code is inherently limited because you have to manually take into account the current state of the codebase. Since that state is constantly changing, code you used a week ago may no longer be usable because it was designed to modify a state of your infrastructure that no longer exists. As a result, procedural code bases tend to grow large and complicated over time. On the other hand, with the kind of declarative approach used in Terraform, the code always represents the latest state of your infrastructure. At a glance, you can tell what’s currently deployed and how it’s configured, without having to worry about history or timing. This also makes it easy to create reusable code, as you don’t have to manually account for the current state of the world. Instead, you just focus on describing your desired state, and Terraform figures out how to get from one state to the other automatically. As a result, Terraform codebases tend to stay small and easy to understand. Of course, there are downsides to declarative languages too. Without access to a full programming language, your expressive power is limited. For example, some types of infrastructure changes, such as a rolling, zero-downtime deployment, are hard to express in purely declarative terms. Similarly, without the ability to do “logic” (e.g. if-statements, loops), creating generic, reusable code can be tricky (especially in CloudFormation). Fortunately, Terraform provides a number of powerful primitives — such as input variables , output variables , modules , create_before_destroy , count , and interpolation functions  — that make it possible to create clean, configurable, modular code even in a declarative language. We’ll discuss these tools more in Part 4, How to create reusable infrastructure with Terraform modules and Part 5, Terraform tips & tricks: loops, if-statements, and pitfalls . Client/Server Architecture vs Client-Only Architecture Chef, Puppet, and SaltStack all use a client/server architecture by default. The client, which could be a web UI or a CLI tool, is what you use to issue commands (e.g “deploy X”). Those commands go to a server, which is responsible for executing your commands and storing the state of the system. To execute those commands, the server talks to agents, which must be running on every server you want to configure. This has a number of downsides: You have to install and run extra software on every one of your servers.You have to deploy an extra server (or even a cluster of servers for high availability) just for configuration management.You not only have to install this extra software and hardware, but you also have to maintain it, upgrade it, make backups of it, monitor it, and restore it in case of outages.Since the client, server, and agents all need to communicate over the network, you have to open extra ports for them, and configure ways for them to authenticate to each other, all of which increases your surface area to attackers.All of these extra moving parts introduce a large number of new failure modes into your infrastructure. When you get a bug report at 3AM, you’ll have to figure out if it’s a bug in your application code, or your IAC code, or the configuration management client software, or the configuration management agent software, or the configuration management server software, or the ports all those configuration management pieces use to communicate, or the way they authenticate to each other, or… CloudFormation, Ansible, and Terraform, use a client-only architecture. Actually, CloudFormation is also client/server, but AWS handles all the server details so transparently, that as an end user, you only have to think about the client code. The Ansible client works by connecting directly to your servers over SSH. Terraform uses cloud provider APIs to provision infrastructure, so there are no new authentication mechanisms beyond what you’re using with the cloud provider already, and there is no need for direct access to your servers. We found this as the best option in terms of ease-of-use, security, and maintainability. Conclusion Putting it all together, below is a table that shows how the most popular IAC tools stack up: At Gruntwork, what we wanted was an open source, cloud-agnostic orchestration tool that supported immutable infrastructure, a declarative language, and a client-only architecture. From the table above, Terraform is the only tool that meets all of our criteria. Of course, Terraform isn’t perfect. It’s younger and less mature than all the other tools on the list: whereas Puppet came out in 2005, Chef in 2009, SaltStack and CloudFormation in 2011, and Ansible in 2012, Terraform came out just 2 years ago, in 2014. Terraform is still pre 1.0.0 (latest version is 0.7.4), so there is no guarantee of a stable or backwards compatible API. Bugs are relatively common (e.g. there are over 800 open issues with the label “bug”), although the vast majority are harmless eventual consistency issues that go away when you rerun Terraform. There are also some issues with how Terraform stores state, although there are effective solutions for those issues that we will discuss in Part 3: How to manage Terraform state . Despite its drawbacks, we find that Terraform’s strengths far outshine its weaknesses, and that no other IAC tool fits our criteria nearly as well. If Terraform sounds like something that may fit your criteria too, head over to Part 2: An Introduction to Terraform , to learn more. For an expanded version of this blog post series, pick up a copy of the book Terraform: Up & Running . If you need help with Terraform, DevOps practices, or AWS at your company, feel free to reach out to us at Gruntwork .

Introduction Docker is a highly optimized platform for building and running containers on servers. It manage containers in a highly efficient manner and works incredibly well with Ansible. Ansible is an automation tool that aims to ease tasks like configuration management, application deployment and intra-service orchestration. It has a built-in Docker module that integrates with Docker for container management. In this guide, you will build and deploy a docker container by using Ansible. Prerequisites Before you begin this guide you'll need the following: One Debian 8 server set up by following the Initial Server Setup with Debian 8 tutorial , including a sudo non-root user. Docker installed on the server. Ansible 2.4 or higher installed on your local machine by following this How to Install and Configure Ansible on Ubuntu 18.04 . Make sure that your local machine can communicate with the server over SSH. Use SSH key instead of typed password for security reason. Step 1 — Build a docker image with Dockerfile There are some number of way to build a docker image but we recommend that you build image using a definition file called a Dockerfile. First, create a directory on the workstation and a Dockerfile within the created directory. mkdir docker cd docker touch Dockerfile For our example, we are going to build a Docker image that contains a simple web server. Edit Dockerfile with nano and add the following content: # Version: 0.0.1 FROM debian:8.5 MAINTAINER maintainer_name "maintainer_email" RUN apt-get update RUN apt-get install -y nginx RUN echo 'Default page. Nginx is in your container. ' \ >/usr/share/nginx/html/index.html EXPOSE 80 Now that we have a Dockerfile, Ansible can help us to ease the building and the deployment on the server. Step 2 — Edit inventory file Inventory file contains IP addresses or domain names where we want deploy container. Add this file in your project: touch ../hosts Edit the hosts file with nano and add the following content within this file. [webserver] your_server_ip Now, check if your local machine can communicate with the server via Ansible. Type the following command in terminal: ansible webserver -m ping -i ../hosts The output of this command should be: your_server_ip | success >> { "changed": false, "ping": "pong" } That means you can reach your server over SSH and you can use Ansible to configure it. Step 3 — Edit Ansible playbook First, change the working directory to the project root directory and create a new Ansible playbook. The directory layout should look like: docker_project/ main.yml hosts docker/ Dockerfile Next, add a task to Ansible playbook to install docker-py on the Docker host. Ansible’s Docker integration requires you to install this python library. We use pipto install the required python library. So we add also a task that install pip before docker-py. Package installation and Docker require a sudo privilege on the server. You can handle this privilege in Ansible by specifying the sudo user name via remote_userand adding become: yes and become_method: sudo directives. Edit the playbook main.yml with nano and add the following: --- - hosts: webserver remote_user: remote_username become: yes become_method: sudo tasks: - name: Install pip apt: name=python-pip state=present - name: install docker-py package pip: name=docker-py The playbook uses the docker module to build an image. We provide a name for the image, the path to the Dockerfile (in our case, inside the docker directory), and tell Ansible via the state parameter whether the image should be present ,absent , or build. So, let's add the following content to main.yml playbook: ... - name: Build Docker image from Dockerfile docker_image: name: web path: docker state: build With this playbook, you only built the new web image but this image never run. You need to add another task to the playbook for running it. Provide the image name, the path to the Dockerfile and use state paramater to tell Ansible the image should be running(via running). Use the latest image by tagging its name with :latest. The playbook should look like this: --- - hosts: webserver remote_user: remote_username become: yes become_method: sudo tasks: - name: Install pip apt: name=python-pip state=present - name: install docker-py pip: name=docker-py - name: Build Docker image from Dockerfile docker_image: name: web path: docker state: build - name: Running the container docker_container: image: web:latest path: docker state: running Now add a task to check if the new container works fine on the server. This task runs docker ps on the server. Finally, your playbook become: --- - hosts: webserver remote_user: linx become: yes become_method: sudo tasks: - name: Install pip apt: name=python-pip state=present - name: install docker-py pip: name=docker-py - name: Build Docker image from Dockerfile docker_image: name: web path: docker state: build - name: Running the container docker_container: image: web:latest path: docker state: running - name: Check if container is running shell: docker ps Our Ansible playbook is now ready for use. Step 3 — Running the playbook Run the playbook by using the following command on your workstation: ansible-playbook -i hosts main.yml --ask-become-pass You are now prompted to type the remote sudo user password. To check if our web server container works fine, type: curl your_server_ip:your_docker_port This should return the Nginx default page created by the Dockerfile. Default page. Nginx is in your container. Conclusion In this article you built and deployed container by using Docker with Ansible. Now you can automate container deployment with Ansible. You can use the Ansible playbook in this guide to deploy any kind of Docker container on your server. You need only to provide your own Dockerfile.