Posts

A collection of thoughts on engineering, AI, and systems.

Writing BDD tests for Terraform Code Using Terratest

Terratest is a popular library for testing Terraform code. Testing Infrastructure As Code (IAC) is not as widespread as it should be. The reasons are multi-fold, ranging from developer’s attitude towards testing to the difficulty of writing unit tests because of inherent side effects of IAC. Nevertheless, testing is no less important, in particular under these scenarios:

  1. When your module gets complicated, with medium to complex behaviour logic
  2. When your module makes underlying assumptions of external dependencies (such as AWS SCPs at Organization level permitting certain actions)

In this post, we will take a look at using Terratest to test Terraform code. A typical Terratest testing pattern involves:

  1. Deploying real infrastructure in real environment
  2. Asserting that the deployed resources behaves as expected
  3. Undeploy everything at the end of the test.

Behavior Driven Test (BDD) uses examples to describe the behavior of a system. It serves the dual purpose of testing the code and documenting it at the same time. Terratest is not a BDD testing framework, however it is possible to write BDD tests that executes Terratest code. In a later section of this post, we will see how this can be achieved using Godog which is a Go BDD testing library.

February 2, 2021

GPT-3 and prospects of Artificial General Intelligence

Last year OpenAI released the Generative Pre-trained Transformer 2 (GPT-2) model. GPT-2 was a language model with 1.5 billion parameters, trained on 8 million web pages. It generated quite a buzz as it could generate coherent text, comprehend paragraphs, answer questions, and summarize text and do all sorts of smart stuff… all without any task-specific learning. OpenAI even deemed the model too dangerous to release but eventually ended up releasing them.

In May 2020, OpenAI released their follow-up GPT-3 model which took the game several notches higher. They trained it with 175 billion parameters, using close to half-a-trillion tokens. The model and its weights alone would take up 300GB VRAM. This is a drastic increase in scale and complexity, anyway you look at it. So what can a huge model like this achieve and why has it reinvigorated the talks ?

GPT-3 Training Size

September 20, 2020

Using OpenCV object detection to keep kids away from TV

Give me a dozen healthy infants, well formed, and my own specified world to bring them up in and I’ll guarantee to take any one at random and train him to become any type of specialist I might select—doctor, lawyer, artist, merchant-chief and yes, even beggar-man thief, regardless of his talents, penchants, tendencies, abilities, vocations, and race of his ancestors.

This was John Watson, one of the founders of Behaviorism, writing around 1925. He believed that human behavior is completely malleable and that it can be shaped into anything given the right environment. While I don’t harbor any grand objectives or sinister experiments like Watson did, I do hope to be able to teach my kids good habits using controlled environments. For instance, my two year old kids started developing the habit of getting too close to the TV. I didn’t want to use force or impose restrictions on them, so I thought I could use technology to discourage them from getting too close to TV.

March 2, 2019

Visualizing air routes for major airports using Spark and Matplotlib

In 2nd Century AD, Hellenic Cartographer Ptolemy was beset with an arbitrary choice of whether his maps should have north on the top or any other direction. Based in Alexandria, he reasoned that all population centers and places of importance lie to the north and would be convenient for study if they were in the upper right corner of the map. This arbitrary choice had long, unintended repercussions for mankind such as Australia being considered “Down under” or even our solar system to be perceived as rotating in counter-clockwise direction. Who would have thought that the stroke of a cartographer carried celestial importance!

January 5, 2019

Book Review - Security Automation with Ansible 2

Security is a huge, complex, rapidly changing field. Advancements in infrastructure hosting, development methodologies has had the most impact on this domain. Thanks to automation, instances are spawned and deleted in a matter of second. Continuous development/Continuous Integration means that an average lifetime of a block of code is ever decreasing. Code review and vulnerability assessments based on static code and IP are hardly affordable at current rate of change. At the same time, the rate of proliferation of technology has seen comparable increase in risk vectors, vulnerabilities and attack methodologies. To keep up with this pace, automation in security operations has become more important than ever.

May 20, 2018

Predicting the playing role of a cricketer using Machine Learning (Part 1)

In this project, we will apply Machine Learning techniques to predict whether a particular cricket player is a batsman or bowler based on his career stats. First we will use Deep Neural Networks (DNN) model and later compare the results with a simpler classifier algorithm such as Random Forest Classifier.

April 23, 2018

Introduction to OpenStack Networking for Network Engineers

This post is a gentle introduction to networking with Openstack using the Neutron module. Being an introduction, we will not focus on setting up OpenStack from scratch. Instead we will familiarize ourselves with core concepts of Neutron and common administrative tasks. We will use the latest release of Openstack, Queens.

March 22, 2018