Sunday, July 28, 2019

Comparing many json files

I wrote a tool to compare JSON files , code and documentation are on github.

SAMPLE OUTPUT


  :color:red:file01.json
  :color:red:file02.json
  :color:red:file03.json
  :color:red:file04.json

  :fruit:apple:file01.json
  :fruit:cherry:file02.json
  :fruit:apple:file03.json
  :fruit:MISSING:file04.json

  :inspiration:art:Pablo Picasso:file01.json
  :inspiration:art:Frida Kahlo:file02.json
  :inspiration:art:Pablo Picasso:file03.json
  :inspiration:art:MISSING:file04.json

  :inspiration:music:Dead Kennedys:file01.json
  :inspiration:music:Dead Kennedys:file02.json
  :inspiration:music:Dead Kennedys:file03.json
  :inspiration:music:MISSING:file04.json

  :inspiration:tools:MISSING:file01.json
  :inspiration:tools:["hammer", "rack"]:file02.json
  :inspiration:tools:MISSING:file03.json
  :inspiration:tools:MISSING:file04.json

  :vegetable:MISSING:file01.json
  :vegetable:MISSING:file02.json
  :vegetable:spinach:file03.json
  :vegetable:MISSING:file04.json

SAMPLE INPUT

file01.json

{
   "color": "red",
   "fruit": "apple",
   "inspiration": {"art":"Pablo Picasso","music":"Dead Kennedys"}
  }

file02.json

  {
   "color": "red",
   "fruit": "cherry",
   "inspiration": {"art":"Frida Kahlo","music":"Dead Kennedys","tools":["hammer","rack"]}
  }

file03.json

 {
   "vegetable": "spinach",
   "color": "red",
   "fruit": "apple",
   "inspiration": {"art":"Pablo Picasso","music":"Dead Kennedys"}
  }

file04.json

{
   "color": "red"
  }

Porting python 2 to python 3 (2to3) without unit tests.

The existing guides for porting python 2 to python 3 generally say "...then you run your unit tests and fix any problems you find." This is awkward if you don't have unit tests. While, you will not escape without testing, (perhaps by your angry users in production), you can save little time by:

  1. Before the conversion, compile your code to python 2 , to establish it isn't already broken 
  2. After the conversion, see if you can compile your code to python 3 
  3. After the conversion, run pylint --py3k 

This snippet will compile your code as python2 , it is meant to be called with from a larger python 3 script with something like subprocess.check_output. This snippet will not catch run time errors like “undefined variable” . It will catch compile time errors like “bad indent”


#!/usr/bin/env python
import py_compile, sys
print("compiling for python 2 %s"%(sys.argv[1]))
py_compile.compile(sys.argv[1], doraise=True)

This sub-routine will compile your code as python3


def compile_python3_ok(file):
    try:
        py_compile.compile(file, doraise=True)
    except py_compile.PyCompileError as the_exception:
        msg = "\nERROR FAIL python3 compile %s\n"%(file)
        sys.stderr.write(msg)
        sys.stderr.write("\n"+ str(the_exception))

You can install pylint with


sudo apt install python3-pip
pip3 install pylint
export PATH=/home/$USER/.local/bin:$PATH

This snippit will run pylint with python 3 checkers


#!/us/bin/python3
import subprocess,sys

def run_(cmd): 
    try:
        log_me = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
        print(log_me)
        except subprocess.CalledProcessError as the_exception:
        msg = "ERROR FAIL %s\n"%(cmd)
        sys.stderr.write(msg)
  sys.stderr(the_exception.output)
  
CHECKS_TO_SKIP='check99,check22'
file='/path/to/file'
run_("pylint --py3k --disable=%s %s"%(CHECKS_TO_SKIP,file))

Pylint is kinda like having unit tests, in that you should understand the problems it calls out and either fix them or skip the checks and kinda not in that unlike 100% test coverage, you will miss some problems.

ACKNOWLEDGMENTS

FUTURE

For simplicity, the code examples here are stripped of useful but distracting bits like logging and conversions that 2to3 doesn't handle. Eventually, I'll strip out my employer's proprietary things and put the full script on GitHub. Sooner rather than later, if somebody shows an interest in the fuller code.

Friday, June 21, 2019

SOLVED restore chef nodes, clients, vaults from knife-ec-backup


RESTORE A CHEF VAULT ITEM

A vault item is made of two data bags.

   Data bag 1 is: VAULTNAME
   Data bag 2 is:  VAULTNAME_KEYS.

Restore the data bags and you restore the vault item.

Restore  just (1) data bag and you break the entire vault so nobody else can use it until you restore the other data bag or delete your first data bag. The general process is:
  1. Make a copy of the chef backup 
  2.  cd to the directory corresponding to the organization and vault you want to restore
  3.  run the knife commands to restore the data bags
cd ~/backup_dir/organizations/your-org-dev/data_bags/certs/
knife data bag from file certs root_cert_wrtc_your-orgdev_com.json --config-file ~/.chef/knife.rb
knife data bag from file certs root_cert_wrtc_your-orgdev_com_keys.json --config-file ~/.chef/knife.rb

RESTORE A CHEF CLIENT


The general process is:
  1. Install the 'jq' JSON manipulation tool
  2. Make a copy of the chef backup 
  3. cd to the directory corresponding to the organization you want to restore a client to
  4. Extract the public key from the backup file
  5. Run the knife client command to create the client

sudo apt install jq
cd ~/backup_dir/organizations/your-org-dev/clients
jq .public_key .json > client-one.example.com.public_key
sed -i 's/\\n/\n/g' client-one.example.com.public_key
knife client create client-one.example.com --public-key client-one.example.com.public_key

NOTE: There is no obvious way to run 'knife client create' without it popping up an editor

RESTORE A CHEF NODE


The general process is:
  1. Make a copy of the chef backup 
  2. cd to the directory corresponding to the organization you want to restore a node to
  3. Run the knife node command to re-create the node
  cd ~/backup_dir/organizations/your-org-dev/nodes
  knife node from file EDV_omacneil_1558372161_AWS.fmr.com