Sunday, July 28, 2019

Comparing many json files

I wrote a tool to compare JSON files , code and documentation are on github.

SAMPLE OUTPUT


  :color:red:file01.json
  :color:red:file02.json
  :color:red:file03.json
  :color:red:file04.json

  :fruit:apple:file01.json
  :fruit:cherry:file02.json
  :fruit:apple:file03.json
  :fruit:MISSING:file04.json

  :inspiration:art:Pablo Picasso:file01.json
  :inspiration:art:Frida Kahlo:file02.json
  :inspiration:art:Pablo Picasso:file03.json
  :inspiration:art:MISSING:file04.json

  :inspiration:music:Dead Kennedys:file01.json
  :inspiration:music:Dead Kennedys:file02.json
  :inspiration:music:Dead Kennedys:file03.json
  :inspiration:music:MISSING:file04.json

  :inspiration:tools:MISSING:file01.json
  :inspiration:tools:["hammer", "rack"]:file02.json
  :inspiration:tools:MISSING:file03.json
  :inspiration:tools:MISSING:file04.json

  :vegetable:MISSING:file01.json
  :vegetable:MISSING:file02.json
  :vegetable:spinach:file03.json
  :vegetable:MISSING:file04.json

SAMPLE INPUT

file01.json

{
   "color": "red",
   "fruit": "apple",
   "inspiration": {"art":"Pablo Picasso","music":"Dead Kennedys"}
  }

file02.json

  {
   "color": "red",
   "fruit": "cherry",
   "inspiration": {"art":"Frida Kahlo","music":"Dead Kennedys","tools":["hammer","rack"]}
  }

file03.json

 {
   "vegetable": "spinach",
   "color": "red",
   "fruit": "apple",
   "inspiration": {"art":"Pablo Picasso","music":"Dead Kennedys"}
  }

file04.json

{
   "color": "red"
  }

Porting python 2 to python 3 (2to3) without unit tests.

The existing guides for porting python 2 to python 3 generally say "...then you run your unit tests and fix any problems you find." This is awkward if you don't have unit tests. While, you will not escape without testing, (perhaps by your angry users in production), you can save little time by:

  1. Before the conversion, compile your code to python 2 , to establish it isn't already broken 
  2. After the conversion, see if you can compile your code to python 3 
  3. After the conversion, run pylint --py3k 

This snippet will compile your code as python2 , it is meant to be called with from a larger python 3 script with something like subprocess.check_output. This snippet will not catch run time errors like “undefined variable” . It will catch compile time errors like “bad indent”


#!/usr/bin/env python
import py_compile, sys
print("compiling for python 2 %s"%(sys.argv[1]))
py_compile.compile(sys.argv[1], doraise=True)

This sub-routine will compile your code as python3


def compile_python3_ok(file):
    try:
        py_compile.compile(file, doraise=True)
    except py_compile.PyCompileError as the_exception:
        msg = "\nERROR FAIL python3 compile %s\n"%(file)
        sys.stderr.write(msg)
        sys.stderr.write("\n"+ str(the_exception))

You can install pylint with


sudo apt install python3-pip
pip3 install pylint
export PATH=/home/$USER/.local/bin:$PATH

This snippit will run pylint with python 3 checkers


#!/us/bin/python3
import subprocess,sys

def run_(cmd): 
    try:
        log_me = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
        print(log_me)
        except subprocess.CalledProcessError as the_exception:
        msg = "ERROR FAIL %s\n"%(cmd)
        sys.stderr.write(msg)
  sys.stderr(the_exception.output)
  
CHECKS_TO_SKIP='check99,check22'
file='/path/to/file'
run_("pylint --py3k --disable=%s %s"%(CHECKS_TO_SKIP,file))

Pylint is kinda like having unit tests, in that you should understand the problems it calls out and either fix them or skip the checks and kinda not in that unlike 100% test coverage, you will miss some problems.

ACKNOWLEDGMENTS

FUTURE

For simplicity, the code examples here are stripped of useful but distracting bits like logging and conversions that 2to3 doesn't handle. Eventually, I'll strip out my employer's proprietary things and put the full script on GitHub. Sooner rather than later, if somebody shows an interest in the fuller code.