Krdytkyu

I have a a file usernames and passwords in JSON format that I want to convert to process.

I have used sed in different commands to process it but what I would like to know is how to lump all three commands into one for the future.

Original format

    { "user.name1" : "hashed_password",

"user.name2" : "hashed_password" }

Desired output

user.name:hashed_password

These are the commands I ran, however I have been unable to chain them together using either piping or simply concatenating them where I get an error, sed: -e expression #1, char 8: unknown option to 's'.

Offending command...

sed -i 's/"//g/s/,/n/g/s//g' input_file 

sed: -e expression #1, char 8: unknown option to `s'

How could the below commands be concatenated into one?

Commands
Remove double quotes

sed -i 's/"//g' input_file

Replace comma with new line

sed -i 's/,/n/g' input_file

Remove whitespace

sed -i 's/s//g input_file

To put multiple sed commands in a single "script", you can use multiple -e flags (which is portable):

sed -i -e 's/"//g' -e 's/,/n/g' -e 's/s//g' input_file

Or the semicolon delimiter (which is not available on all implementations):

sed -i 's/"//g;s/,/n/g;s/s//g' input_file

You'll need to add handling for the braces too - {}...

Having said this, to parse and handle JSON properly, you shouldn't really use sed... perhaps try jq!

jq -r 'keys as $k | "($k):(.[$k])"' input_file

Output:

user.name1:hashed_password

user.name2:hashed_password

• 
  keys as $k will iterate through each key storing its value in $k
  
  
  
  
  • i.e: user.name1, user.name2
    
  
  
  
  


• 
  "($k):(.[$k])" will form a string, substituting in $k and .[$k]
  


• Using -r removes the quotes from output strings (raw mode)

Using sed to process JSON is going to open you up to all sorts of issues... for example, how would you deal with the following (completely valid JSON) input?

{

    "user.name1" :

        "hashed_password",

    "user.name2" :

        "hashed_password"

}

When you're dealing with standardised input like JSON, it's generally better to use a proper parser rather than regex. For example, you'll correctly convert any escape sequences (though that may not be possible with your particular input data!).

Unfortunately, there's no great tools for dealing with JSON within coreutils. Attie's provided jq as a decent option if you are free to install packages.

If you're unable to install additional packages, it's not particularly difficult in Python. Take this script for example:

import json,sys

for (k, v) in json.load(sys.stdin):

    print(k + ":" + v)

Which can be compressed into one line:

cat inputdata | python -c 'import json,sys;print("n".join((k + ":" + v) for (k, v) in json.load(sys.stdin).items()))'

For the simple character deletion you're doing in these sed commands I would instead recommend you use tr, whose sole purpose is to delete, squeeze, or replace individual characters, including newlines (sed is based on regex's, which normally rely on newlines as buffer separators, so using sed to modify newlines is tricky). I think this tr command does everything you're looking for:

cat json_filename | tr -d "{}" 121115" | tr "," "12"

The first tr command deletes all curly braces, double-quotes, spaces, carriage returns (octal 012, ascii 10), tabs (octal 011, ascii 9, and linefeed (octal 015, ascii 13) characters. The second tr command replaces all commas with carriage returns. As long as your JSON file's variable names and values don't contain commas, these commands would allow you to avoid needing a dedicated JSON parser.

That said, if you have a set of sed commands that each work independently, combining them may be most easily accomplished using the "-f" sed option to read the separate commands from a file. You just put the s/.../.../g strings into a file, each string on its own line, then specify that filename after the "-f" option. For example, if the three sed commands you listed are satisfactory, you could put them into a file named "json.convert.sed" that simply contained this:

s/"//g 

s/,/n/g

s/s//g

Then you would invoke sed with this command file using:

sed -f json.convert.sed

That said, these sed commands don't work for me to accomplish what you want, and I'm not sure you can ever get sed to modify newline characters. This is because sed is based on the old "ed" line editor, designed to edit single lines at a time (a "script"-able version of it), so each line of input is "parsed" using newlines as the delimiters, then the line (without the newline) is passed to the editing engine, the editing commands are applied, then the edited line is output with a newline. Then the loop repeats. I've only ever been able to use sed to modify newline by first changing the newlines to some distinct character (that doesn't otherwise appear in the input) using tr. There's no point to using tr this way if all you want to do is delete newlines, since tr will do that for you. But if, for instance, you wanted to convert newlines to semicolons with a trailing space, one way to do that would be:

cat input_file | tr "12" "%" | sed "s/%/; /g"

(newlines are converted to % by tr, then sed converts all % characters to "; " character pairs.)

Sed can handle multi-line editing, but I agree with Attie and Bob, parse json with sed regex can become a nightmare.

sed -nr '/{/ b Load ; d

: Load

/}/ b Edit ; N ; b Load

: Edit ; s/[^"]+"([^"]+)"[^"]+"([^"]+)"(.*)/1:2n3/ ; t Print ; d

: Print ; P ; s/[^n]+n// ; t Edit' <<'eof'

{

    "user.name1" :

        "hashed_password1",

    "user.name2" :

        "hashed_password2"

}

    { "user.name3" : "hashed_password3",

"user.name4" : "hashed_password4" }



{ "user.name5":"hashed_password5"}

eof



user.name1:hashed_password1

user.name2:hashed_password2

user.name3:hashed_password3

user.name4:hashed_password4

user.name5:hashed_password5

You could combine it like this:

sed -i 's/"//g;s/,/n/g;s/s//g' input_file

You forgot to add the removal of {}. So you probably want:

sed -i 's/"//g;s/,/n/g;s/s//g;s/{//g;s/}//g' input_file