json stream filter documentation
This commit is contained in:
parent
90693c18d6
commit
c2f1c7c03f
96
Project/Tools/json_stream_filter/README.md
Normal file
96
Project/Tools/json_stream_filter/README.md
Normal file
@ -0,0 +1,96 @@
|
|||||||
|
# JSON stream filter
|
||||||
|
|
||||||
|
a little tool for performing regex operations on string-values in json files (or streams)
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
|
## Basics
|
||||||
|
|
||||||
|
let this be an example set:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "obj_1",
|
||||||
|
"key1": "some example text! 1234",
|
||||||
|
"key2": "another example"
|
||||||
|
}
|
||||||
|
{
|
||||||
|
"id": "obj_2",
|
||||||
|
"key1": "...",
|
||||||
|
"key2": "..."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
an example regex operation replacing all numbers in all `key1`-fields with the string `NUMBER` and storing all matches in a new field called `found_numbers` can be defined as:
|
||||||
|
|
||||||
|
```
|
||||||
|
;[0-9]+;key1;NUMBER;found_numbers
|
||||||
|
```
|
||||||
|
|
||||||
|
the first character (here `;`) is used as separater for the following fields. It can be any unicode character. The following fields in detail:
|
||||||
|
|
||||||
|
* `[0-9]+`: the regex expression which is used for finding matches. In this case at least one character in the range 0-9
|
||||||
|
* `key1`: the key on which the expression is performed. All other keys are ignored
|
||||||
|
* `NUMBER` the string to replace found matches. Can be any string (also an empty string) or regular expression (which is accepted by pythons `re` library for a substitution).
|
||||||
|
* **NOTE**: use `null` here for not doing any substitution
|
||||||
|
* `found_numbers`: key name for storing found matches as a list there. Set to an empty string for not storing the matches
|
||||||
|
|
||||||
|
the result for the example:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "obj_1",
|
||||||
|
"key1": "some example text! NUMBER",
|
||||||
|
"key2": "another example",
|
||||||
|
"found_numbers": [
|
||||||
|
"1234"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
{
|
||||||
|
"id": "obj_2",
|
||||||
|
"key1": "...",
|
||||||
|
"key2": "...",
|
||||||
|
"found_numbers": []
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
|
## Command line interface
|
||||||
|
|
||||||
|
just run the python file with every regex operation as own argument. Output is written to stdout, progress information and errors to stderr. Input will be collected by stdin.
|
||||||
|
|
||||||
|
Assuming the example file above is stored in `example.json` and we can store the result to `result.json` by doing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat example.json | json_stream_filter.py ";[0-9]+;key1;NUMBER;found_numbers" > result.json
|
||||||
|
```
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
|
## using as a python module
|
||||||
|
|
||||||
|
the file can regularly be imported. It contains two classes:
|
||||||
|
|
||||||
|
* `regex_tuple`: for handling the regex operations (like mentioned above). A tuple can be initialized either as a string like above:
|
||||||
|
|
||||||
|
* `regex_tuple.create_from_string(s)`
|
||||||
|
|
||||||
|
or by passing the already separated fields to the constructor:
|
||||||
|
|
||||||
|
* `regex_tuple(regexp_str, keys_to_process, replace_str = None, key_to_store_match = None)`
|
||||||
|
* **NOTE**: `keys_to_process` has to be a list of keys (for future modifications), but so far only one key is supported, so at the moment this should be initialized as a list of one key
|
||||||
|
|
||||||
|
the second class is the file processor:
|
||||||
|
|
||||||
|
* `json_streamer`: is initialized the following way:
|
||||||
|
|
||||||
|
* `json_streamer(output_keys, regex_tuple_list, json_indent=JSON_INDENT)`
|
||||||
|
* `output_keys`: ignored so far. will be used for just filtering out by key names
|
||||||
|
* `regex_tuple_list` list of used `regex_tuple` objects
|
||||||
|
* `json_indent` indent of formatted json output. Set to `2` by default
|
||||||
|
|
||||||
|
for starting the main filter process:
|
||||||
|
|
||||||
|
* `main_stream(stream_input=sys.stdin, stream_output=sys.stdout, stream_error=sys.stderr)`
|
||||||
|
* by default it is using the default system inputs/outputs, but any file object can be set as input/output parameter
|
Loading…
Reference in New Issue
Block a user