Improve visualization branch + Feature comparing by kntrain · Pull Request #50 · JohannesEbke/aws_list_all

kntrain · 2021-04-06T03:03:31Z

No description provided.

Bring Python to 3.9 in Travis. Thanks!

Boto3 update 1.16.24

-replaced run_raw call with file reference in wrapper

-moved resources execution to apply_filter -added command line option for resource filters

…run without credentials, as it will run very slowly.

-renamed ListingFile -> FilteredListing -moved non-default filters to fixing_filter.py

- added print-html from old branch - added count number

- moved most html-printing functionality to new py-file

- fixed html-file call from directory

- Combine title and searchbox in header - Move timestamps to footer

-Remove stray comments -Remove convert_unfilterList -Add error-hadling to JSON-dependent resources-block -Adjust resources-dependent query functions

-Finish display of comparison in HTML -Add display of resource IDs for found resources -Change acquire_listing return value to ResultListing object -Refactor do_list_files into info summary and resource ID search

-Fix comparing ResultListing-functions -Add JSON-functions to FIlteredListing -Save dir + unfilter in JSON-files -Delete stray comments

This list of patches splits out the raw listing from the filtered listings, and should allow better handling of different kinds of errors in the long term. In the short term, it adds the possibility to "unfilter" the automatically applied filters that until now were baked in.

JohannesEbke

Thanks, looking almost ready!

JohannesEbke · 2021-04-13T14:12:02Z

aws_list_all/__main__.py

+    viewhtml.add_argument('-p', '--parallel', default=32, type=int, help='Number of request to do in parallel')
+    viewhtml.add_argument('-d', '--directory', default='.', help='Directory to save result listings to')
+    viewhtml.add_argument('-v', '--verbose', action='count', help='Print detailed info during run')
+    viewhtml.add_argument('-c', '--profile', help='Use a specific .aws/credentials profile.')


Shouldn't viewhtml be more similar to "show"? This looks like it's doing all the querying again.

JohannesEbke · 2021-04-13T14:12:30Z

aws_list_all/__main__.py

+
+        new = 2
+        url = os.getcwd() + "/compare.html"
+        webbrowser.open(url,new=new)


This section should go into its own module

JohannesEbke · 2021-04-13T14:14:28Z

aws_list_all/generate_html.py

+    print('  margin-bottom: 12px;\n')
+    print('}\n')
+    print('</style>\n')
+    print('</head>\n')


The static stuff should probably go into a CONSTANT with triple quotes """, then it can be multiline?

JohannesEbke · 2021-04-13T14:15:26Z

aws_list_all/generate_html.py

+            print('        </td>')
+        print('    </tr>\n')
+    print('</table>')
+    print('</div>')


I am tempted to suggest a more generic solution here, but this is actually quite readable.

JohannesEbke · 2021-04-13T14:15:55Z

aws_list_all/resource_filter.py

+        if listing.service == 'elasticache' and listing.operation == 'DescribeCacheSubnetGroups':
+            response['CacheSubnetGroups'] = [
+                g for g in response.get('CacheSubnetGroups', []) if g.get('CacheSubnetGroupName') != 'default'
+            ]


This branch probably needs a rebase to the current master since I merged the other PR

-Move access to html-file from main to generate_html -Refactor sections from html-head into string constants

-Delete old print-html command -Set html as option to 'view' -Implement view's function to execute query+show or print html results

JohannesEbke

Looks good to me, modulo the issues raised. I'd probably merge and fix up as is, but feel free to update the PR!

JohannesEbke · 2021-05-11T19:38:06Z

aws_list_all/__main__.py

+            os.chdir(args.directory)
+        increase_limit_nofiles()
+        services = args.service or get_services()
+        do_consecutive(


I think this could be expressed as a sequence of query & view.

JohannesEbke · 2021-05-11T19:38:42Z

aws_list_all/__main__.py

+    # Combine the features of query and show and optionally print the findings
+    # from query to an HTML file to be viewed in a browser
+    view = subparsers.add_parser(
+        'view', description='Print a listing to an HTML file to be viewed in a browser',


This should probably be clearer about that it is actually querying the data.

JohannesEbke · 2021-05-11T19:41:37Z

aws_list_all/generate_html.py

+    """Open a file to write HTML-content in and return the original system output and file path"""
+    origout = sys.stdout
+    f = open(name, 'w')
+    sys.stdout = f


I really don't like bending sys.stdout here like this. Ideally, all these functions should just add a string together (lines = []; lines.append instead of print; return "\n".join(lines))

JohannesEbke · 2021-05-11T19:49:30Z

aws_list_all/listing.py

+    def diffInListing(cls, base, diff):
+        return cls(base.input, base.result_type, base.details, base.id_list, diff)
+
+    def __eq__(self, other):


Here you can just define eq and lt and then use https://docs.python.org/3/library/functools.html#functools.total_ordering to generate the rest

-Change printing of html-strings to returning string at the end -Write to html-file after finished joining all strings at the end -Adjust view-command description and help

JohannesEbke · 2021-05-17T12:42:16Z

OK, looks good for now. There are still some merge conflicts now?

gliptak and others added 30 commits October 31, 2020 11:32

Bring Python to 3.9 in Travis

74c794c

Merge pull request JohannesEbke#44 from gliptak/patch-2

1a5e53a

Bring Python to 3.9 in Travis. Thanks!

-boto3 update and recreate caches

5ab67b4

-updated boto3

976688e

Merge pull request JohannesEbke#46 from kntrain/boto3-update-1.16.21

8c300df

Boto3 update 1.16.24

-added additional show options

67defae

-replaced run_raw call with file reference in wrapper

-moved modifications in resources to filter class

faec31c

-fixed restructuring issues

889c555

-moved resources execution to apply_filter -added command line option for resource filters

-removed call to file-reference filter in listing

d7a4a21

recreate_caches: Only accept "Host not found" as error, also Warn if …

2f65155

…run without credentials, as it will run very slowly.

Upgrade to boto3 1.16.57

6d8fcc7

Add heuristic for new host prefixes and recreate caches

e4a37de

Fix up tests

173a70b

Yapf

80c4c8c

-renamed Listing -> RawListing

f2d06fc

-renamed ListingFile -> FilteredListing -moved non-default filters to fixing_filter.py

changed fixing_filters to always apply

65a4788

- new feature branch for visual improvements

3199531

- added print-html from old branch - added count number

- added html search feature

a7016d5

- moved most html-printing functionality to new py-file

- fixed changing column width

06537d5

- fixed dynamically changing count number

63df2d0

- upgraded html style

1076f6b

- fixed html-file call from directory

- added finish time to html

b5a8b61

- added handling in case special ops are specified

c92bb38

- removed empty line

5d120bb

- added unfilter options description

8e40189

Create HTML header and footer

1ea97ae

- Combine title and searchbox in header - Move timestamps to footer

Add comparing feature to available commands

a6c788f

Fix dependent ops and show command flags Bugs

9753209

Implement compare command between 2 directories

5855f94

Move html related code to generate-html.py

ec3fa8e

kntrain and others added 9 commits March 25, 2021 18:42

Restructure to get resource ID during query

4c05e52

Remove stray comment and unfilter-list conversion

aefb2a5

Add error-handling to JSON-file dependent function

3d0565f

Adopt suggestions from errorhandle-branch

dbe60a7

-Remove stray comments -Remove convert_unfilterList -Add error-hadling to JSON-dependent resources-block -Adjust resources-dependent query functions

Fix wrong do_list_files behaviour for errors

7e2cde7

Add ResultListing as return Obj of acquire_listing

2055571

Enhance visualization-feature

436a66c

-Finish display of comparison in HTML -Add display of resource IDs for found resources -Change acquire_listing return value to ResultListing object -Refactor do_list_files into info summary and resource ID search

Fix comparing algorithm bug

02b15a9

-Fix comparing ResultListing-functions -Add JSON-functions to FIlteredListing -Save dir + unfilter in JSON-files -Delete stray comments

JohannesEbke requested changes Apr 13, 2021

View reviewed changes

kntrain added 4 commits April 15, 2021 14:02

Do minor restrcuturing in main and generate_html

e7b75a4

-Move access to html-file from main to generate_html -Refactor sections from html-head into string constants

Incorporate print-html into new command 'view'

5d92c58

-Delete old print-html command -Set html as option to 'view' -Implement view's function to execute query+show or print html results

Incorporate html+compare option into show command

320f99e

Fix html-file save location

7b26c47

JohannesEbke reviewed May 11, 2021

View reviewed changes

Change creation and generating of html-content

e5d169f

-Change printing of html-strings to returning string at the end -Write to html-file after finished joining all strings at the end -Adjust view-command description and help

Merge branch 'master' into feature_comparing

eec28a0

JohannesEbke force-pushed the master branch from 5f4e644 to b063e0c Compare November 10, 2022 17:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve visualization branch + Feature comparing#50

Improve visualization branch + Feature comparing#50
kntrain wants to merge 45 commits intoJohannesEbke:masterfrom
kntrain:feature_comparing

kntrain commented Apr 6, 2021

Uh oh!

JohannesEbke left a comment

Uh oh!

JohannesEbke Apr 13, 2021

Uh oh!

JohannesEbke Apr 13, 2021

Uh oh!

JohannesEbke Apr 13, 2021

Uh oh!

JohannesEbke Apr 13, 2021

Uh oh!

JohannesEbke Apr 13, 2021

Uh oh!

JohannesEbke left a comment

Uh oh!

JohannesEbke May 11, 2021

Uh oh!

JohannesEbke May 11, 2021

Uh oh!

JohannesEbke May 11, 2021

Uh oh!

JohannesEbke May 11, 2021

Uh oh!

JohannesEbke commented May 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kntrain commented Apr 6, 2021

Uh oh!

JohannesEbke left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohannesEbke left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohannesEbke commented May 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants