Steampipe - use SQL to retrieve data from cloud, platforms and files (Code Café May 2022)
Download as PPTX, PDF0 likes549 views
Steampipe is a tool that provides a SQL interface to external data sources, allowing users to retrieve and visualize metadata from various services through PostgreSQL. It offers features such as querying multiple data formats (CSV, JSON, etc.), creating dashboards, and running benchmarks for security and compliance. Overall, it is valuable for development and operations but not intended for runtime data retrieval in applications.
![Classificatie: vertrouwelijk
select * from multiple data sources (CSV, JSON, YAML, INI)
Conclusion Code Café - Steampipe - select * from cloud; 6
with books_from_json as
( select e.item ->> 'title' as title
, e.item ->> 'author' as author
..
from json_file b
cross join
jsonb_path_query(b.content, '$.books[*]') as e(item)
where
b.path like '%books.json'
)
, books_from_csv as
( select "Title" as title
, "Genre" as genre
from
csv.books
)
select bj.title
, bc.genre
..
from
books_from_json bj
join
books_from_csv bc
on
bj.title = bc.title
books.csv
books.json](https://image.slidesharecdn.com/codecafe-steampipe-mei2022-220527203154-252ac0d0/85/Steampipe-use-SQL-to-retrieve-data-from-cloud-platforms-and-files-Code-Cafe-May-2022-6-320.jpg)
![Classificatie: vertrouwelijk
Under the hood
• Steampipe leverages PostgreSQL Foreign Data Wrappers to provide a SQL
interface to external services and systems.
• Steampipe uses an embedded PostgreSQL database and you can use
standard Postgres syntax to query Steampipe.
• note: no data is persisted in Steampipe: every table access may correspond
to one (or more) API calls
• query results are cached for 5 minutes by default
(configurable per data source)
• select only columns actually needed. Do NOT use select *
• [For non-key columns,] data for all rows must be collected, and then
filtered
• custom Steampipe plugins can be created and easily be integrated
• Steampipe plugins are programmed in Go
Conclusion Code Café - Steampipe - select * from cloud; 13](https://image.slidesharecdn.com/codecafe-steampipe-mei2022-220527203154-252ac0d0/85/Steampipe-use-SQL-to-retrieve-data-from-cloud-platforms-and-files-Code-Cafe-May-2022-13-320.jpg)
Steampipe - use SQL to retrieve data from cloud, platforms and files (Code Café May 2022)
- 1. Classificatie: vertrouwelijk Steampipe select * from <cloud> Conclusion Code Café – 31 mei 2022 Lucas Jellema, CTO & Architect AMIS | Conclusion
- 2. Classificatie: vertrouwelijk Steampipe • select from csv • select from json – incl aggregate • select from json join csv • select from cloud • under the hood: PostgreSQL – foreign interface (plug in external data sources and access them like read only tables) • scripts • modules • Steampipe dashboards allow you to visualize your steampipe data. • Steampipe mods can also define benchmarks and controls to assess your environment against security, compliance, operational, and cost controls. Conclusion Code Café - Steampipe - select * from cloud; 2
- 3. Classificatie: vertrouwelijk select * from CSV Conclusion Code Café - Steampipe - select * from cloud; 3 select upper("Title") as title , "Author“ as author from csv.books where "Genre" = 'economics' books.csv
- 4. Classificatie: vertrouwelijk select * from CSV Conclusion Code Café - Steampipe - select * from cloud; 4 select upper("Title") as title , "Author“ as author from csv.books where "Genre" = 'economics' books.csv
- 5. Classificatie: vertrouwelijk select * from multiple CSV files Conclusion Code Café - Steampipe - select * from cloud; 5 select "Title" , "Author" , "Genre" , "description" as "Genre Description" , "status_points" as "Genre Status Score" from csv.books b join csv.genres g on b."Genre" = g."genre" books.csv genres.csv
- 6. Classificatie: vertrouwelijk select * from multiple data sources (CSV, JSON, YAML, INI) Conclusion Code Café - Steampipe - select * from cloud; 6 with books_from_json as ( select e.item ->> 'title' as title , e.item ->> 'author' as author .. from json_file b cross join jsonb_path_query(b.content, '$.books[*]') as e(item) where b.path like '%books.json' ) , books_from_csv as ( select "Title" as title , "Genre" as genre from csv.books ) select bj.title , bc.genre .. from books_from_json bj join books_from_csv bc on bj.title = bc.title books.csv books.json
- 7. Classificatie: vertrouwelijk Query for Compute Instances on Oracle Cloud Conclusion Code Café - Steampipe - select * from cloud; 7
- 8. Classificatie: vertrouwelijk Query the Cloud Conclusion Code Café - Steampipe - select * from cloud; 8 OCI REST API Calls Result in tabular, csv, json format
- 9. Classificatie: vertrouwelijk Steampipe query features • Output format: table, json, csv • Named queries (defined in .sql file, reused like views in SQL statements) • Interactive shell • Batch mode (background, one or more .sql scripts) • => write results to standard output • steampipe query oci-vms.sql --output json > oci_compute_instances.json • Service Mode => start embedded PostgreSQL database – allowing access from any tool that can talk to PostgreSQL • leverage Steampipe database views • Container image is available • Aggregator connections • one query across multiple data sources • Cast values – for example text to int (::int) Conclusion Code Café - Steampipe - select * from cloud; 9
- 10. Classificatie: vertrouwelijk Steampipe – plugin framework with embedded PostgresSQL Conclusion Code Café - Steampipe - select * from cloud; 10
- 11. Classificatie: vertrouwelijk Embedded PostgreSQL • each plugin corresponds to a database schema Conclusion Code Café - Steampipe - select * from cloud; 11
- 12. Classificatie: vertrouwelijk Embedded PostgreSQL • each plugin corresponds to a database schema • each resource is exposed as a database view • data populated from external data source through foreign data wrapper Conclusion Code Café - Steampipe - select * from cloud; 12 imap_message gitlab_branch azure_storage_blob aws_ec2_instance twitter_search_recent {csv-file-name}
- 13. Classificatie: vertrouwelijk Under the hood • Steampipe leverages PostgreSQL Foreign Data Wrappers to provide a SQL interface to external services and systems. • Steampipe uses an embedded PostgreSQL database and you can use standard Postgres syntax to query Steampipe. • note: no data is persisted in Steampipe: every table access may correspond to one (or more) API calls • query results are cached for 5 minutes by default (configurable per data source) • select only columns actually needed. Do NOT use select * • [For non-key columns,] data for all rows must be collected, and then filtered • custom Steampipe plugins can be created and easily be integrated • Steampipe plugins are programmed in Go Conclusion Code Café - Steampipe - select * from cloud; 13
- 14. Classificatie: vertrouwelijk Steampipe Dashboards • Steampipe can run dashboards that consolidate and visualize results from Steampipe queries • Dashboards are defined in HCL (language of Terraform) • per element: define title, SQL query, type of widget, width • 100s of predefined dashboards are available • Data can be downloaded from Dashboard • SQL queries can easily be copied from Dashboard • Queries can be parametrized – with values coming from user input UI elements – text or select (populated from SQL) Conclusion Code Café - Steampipe - select * from cloud; 14
- 15. Classificatie: vertrouwelijk Steampipe Benchmarks & Controls • a generic mechanism for defining and running control frameworks such as CIS, NIST, HIPAA, etc, as well as your own customized groups of controls • simplified, consistent mechanism for defining, running, and returning output from disparate benchmark and control frameworks. • checks in the SQL results for violations of rules & recommendations • on security, cost control Conclusion Code Café - Steampipe - select * from cloud; 15
- 16. Classificatie: vertrouwelijk Summary • steampipe.io • very convenient way of retrieving and combining resource (meta) data from many different sources • valuable for development and operations • not for runtime data retrieval in applications Conclusion Code Café - Steampipe - select * from cloud; 16
Editor's Notes
- #14: https://steampipe.io/docs/sql/querying-json