Skip to content

Commit 1126f7e

Browse files
jeffreyavenjeffreyaven
authored
Merge pull request #10 from stackql/feature/updates
Feature/updates
2 parents 9a6e205 + 1411292 commit 1126f7e

File tree

7 files changed

+179
-27
lines changed

7 files changed

+179
-27
lines changed

‎.github/PULL_REQUEST_TEMPLATE.md

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
## Description
2+
3+
Please include a summary of the changes and the related issue(s). Please also include relevant motivation and context.
4+
5+
Fixes #
6+
7+
## Checklist
8+
9+
Please make sure that the following criteria are met:
10+
11+
- [ ] The PR title is descriptive.
12+
- [ ] Include documentation for example queries (comments or markdown in notebooks)
13+
- [ ] I have ⭐'ed the [stackql](https://github.com/stackql/stackql) and [stackql-jupyter-demo](https://github.com/stackql/stackql-jupyter-demo) repos.
14+
15+
## Additional Notes
16+
17+
Add any additional information or context that might help the reviewers.

‎.github/workflows/main.yml

-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ name: Build and Publish
33
on:
44
push:
55
branches: [main]
6-
pull_request:
76

87
jobs:
98
build-and-push-docker-image:

‎.github/workflows/star-check.yml

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Check if PR author has starred the repository
2+
on:
3+
pull_request:
4+
types: [opened, synchronize, reopened]
5+
6+
jobs:
7+
check-starred:
8+
runs-on: ubuntu-latest
9+
10+
steps:
11+
- name: Checkout code
12+
uses: actions/checkout@v4.1.7
13+
14+
- name: Get PR author username
15+
id: get-author
16+
run: echo "username=${{ github.event.pull_request.user.login }}" >> $GITHUB_ENV
17+
18+
- name: Pull github provider
19+
uses: stackql/stackql-exec@v2.2.1
20+
with:
21+
is_command: 'true'
22+
query: "REGISTRY PULL github;"
23+
24+
- name: Run stackql query
25+
id: check-star
26+
uses: stackql/stackql-assert@v2.2.1
27+
with:
28+
test_query: |
29+
SELECT repo, count(*) as has_starred
30+
FROM github.activity.repo_stargazers
31+
WHERE owner = 'stackql' and repo in ('stackql', 'stackql-jupyter-demo') and login = '${{ env.username }}'
32+
GROUP BY repo;
33+
expected_results_str: '[{"has_starred":"1","repo":"stackql"},{"has_starred":"1","repo":"stackql-jupyter-demo"}]'
34+
continue-on-error: true
35+
36+
- name: Check if starred
37+
if: always() # Ensures this runs regardless of check-star outcome
38+
run: |
39+
if [ "${{ steps.check-star.outcome }}" = "success" ]; then
40+
echo "::notice::Thanks for your support!"
41+
else
42+
echo "::error::It seems you haven't starred the StackQL repositories. Please star the following repos before proceeding: https://github.com/stackql/stackql-jupyter-demo (this repo) and https://github.com/stackql/stackql (our core repo)"
43+
exit 1
44+
fi
45+

‎Dockerfile

+100-21
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,71 @@
1+
# FROM stackql/stackql:latest AS stackql
2+
# EXPOSE 5444
3+
# WORKDIR /home/stackql
4+
# RUN adduser --system --uid 1001 stackql
5+
# RUN addgroup --system --gid 1001 stackql
6+
# RUN chown stackql:stackql /home/stackql
7+
# RUN chown stackql:stackql /srv
8+
# USER stackql
9+
# # pull stackql providers
10+
# RUN stackql exec 'registry pull aws' || (echo "Failed to pull aws provider" && exit 1)
11+
# RUN stackql exec 'registry pull google' || (echo "Failed to pull google provider" && exit 1)
12+
# RUN stackql exec 'registry pull github' || (echo "Failed to pull github provider" && exit 1)
13+
# RUN stackql exec 'registry pull azure' || (echo "Failed to pull azure provider" && exit 1)
14+
# # RUN stackql exec 'registry pull k8s'
15+
# # RUN stackql exec 'registry pull netlify'
16+
# # RUN stackql exec 'registry pull okta'
17+
# # RUN stackql exec 'registry pull sumologic'
18+
# # RUN stackql exec 'registry pull digitalocean'
19+
20+
# FROM jupyter/base-notebook:latest AS jupyter
21+
# WORKDIR /jupyter
22+
# USER root
23+
# RUN apt-get update && \
24+
# apt-get upgrade -y
25+
# # copy example notebooks to Jupyter workspace
26+
# COPY ./notebooks/ /jupyter/
27+
# RUN chmod 644 *.ipynb && \
28+
# chown jovyan:users *.ipynb
29+
# # copy magic extensions
30+
# RUN mkdir -p /jupyter/ext
31+
# COPY ./extensions/* /jupyter/ext
32+
# RUN chmod 644 /jupyter/ext/*.py && \
33+
# chown jovyan:users /jupyter/ext/*.py
34+
# # copy entrypoint script
35+
# RUN mkdir -p /scripts
36+
# COPY ./scripts/start-server.sh /scripts
37+
# COPY ./scripts/entrypoint.sh /scripts
38+
# RUN chmod +x /scripts/start-server.sh
39+
# RUN chmod +x /scripts/entrypoint.sh
40+
# # set up matplotlib temp dir
41+
# RUN mkdir -p /tmp/matplotlib
42+
# RUN chmod 777 /tmp/matplotlib
43+
# ENV MPLCONFIGDIR=/tmp/matplotlib
44+
# ENV PYDEVD_DISABLE_FILE_VALIDATION=1
45+
# # setup python environment
46+
# ENV PYTHON_PACKAGES="\
47+
# pystackql>=3.6.4 \
48+
# matplotlib \
49+
# pandas \
50+
# mplfinance \
51+
# psycopg2-binary \
52+
# nest_asyncio \
53+
# plotly \
54+
# ipytree \
55+
# nbformat \
56+
# networkx \
57+
# "
58+
# RUN pip install --upgrade pip \
59+
# && pip install --no-cache-dir $PYTHON_PACKAGES
60+
# # copy stackql providers from stackql container
61+
# COPY --from=stackql /home/stackql/.stackql /jupyter/.stackql
62+
# RUN ls -al /jupyter/.stackql/src/aws || (echo "aws provider not present" && exit 1)
63+
# RUN ls -al /jupyter/.stackql/src/googleapis.com || (echo "google provider not present" && exit 1)
64+
# RUN ls -al /jupyter/.stackql/src/github || (echo "github provider not present" && exit 1)
65+
# # copy stackql binary from stackql container (service instance)
66+
# COPY --from=stackql /srv/stackql/stackql /srv/stackql/stackql
67+
68+
# Stage 1: StackQL setup
169
FROM stackql/stackql:latest AS stackql
270
EXPOSE 5444
371
WORKDIR /home/stackql
@@ -6,59 +74,70 @@ RUN addgroup --system --gid 1001 stackql
674
RUN chown stackql:stackql /home/stackql
775
RUN chown stackql:stackql /srv
876
USER stackql
9-
# pull stackql providers
77+
78+
# Pull stackql providers
1079
RUN stackql exec 'registry pull aws' || (echo "Failed to pull aws provider" && exit 1)
1180
RUN stackql exec 'registry pull google' || (echo "Failed to pull google provider" && exit 1)
1281
RUN stackql exec 'registry pull github' || (echo "Failed to pull github provider" && exit 1)
1382
RUN stackql exec 'registry pull azure' || (echo "Failed to pull azure provider" && exit 1)
14-
# RUN stackql exec 'registry pull k8s'
15-
# RUN stackql exec 'registry pull netlify'
16-
# RUN stackql exec 'registry pull okta'
17-
# RUN stackql exec 'registry pull sumologic'
18-
# RUN stackql exec 'registry pull digitalocean'
1983

84+
# Stage 2: Jupyter setup
2085
FROM jupyter/base-notebook:latest AS jupyter
2186
WORKDIR /jupyter
2287
USER root
2388
RUN apt-get update && \
2489
apt-get upgrade -y
25-
# copy example notebooks to Jupyter workspace
90+
91+
# Copy example notebooks to Jupyter workspace
2692
COPY ./notebooks/ /jupyter/
27-
RUN chmod 644 *.ipynb && \
28-
chown jovyan:users *.ipynb
29-
# copy magic extensions
93+
RUN chmod 644 /jupyter/*.ipynb && \
94+
chown jovyan:users /jupyter/*.ipynb
95+
96+
# Copy magic extensions
3097
RUN mkdir -p /jupyter/ext
31-
COPY ./extensions/* /jupyter/ext
98+
COPY ./extensions/* /jupyter/ext/
3299
RUN chmod 644 /jupyter/ext/*.py && \
33100
chown jovyan:users /jupyter/ext/*.py
34-
# copy entrypoint script
101+
102+
# Copy entrypoint script
35103
RUN mkdir -p /scripts
104+
COPY ./scripts/start-server.sh /scripts
36105
COPY ./scripts/entrypoint.sh /scripts
37-
RUN chmod +x /scripts/entrypoint.sh
38-
# set up matplotlib temp dir
106+
RUN chmod +x /scripts/start-server.sh /scripts/entrypoint.sh
107+
108+
# Set up matplotlib temp dir
39109
RUN mkdir -p /tmp/matplotlib
40110
RUN chmod 777 /tmp/matplotlib
41111
ENV MPLCONFIGDIR=/tmp/matplotlib
42112
ENV PYDEVD_DISABLE_FILE_VALIDATION=1
43-
# setup python environment
113+
114+
# Setup python environment
44115
ENV PYTHON_PACKAGES="\
45116
pystackql>=3.6.4 \
46117
matplotlib \
47118
pandas \
48-
mplfinance \
119+
mplfinance \
49120
psycopg2-binary \
50121
nest_asyncio \
51122
plotly \
52123
ipytree \
53124
nbformat \
54125
networkx \
55126
"
56-
RUN pip install --upgrade pip \
57-
&& pip install --no-cache-dir $PYTHON_PACKAGES
58-
# copy stackql providers from stackql container
127+
RUN pip install --upgrade pip && pip install --no-cache-dir $PYTHON_PACKAGES
128+
129+
# Copy stackql providers and binary from stackql container
59130
COPY --from=stackql /home/stackql/.stackql /jupyter/.stackql
131+
COPY --from=stackql /srv/stackql/stackql /srv/stackql/stackql
132+
133+
# Ensure providers are present
60134
RUN ls -al /jupyter/.stackql/src/aws || (echo "aws provider not present" && exit 1)
61135
RUN ls -al /jupyter/.stackql/src/googleapis.com || (echo "google provider not present" && exit 1)
62136
RUN ls -al /jupyter/.stackql/src/github || (echo "github provider not present" && exit 1)
63-
# copy stackql binary from stackql container (service instance)
64-
COPY --from=stackql /srv/stackql/stackql /srv/stackql/stackql
137+
138+
# Set user back to default jovyan
139+
USER jovyan
140+
141+
# Set entrypoint
142+
# ENTRYPOINT ["/scripts/entrypoint.sh"]
143+
# CMD ["start-notebook.sh"]

‎README.md

+11
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,14 @@
1+
> ⚡ **Calling All Cloud/Data/Security Enthusiasts, Hacktoberfest 2024 is here!** ⚡
2+
> Interested in contributing StackQL (SQL) queries, notebooks and visualizations for **Cloud Security Posture Management (CSPM)**, **FinOps**, **Cloud Inventory Analysis**, or **Infrastructure-as-Code (IaC)**?
3+
>
4+
> Check out the issues and get started with your first pull request!, Let’s build something amazing together this Hacktoberfest!
5+
6+
💡 **Explore our repositories:** [StackQL](https://github.com/stackql/stackql), [StackQL Deploy](https://stackql-deploy.io/docs/), find provider documentation in the [StackQL Provider Registry Docs](https://registry.stackql.io/)
7+
8+
🔎 Build out example queries for [`aws`](https://aws.stackql.io/providers/aws/), [`gcp`](https://google.stackql.io/providers/google/), [`azure`](https://azure.stackql.io/providers/azure/), [`digitalocean`](https://digitalocean.stackql.io/providers/digitalocean/), [`linode`](https://linode.stackql.io/providers/linode/), [`okta`](https://okta.stackql.io/providers/okta/) and more, including multicloud queries!
9+
10+
---
11+
112
[![StackQL](https://stackql.io/img/stackql-logo-bold.png)](https://stackql.io/)
213
<br />
314
[![GitHub Actions](https://github.com/stackql/stackql-jupyter-demo/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/stackql/stackql-jupyter-demo/actions/workflows/main.yml)

‎scripts/entrypoint.sh

+1-5
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,2 @@
11
#!/bin/sh
2-
/srv/stackql/stackql --version
3-
echo "starting stackql server..."
4-
nohup /srv/stackql/stackql --execution.concurrency.limit=-1 --dataflow.components.max=50 --dataflow.dependency.max=50 --http.response.pageLimit=-1 --pgsrv.port=5466 srv &
5-
echo "stackql server started"
6-
start-notebook.sh --NotebookApp.token=''
2+
sh /scripts/start-server.sh && start-notebook.sh --NotebookApp.token=''

‎scripts/start-server.sh

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/sh
2+
/srv/stackql/stackql --version
3+
echo "starting stackql server..."
4+
nohup /srv/stackql/stackql --execution.concurrency.limit=-1 --dataflow.components.max=50 --dataflow.dependency.max=50 --http.response.pageLimit=-1 --pgsrv.port=5466 srv &
5+
echo "stackql server started"

0 commit comments

Comments
 (0)