Merge pull request #218 from censys/adh/virtual-host-update

feat(api): Search Virtual Hosts

Author: thehappydinoa

.github/CODE_OF_CONDUCT.md

@@ -1,4 +1,3 @@
-
 # Contributor Covenant Code of Conduct
 
 ## Our Pledge
@@ -18,23 +17,23 @@ diverse, inclusive, and healthy community.
 Examples of behavior that contributes to a positive environment for our
 community include:
 
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes,
   and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the
+- Focusing on what is best not just for us as individuals, but for the
   overall community
 
 Examples of unacceptable behavior include:
 
-* The use of sexualized language or imagery, and sexual attention or
+- The use of sexualized language or imagery, and sexual attention or
   advances of any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email
   address, without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
+- Other conduct which could reasonably be considered inappropriate in a
   professional setting
 
 ## Enforcement Responsibilities
@@ -107,7 +106,7 @@ Violating these terms may lead to a permanent ban.
 ### 4. Permanent Ban
 
 **Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior,  harassment of an
+standards, including sustained inappropriate behavior, harassment of an
 individual, or aggression toward or disparagement of classes of individuals.
 
 **Consequence**: A permanent ban from any sort of public interaction within
@@ -119,15 +118,15 @@ This Code of Conduct is adapted from the [Contributor Covenant][homepage],
 version 2.0, available at
 [https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
 
-Community Impact Guidelines were inspired by 
-[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][mozilla coc].
 
 For answers to common questions about this code of conduct, see the FAQ at
-[https://www.contributor-covenant.org/faq][FAQ]. Translations are available 
+[https://www.contributor-covenant.org/faq][faq]. Translations are available
 at [https://www.contributor-covenant.org/translations][translations].
 
 [homepage]: https://www.contributor-covenant.org
 [v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
-[Mozilla CoC]: https://github.com/mozilla/diversity
-[FAQ]: https://www.contributor-covenant.org/faq
+[mozilla coc]: https://github.com/mozilla/diversity
+[faq]: https://www.contributor-covenant.org/faq
 [translations]: https://www.contributor-covenant.org/translations

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,27 +1,28 @@
 ---
 name: Bug report
 about: Create a report to help us improve
-title: ''
-labels: 'type: bug'
-assignees: ''
-
+title: ""
+labels: "type: bug"
+assignees: ""
 ---
 
 **Describe the bug**
 A clear and concise description of what the bug is.
 
 **To Reproduce**
 Steps to reproduce the behavior:
+
 1. Run '...'
 2. Then '....'
 
 **Expected behavior**
 A clear and concise description of what you expected to happen.
 
 **Device (please complete the following information):**
- - OS [e.g. macOS]:
- - Censys Python Version [e.g. 1.1.0]:
- - Shell [e.g. bash, zsh]:
+
+- OS [e.g. debian, macOS]:
+- Censys Python Version [e.g. 2.1.0]: <!-- $ censys --version -->
+- Shell [e.g. bash, zsh]: <!-- $ echo $SHELL -->
 
 **Additional context**
 Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,10 +1,9 @@
 ---
 name: Feature request
 about: Suggest an idea for this project
-title: ''
-labels: 'type: feature request'
-assignees: ''
-
+title: ""
+labels: "type: feature request"
+assignees: ""
 ---
 
 **Is your feature request related to a problem? Please describe.**

.github/ISSUE_TEMPLATE/question.md

@@ -1,11 +1,10 @@
 ---
 name: Question
 about: Ask a question about this project
-title: ''
-labels: 'type: question'
-assignees: ''
-
+title: ""
+labels: "type: question"
+assignees: ""
 ---
 
 **What is your question?**
-A clear and concise question. Ex. What can I use ___ for? [...]
+A clear and concise question. Ex. What can I use \_\_\_ for? [...]

.github/PULL_REQUEST_TEMPLATE/pull_request_template.md

@@ -1,8 +1,8 @@
 # Description
 
 <!--
-Please include a summary of the change(s) and which issue(s) is fixed. 
-Please also include relevant motivation and context. 
+Please include a summary of the change(s) and which issue(s) is fixed.
+Please also include relevant motivation and context.
 List any dependencies that are required for this change.
 -->
 

.github/workflows/stale.yml

@@ -8,22 +8,21 @@ name: Mark stale issues and pull requests
 on:
   workflow_dispatch:
   schedule:
-  - cron: '30 12 * * *'
+    - cron: "30 12 * * *"
 
 jobs:
   stale:
-
     runs-on: ubuntu-latest
     timeout-minutes: 5
     permissions:
       issues: write
       pull-requests: write
 
     steps:
-    - uses: actions/stale@v3
-      with:
-        repo-token: ${{ secrets.GITHUB_TOKEN }}
-        days-before-stale: 60
-        days-before-close: 7
-        stale-issue-label: 'stale'
-        stale-pr-label: 'stale'
+      - uses: actions/stale@v3
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          days-before-stale: 60
+          days-before-close: 7
+          stale-issue-label: "stale"
+          stale-pr-label: "stale"

censys/cli/commands/search.py

@@ -101,6 +101,9 @@ def cli_search(args: argparse.Namespace):
         if args.pages:
             search_args["pages"] = args.pages
 
+        if args.virtual_hosts:
+            search_args["virtual_hosts"] = args.virtual_hosts
+
         with err_console.status("Searching"):
             query = index.search(args.query, **search_args)
 
@@ -183,6 +186,14 @@ def include(parent_parser: argparse._SubParsersAction, parents: dict):
         type=int,
         help="number of pages of results to return (when set to -1 returns all pages available)",
     )
+    v2_group.add_argument(
+        "--virtual-hosts",
+        type=str,
+        default="EXCLUDE",
+        choices=["INCLUDE", "EXCLUDE", "ONLY"],
+        metavar="INCLUDE|EXCLUDE|ONLY",
+        help="whether to include virtual hosts in the results",
+    )
 
     v1_group = search_parser.add_argument_group(
         f"v1 specific arguments ({', '.join(V1_INDEXES)})"

censys/search/v2/api.py

@@ -111,6 +111,7 @@ def __init__(
             per_page: Optional[int] = None,
             cursor: Optional[str] = None,
             pages: int = 1,
+            **kwargs: Any,
         ):
             """Inits Query.
 
@@ -120,6 +121,7 @@ def __init__(
                 per_page (int): Optional; The number of results to be returned for each page. Defaults to 100.
                 cursor (int): Optional; The cursor of the desired result set.
                 pages (int): Optional; The number of pages returned. Defaults to 1. If you set this to -1, it will return all pages.
+                **kwargs (Any): Optional; Additional arguments to be passed to the query.
             """
             self.api = api
             self.query = query
@@ -131,6 +133,7 @@ def __init__(
                 self.pages = float("inf")
             else:
                 self.pages = pages
+            self.extra_args = kwargs
 
         def __call__(self, per_page: Optional[int] = None) -> List[dict]:
             """Search current index.
@@ -151,6 +154,7 @@ def __call__(self, per_page: Optional[int] = None) -> List[dict]:
                 "q": self.query,
                 "per_page": per_page or self.per_page or 100,
                 "cursor": self.nextCursor or self.cursor,
+                **self.extra_args,
             }
             payload = self.api._get(self.api.search_path, args)
             self.page += 1
@@ -209,6 +213,7 @@ def search(
         per_page: Optional[int] = None,
         cursor: Optional[str] = None,
         pages: int = 1,
+        **kwargs: Any,
     ) -> Query:
         """Search current index.
 
@@ -220,11 +225,12 @@ def search(
             per_page (int): Optional; The number of results to be returned for each page. Defaults to 100.
             cursor (int): Optional; The cursor of the desired result set.
             pages (int): Optional; The number of pages returned. Defaults to 1.
+            **kwargs (Any): Optional; Additional arguments to be passed to the query.
 
         Returns:
             Query: Query object that can be a callable or an iterable.
         """
-        return self.Query(self, query, per_page, cursor, pages)
+        return self.Query(self, query, per_page, cursor, pages, **kwargs)
 
     def view(
         self,

censys/search/v2/hosts.py

@@ -1,5 +1,5 @@
 """Interact with the Censys Search Host API."""
-from typing import List, Optional
+from typing import Any, List, Optional
 
 from .api import CensysSearchAPIv2
 from censys.common.types import Datetime
@@ -72,6 +72,35 @@ class CensysHosts(CensysSearchAPIv2):
     INDEX_NAME = "hosts"
     """Name of Censys Index."""
 
+    def search(
+        self,
+        query: str,
+        per_page: Optional[int] = None,
+        cursor: Optional[str] = None,
+        pages: int = 1,
+        virtual_hosts: Optional[str] = None,
+        **kwargs: Any,
+    ) -> CensysSearchAPIv2.Query:
+        """Search host index.
+
+        Searches the given index for all records that match the given query.
+        For more details, see our documentation: https://search.censys.io/api
+
+        Args:
+            query (str): The query to be executed.
+            per_page (int): Optional; The number of results to be returned for each page. Defaults to 100.
+            cursor (int): Optional; The cursor of the desired result set.
+            virtual_hosts (str): Optional; Whether to include virtual hosts in the results. Valid values are "EXCLUDE", "INCLUDE", and "ONLY".
+            pages (int): Optional; The number of pages returned. Defaults to 1.
+            **kwargs (Any): Optional; Additional arguments to be passed to the query.
+
+        Returns:
+            Query: Query object that can be a callable or an iterable.
+        """
+        if virtual_hosts:
+            kwargs["virtual_hosts"] = virtual_hosts
+        return super().search(query, per_page, cursor, pages, **kwargs)
+
     def view_host_names(self, ip_address: str) -> List[str]:
         """Fetches a list of host names for the specified IP address.
 

examples/search/search_hosts.py

@@ -15,3 +15,9 @@
 # View all results
 query = c.v2.hosts.search("service.service_name: HTTP", per_page=5, pages=2)
 print(query.view_all())
+
+# Search for virtual hosts
+query = c.v2.hosts.search(
+    "NOT services.service_name: HTTP", per_page=5, virtual_hosts="ONLY"
+)
+print(query())