Merge pull request #38 from lenarsaitov/significant-refactor-code

update file name, add suburban type, refactor, update description on …

Author: lenarsaitov

README.md

@@ -21,7 +21,7 @@ print(data[0])
 ```
                               Preparing to collect information from pages..
 The absolute path to the file: 
- /Users/macbook/some_project/cianparser/cian_parsing_result_flat_sale_1_2_moskva_12_Jan_2024_21_48_43_100892.csv 
+ /Users/macbook/some_project/cianparser/cian_flat_sale_1_2_moskva_12_Jan_2024_21_48_43_100892.csv 
 
 The page from which the collection of information begins: 
  https://cian.ru/cat.php?engine_version=2&p=1&with_neighbors=0&region=1&deal_type=sale&offer_type=flat&room1=1&room2=1
@@ -53,7 +53,7 @@ Total number of parsed offers: 56.
 ```
 ### Инициализация
 Параметры, используемые при инициализации парсера через функциою CianParser:
-* __location__ - локация объявления, к примеру, _Санкт-Петербург_ (для просмотра доступных мест используйте _cianparser.list_locations())_
+* __location__ - локация объявления, к примеру, _Москва_ (для просмотра доступных мест используйте _cianparser.list_locations())_
 * __proxies__ - прокси (см раздел __Cloudflare, CloudScraper, Proxy__), по умолчанию _None_
 
 ### Метод get_flats
@@ -64,13 +64,19 @@ Total number of parsed offers: 56.
 * __with_extra_data__ - необходимо ли сбор дополнительных данных, но с кратным продолжительности по времени (см. ниже в __Примечании__), по умолчанию _False_
 * __additional_settings__ - дополнительные настройки поиска (см. ниже в __Дополнительные настройки поиска__), по умолчанию _None_
 
-Пример использования данного метода представлен выше
+Пример:
+```python
+import cianparser
+
+moscow_parser = cianparser.CianParser(location="Москва")
+data = moscow_parser.get_flats(deal_type="sale", rooms=(1, 2), additional_settings={"start_page":1, "end_page":2})
+```
 
 В проекте предусмотрен функционал корректного завершения в случае окончания страниц. По данному моменту, следует изучить раздел __Ограничения__
 
 ### Метод get_suburban (сбор объявлений домов/участков/танхаусав итп)
 Данный метод принимает следующий аргументы:
-* __object_type__ - тип здания, к примеру, дом/дача, часть дома, участок, танхаус _("house", "house-part", "land-plot", "townhouse")_
+* __suburban_type__ - тип здания, к примеру, дом/дача, часть дома, участок, танхаус _("house", "house-part", "land-plot", "townhouse")_
 * __deal_type__ - тип объявления, к примеру, долгосрочная аренда, продажа _("rent_long", "sale")_
 * __with_saving_csv__ - необходимо ли сохранение собираемых данных (в реальном времени в процессе сбора данных) или нет, по умолчанию _False_
 * __with_extra_data__ - необходимо ли сбор дополнительных данных, но с кратным продолжительности по времени, по умолчанию _False_
@@ -81,7 +87,7 @@ Total number of parsed offers: 56.
 import cianparser
 
 moscow_parser = cianparser.CianParser(location="Москва")
-data = moscow_parser.get_suburban(object_type="townhouse", deal_type="sale", with_saving_csv=True, additional_settings={"start_page":1, "end_page":1})
+data = moscow_parser.get_suburban(suburban_type="townhouse", deal_type="sale", additional_settings={"start_page":1, "end_page":1})
 ```
 
 ### Метод get_newobjects (сбор даннных по новостройкам)
@@ -93,7 +99,7 @@ data = moscow_parser.get_suburban(object_type="townhouse", deal_type="sale", wit
 import cianparser
 
 moscow_parser = cianparser.CianParser(location="Москва")
-data = moscow_parser.get_newobjects(with_saving_csv=True)
+data = moscow_parser.get_newobjects()
 ```
 
 ### Дополнительные настройки поиска
@@ -227,7 +233,7 @@ __with_saving_csv__ значение ___True___.
 #### Пример получаемого файла при вызове метода __get_flats__ с __with_extra_data__ = __True__:
 
 ```bash
-cian_parsing_result_flat_sale_1_1_moskva_12_Jan_2024_22_29_48_117413.csv
+cian_flat_sale_1_1_moskva_12_Jan_2024_22_29_48_117413.csv
 ```
 | author | author_type | url | location | deal_type | accommodation_type | floor | floors_count | rooms_count | total_meters | price_per_m2 | price | year_of_construction | object_type | house_material_type | heating_type | finish_type | living_meters | kitchen_meters | phone | district | street | house_number | underground | residential_complex
 | ------ | ------ | ------ | ------ | ------ | ------ | ----------- | ---- | ---- | --------- | ------------------ | ----- | ------------ | ----------- | ------------ | --------------- | ----------- | ----------- | -------------------- | --- | --- | --- | --- | --- | ---
@@ -238,7 +244,7 @@ cian_parsing_result_flat_sale_1_1_moskva_12_Jan_2024_22_29_48_117413.csv
 #### Пример получаемого файла при вызове метода __get_suburban__ с __with_extra_data__ = __True__:
 
 ```bash
-cian_parsing_result_suburban_sale_15_15_moskva_13_Jan_2024_04_30_47_963046.csv
+cian_suburban_townhouse_sale_15_15_moskva_13_Jan_2024_04_30_47_963046.csv
 ```
 | author | author_type | url | location | deal_type | accommodation_type | price | year_of_construction | house_material_type | land_plot | land_plot_status | heating_type | gas_type | water_supply_type | sewage_system | bathroom | living_meters | floors_count | phone | district | underground | street | house_number
  | -----  | -----  | -----  | -----  | -----  | -----  | -----  | -----  | -----  | ----- | ------------ | ----------- | ------------ | --------------- | ----------- | ----------- | -------------------- | --- | --- | --- | --- | --- | ---
@@ -249,7 +255,7 @@ cian_parsing_result_suburban_sale_15_15_moskva_13_Jan_2024_04_30_47_963046.csv
 #### Пример получаемого файла при вызове метода __get_newobjects__:
 
 ```bash
-cian_parsing_result_newobject_sale_1_1_moskva_13_Jan_2024_01_27_32_734734.csv
+cian_newobject_13_Jan_2024_01_27_32_734734.csv
 ```
 | name | location | accommodation_type | url | full_location_address | year_of_construction | house_material_type | finish_type | ceiling_height | class | parking_type | floors_from | floors_to | builder
  | ----- | ------------ | ----------- | ------------ | --------------- | ----------- | ----------- | -------------------- | --- | --- | --- | --- | --- | ---

cianparser/base.py

@@ -1,10 +1,7 @@
 import math
 import csv
-import pathlib
-from datetime import datetime
-import transliterate
 
-from cianparser.constants import FILE_NAME_BASE, SPECIFIC_FIELDS_FOR_RENT_LONG, SPECIFIC_FIELDS_FOR_RENT_SHORT, SPECIFIC_FIELDS_FOR_SALE
+from cianparser.constants import SPECIFIC_FIELDS_FOR_RENT_LONG, SPECIFIC_FIELDS_FOR_RENT_SHORT, SPECIFIC_FIELDS_FOR_SALE
 
 
 class BaseListPageParser:
@@ -41,9 +38,13 @@ def is_rent_short(self):
         return self.deal_type == "rent" and self.rent_period_type == 2
 
     def build_file_path(self):
-        now_time = datetime.now().strftime("%d_%b_%Y_%H_%M_%S_%f")
-        file_name = FILE_NAME_BASE.format(self.accommodation_type, self.deal_type, self.start_page, self.end_page, transliterate.translit(self.location_name.lower(), reversed=True), now_time)
-        return pathlib.Path(pathlib.Path.cwd(), file_name.replace("'", ""))
+        pass
+
+    def define_average_price(self, price_data):
+        if "price" in price_data:
+            self.average_price = (self.average_price * self.count_parsed_offers + price_data["price"]) / self.count_parsed_offers
+        elif "price_per_month" in price_data:
+            self.average_price = (self.average_price * self.count_parsed_offers + price_data["price_per_month"]) / self.count_parsed_offers
 
     def print_parse_progress(self, page_number, count_of_pages, offers, ind):
         total_planed_offers = len(offers) * count_of_pages

cianparser/cianparser.py

@@ -118,22 +118,22 @@ def get_flats(self, deal_type: str, rooms, with_saving_csv=False, with_extra_dat
                                additional_settings=additional_settings))
         return self.__parser__.result
 
-    def get_suburban(self, object_type: str, deal_type: str, with_saving_csv=False, with_extra_data=False, additional_settings=None):
+    def get_suburban(self, suburban_type: str, deal_type: str, with_saving_csv=False, with_extra_data=False, additional_settings=None):
         """
         Parse information of suburbans from cian website
         Examples:
             >>> moscow_parser = cianparser.CianParser(location="Москва")
-            >>> data = moscow_parser.get_suburbans(object_type="house",deal_type="rent_long")
-            >>> data = moscow_parser.get_suburbans(object_type="house",deal_type="rent_short", with_saving_csv=True)
-            >>> data = moscow_parser.get_suburbans(object_type="townhouse",deal_type="sale", additional_settings={"start_page": 1, "end_page": 1, "sort_by":"price_from_min_to_max"})
-        :param object_type: type of object, e.g. "house", "house-part", "land-plot", "townhouse"
+            >>> data = moscow_parser.get_suburbans(suburban_type="house",deal_type="rent_long")
+            >>> data = moscow_parser.get_suburbans(suburban_type="house",deal_type="rent_short", with_saving_csv=True)
+            >>> data = moscow_parser.get_suburbans(suburban_type="townhouse",deal_type="sale", additional_settings={"start_page": 1, "end_page": 1, "sort_by":"price_from_min_to_max"})
+        :param suburban_type: type of suburban building, e.g. "house", "house-part", "land-plot", "townhouse"
         :param deal_type: type of deal, e.g. "rent_long", "rent_short", "sale"
         :param with_saving_csv: is it necessary to save data in csv, default False
         :param with_extra_data:  is it necessary to collect additional data (but with increasing time duration), default False
         :param additional_settings:  additional settings such as min_price, sort_by and others, default None
         """
 
-        __validation_get_suburban__(deal_type=deal_type, object_type=object_type)
+        __validation_get_suburban__(suburban_type=suburban_type, deal_type=deal_type)
         deal_type, rent_period_type = __define_deal_type__(deal_type)
         self.__parser__ = SuburbanListPageParser(
             session=self.__session__,
@@ -143,10 +143,11 @@ def get_suburban(self, object_type: str, deal_type: str, with_saving_csv=False,
             with_saving_csv=with_saving_csv,
             with_extra_data=with_extra_data,
             additional_settings=additional_settings,
+            object_type=suburban_type,
         )
         self.__run__(
             __build_url_list__(location_id=self.__location_id__, deal_type=deal_type, accommodation_type="suburban",
-                               rooms=None, rent_period_type=rent_period_type, object_type=object_type,
+                               rooms=None, rent_period_type=rent_period_type, suburban_type=suburban_type,
                                additional_settings=additional_settings))
         return self.__parser__.result
 
@@ -213,18 +214,18 @@ def __validation_get_flats__(deal_type, rooms):
                          f'It is correct int, str and tuple types. Example 1, (1,3, "studio"), "studio, "all".')
 
 
-def __validation_get_suburban__(deal_type, object_type):
+def __validation_get_suburban__(suburban_type, deal_type):
+    if suburban_type not in OBJECT_SUBURBAN_TYPES.keys():
+        raise ValueError(f'You entered suburban_type={suburban_type}, which is not valid value. '
+                         f'Try entering one of these values: "house", "house-part", "land-plot", "townhouse".')
+
     if deal_type not in DEAL_TYPES:
         raise ValueError(f'You entered deal_type={deal_type}, which is not valid value. '
                          f'Try entering one of these values: "rent_long", "sale".')
 
-    if object_type not in OBJECT_SUBURBAN_TYPES.keys():
-        raise ValueError(f'You entered object_type={object_type}, which is not valid value. '
-                         f'Try entering one of these values: "house", "house-part", "land-plot", "townhouse".')
-
 
 def __build_url_list__(location_id, deal_type, accommodation_type, rooms=None, rent_period_type=None,
-                       object_type=None, additional_settings=None):
+                       suburban_type=None, additional_settings=None):
     url_builder = URLBuilder(accommodation_type == "newobject")
     url_builder.add_location(location_id)
     url_builder.add_deal_type(deal_type)
@@ -236,8 +237,8 @@ def __build_url_list__(location_id, deal_type, accommodation_type, rooms=None, r
     if rent_period_type is not None:
         url_builder.add_rent_period_type(rent_period_type)
 
-    if object_type is not None:
-        url_builder.add_object_suburban_type(object_type)
+    if suburban_type is not None:
+        url_builder.add_object_suburban_type(suburban_type)
 
     if additional_settings is not None:
         url_builder.add_additional_settings(additional_settings)

cianparser/constants.py

@@ -8,7 +8,9 @@
 
 FLOATS_NUMBERS_REG_EXPRESSION = r"[+-]? *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?"
 
-FILE_NAME_BASE = 'cian_parsing_result_{}_{}_{}_{}_{}_{}.csv'
+FILE_NAME_FLAT_FORMAT = 'cian_{}_{}_{}_{}_{}_{}.csv'
+FILE_NAME_SUBURBAN_FORMAT = 'cian_{}_{}_{}_{}_{}_{}_{}.csv'
+FILE_NAME_NEWOBJECT_FORMAT = 'cian_{}_{}_{}.csv'
 
 BASE_URL = "https://cian.ru"
 DEFAULT_POSTFIX_PATH = "/cat.php?"

cianparser/flat_list.py

@@ -1,12 +1,21 @@
 import bs4
 import time
+import pathlib
+from datetime import datetime
+from transliterate import translit
 
+from cianparser.constants import FILE_NAME_FLAT_FORMAT
 from cianparser.helpers import union_dicts, define_author, define_location_data, define_specification_data, define_deal_url_id, define_price_data
 from cianparser.flat import FlatPageParser
 from cianparser.base import BaseListPageParser
 
 
 class FlatListPageParser(BaseListPageParser):
+    def build_file_path(self):
+        now_time = datetime.now().strftime("%d_%b_%Y_%H_%M_%S_%f")
+        file_name = FILE_NAME_FLAT_FORMAT.format(self.accommodation_type, self.deal_type, self.start_page, self.end_page, translit(self.location_name.lower(), reversed=True), now_time)
+        return pathlib.Path(pathlib.Path.cwd(), file_name.replace("'", ""))
+
     def parse_list_offers_page(self, html, page_number: int, count_of_pages: int, attempt_number: int):
         list_soup = bs4.BeautifulSoup(html, 'html.parser')
 
@@ -55,6 +64,7 @@ def parse_offer(self, offer):
             time.sleep(4)
 
         self.count_parsed_offers += 1
+        self.define_average_price(price_data=price_data)
         self.result_set.add(define_deal_url_id(common_data["url"]))
         self.result.append(union_dicts(author_data, common_data, specification_data, price_data, page_data, location_data))
 

cianparser/newobject_list.py

@@ -4,10 +4,10 @@
 import csv
 import pathlib
 from datetime import datetime
-import transliterate
+from transliterate import translit
 import urllib.parse
 
-from cianparser.constants import FILE_NAME_BASE
+from cianparser.constants import FILE_NAME_NEWOBJECT_FORMAT
 from cianparser.helpers import union_dicts
 from cianparser.newobject import NewObjectPageParser
 
@@ -30,7 +30,7 @@ def __init__(self, session, location_name: str, with_saving_csv=False):
 
     def build_file_path(self):
         now_time = datetime.now().strftime("%d_%b_%Y_%H_%M_%S_%f")
-        file_name = FILE_NAME_BASE.format(self.accommodation_type, self.deal_type, self.start_page, self.end_page, transliterate.translit(self.location_name.lower(), reversed=True), now_time)
+        file_name = FILE_NAME_NEWOBJECT_FORMAT.format(self.accommodation_type, translit(self.location_name.lower(), reversed=True), now_time)
         return pathlib.Path(pathlib.Path.cwd(), file_name.replace("'", ""))
 
     def print_parse_progress(self, page_number, count_of_pages, offers, ind):

cianparser/suburban_list.py

@@ -1,12 +1,21 @@
 import bs4
 import time
+import pathlib
+from datetime import datetime
+from transliterate import translit
 
+from cianparser.constants import FILE_NAME_SUBURBAN_FORMAT
 from cianparser.helpers import union_dicts, define_author, parse_location_data, define_price_data, define_deal_url_id
 from cianparser.suburban import SuburbanPageParser
 from cianparser.base import BaseListPageParser
 
 
 class SuburbanListPageParser(BaseListPageParser):
+    def build_file_path(self):
+        now_time = datetime.now().strftime("%d_%b_%Y_%H_%M_%S_%f")
+        file_name = FILE_NAME_SUBURBAN_FORMAT.format(self.accommodation_type, self.object_type, self.deal_type, self.start_page, self.end_page, translit(self.location_name.lower(), reversed=True), now_time)
+        return pathlib.Path(pathlib.Path.cwd(), file_name.replace("'", ""))
+
     def parse_list_offers_page(self, html, page_number: int, count_of_pages: int, attempt_number: int):
         list_soup = bs4.BeautifulSoup(html, 'html.parser')
 
@@ -39,7 +48,7 @@ def parse_offer(self, offer):
         common_data["location"] = self.location_name
         common_data["deal_type"] = self.deal_type
         common_data["accommodation_type"] = self.accommodation_type
-        common_data["object_type"] = self.object_type
+        common_data["suburban_type"] = self.object_type
 
         author_data = define_author(block=offer)
         location_data = parse_location_data(block=offer)
@@ -55,6 +64,7 @@ def parse_offer(self, offer):
             time.sleep(4)
 
         self.count_parsed_offers += 1
+        self.define_average_price(price_data=price_data)
         self.result_set.add(define_deal_url_id(common_data["url"]))
         self.result.append(union_dicts(author_data, common_data, price_data, page_data, location_data))
 

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = cianparser
-version = 1.0.0
+version = 1.0.1
 description = Parser information from Cian website
 url = https://github.com/lenarsaitov/cianparser
 author = Lenar Saitov

setup.py

@@ -6,7 +6,7 @@
 
 setup(
     name='cianparser',
-    version='1.0.0',
+    version='1.0.1',
     description='Parser information from Cian website',
     url='https://github.com/lenarsaitov/cianparser',
     author='Lenar Saitov',