o h;@sdZddlZddlZddlmZddlmZddlmZm Z m Z ddl m Z ddl Z ddl mZmZddlZddlZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZm Z ddl!m"Z"ddl#Z$ddl%m&Z&eGdddZ'eGdddZ(eGdddZ)GdddZ*GdddZ+GdddZ,GdddZ-GdddZ.Gd d!d!Z/d"d#Z0d$d%Z1e2d&kre1dSdS)'zv TAS License Checker ================================================================================================ N)Path) dataclass)ListDictOptional)datetime)ThreadPoolExecutor as_completed webdriverOptionsBy WebDriverWaitexpected_conditions)TimeoutExceptionWebDriverExceptionNoSuchElementException)fuzz)parserc@s~eZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z e ed <d Z eed <dZeed<dZeed<dZeed<dS) SimpleConfigzSimple configurationIhttps://occupationallicensing.justice.tas.gov.au/Search/onlinesearch.aspx website_urlUname_similarity_requiredmax_retry_attemptsrequest_timeout_seconds?delay_between_requestsparallel_browsersT test_modetest_record_limitshow_browser_for_captchaN)__name__ __module__ __qualname____doc__rstr__annotations__rintrr!r#floatr%r&boolr'r(r2r2$/var/www/html/captchaScrapper/tas.pyr#s         rc@sTeZdZUdZeed<eed<eed<eed<dZeed<dd Zd e fd d Z d S)EmployeeRecordz)Holds one employee's information from CSVpayroll_number employee_namelicense_number csv_expiryrcsv_row_numbercCsDt|j|_t|j|_t|j|_t|j|_dS)z)Remove extra spaces and clean up the dataN)r-r5stripr6r7r8selfr2r2r3 clean_dataNszEmployeeRecord.clean_datareturncCst|jo|jS)z9Check if this record has the minimum required information)r1r7r6r;r2r2r3has_required_dataUsz EmployeeRecord.has_required_dataN) r)r*r+r,r-r.r9r/r=r1r?r2r2r2r3r4Es  r4c@seZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<d Z eed <d Z e ed <d Zeed<dZeed<defddZdS) SearchResultz)Holds the result of searching TAS website Not Found found_name license_typelicense_expiry No License name_matches expiry_status error_messagerhow_many_retriesgsearch_time_secondsF was_cachedr>cCs|jdko|j S)z-Did we successfully find license information?rA)rBrIr;r2r2r3 is_successfulgszSearchResult.is_successfulN)r)r*r+r,rBr-r.rCrDrFrGrIrJr/rKr0rLr1rMr2r2r2r3r@Zs          r@c@s^eZdZdZdefddZddZdejfdd Z d d Z d d Z ddZ ddZ ddZdS)OptimizedBrowserManagerzAManages Chrome browser for reliable scraping with CAPTCHA supportconfigcCs,||_d|_d|_d|_t|_d|_dS)NF)rObrowser wait_helpersetup_complete threadingLocklockcaptcha_handled)r<rOr2r2r3__init__ss   z OptimizedBrowserManager.__init__cCs|jrdS|j2|jr WddStd||_t|j|jj|_| d|_tdWddS1s=wYdS)z$Create Chrome browser for processingNz5Setting up Chrome browser for TAS license checking...Tz)Browser ready for TAS license processing!) rRrUprint_create_chrome_browserrPrrOr!rQ_initialize_search_pager;r2r2r3setup_browsers{s  "z&OptimizedBrowserManager.setup_browsersr>cCst}|jjs |d|d|d|d|d|d|d|d|d d g|d d tj|d }|d|S)z+Create Chrome browser with optimal settings --headless --no-sandbox--disable-dev-shm-usagez --disable-gpuz--disable-extensionsz--disable-loggingz--window-size=1280,720z---disable-blink-features=AutomationControlledexcludeSwitcheszenable-automationuseAutomationExtensionFoptionszEObject.defineProperty(navigator, 'webdriver', {get: () => undefined}))r rOr( add_argumentadd_experimental_optionr Chromeexecute_script)r<chrome_optionsdriverr2r2r3rYs           z.OptimizedBrowserManager._create_chrome_browserc Csz/|j|jj|jttj df}| |jttj df}| t dWdSt yC}zt d|d}~ww)z5Initialize the TAS search page with required settingsn//*[@id="ctl00_ctl00_ctl00_ctlMainContent_ctlMainContent_MainContent_ctlOnlineSearch_rblLicenceTypeOptions_3"]v//*[@id="ctl00_ctl00_ctl00_ctlMainContent_ctlMainContent_MainContent_ctlOnlineSearch_rblLicenceTypeCategoryOptions_1"]u,✓ TAS search page initialized successfullyz-ERROR: Failed to initialize TAS search page: N) rPgetrOrrQuntilECelement_to_be_clickablerXPATHclickrX Exception)r<radio_option_3radio_category_1er2r2r3rZs(  z/OptimizedBrowserManager._initialize_search_pagecCs|js||j|jfS)z@Get the browser instance (single browser for TAS due to CAPTCHA))rRr[rPrQr;r2r2r3 get_browsers z#OptimizedBrowserManager.get_browsercCsdS)z0Return browser - no-op for single browser systemNr2)r<rPr2r2r3return_browsersz&OptimizedBrowserManager.return_browsercCs|jrdSz:t|jddd}|r=tdtdtdtdtd td tdtd d |_td WdSWdStyIYdSw)zHandle CAPTCHA if it appearsNcSstdd|tjdDS)Ncss|]}|VqdSN) is_displayed).0iframer2r2r3 szUOptimizedBrowserManager.handle_captcha_if_needed....z%//iframe[contains(@src, "recaptcha")])any find_elementsrro)dr2r2r3s zBOptimizedBrowserManager.handle_captcha_if_needed..= ============================================================u⚠️ CAPTCHA DETECTED!<============================================================z*A CAPTCHA has appeared on the TAS website.z/Please solve the CAPTCHA in the browser window,z/then press ENTER here to continue processing...z#Press ENTER after solving CAPTCHA: Tu-✓ CAPTCHA handled, continuing processing...)rVrrPrlrXinputr)r<captcha_iframer2r2r3handle_captcha_if_neededs*  z0OptimizedBrowserManager.handle_captcha_if_neededcCs0|jrz|jWdStdYdSdS)zClose browser when program endsz)WARNING: Browser failed to close properlyN)rPquitrXr;r2r2r3cleanup_all_browserss z,OptimizedBrowserManager.cleanup_all_browsersN)r)r*r+r,rrWr[r rerYrZrurvrrr2r2r2r3rNps rNc@seZdZdZdedefddZdedefdd Zdedefd d Z d ed edefddZ dededefddZ de de fddZde de fddZdejdede de fddZdejdefddZdejdede fd d!Zd"S)#SeleniumLicenseSearcherz.Searches TAS government website using SeleniumrObrowser_managercCs&||_||_i|_t|_i|_dSrx)rOr search_cacherSrT cache_lock name_cache)r<rOrr2r2r3rWs   z SeleniumLicenseSearcher.__init__namer>cCs~|sdS||jvr|j|S||}|}t|dkr2|ddd|dd}n|}||j|<|S)zEConvert name to standard format for comparison with enhanced cleaningrHr,  N)r_clean_name_for_matchingr:uppersplitlenjoin)r<r cleaned_nameparts normalizedr2r2r3normalize_employee_names    "  z/SeleniumLicenseSearcher.normalize_employee_namecCs|sdSt|}d|}|}g}|D] }t|dkr*t|dkr*q|dr6t|dkr6q||qg}|D]}d|vrQ|d}||q@||q@dd|D}|red|S|S) z>Clean name by removing common variations and formatting issuesrHrr$.rw-cSsg|]}|r|qSr2)r:)rzwordr2r2r3 5szDSeleniumLicenseSearcher._clean_name_for_matching..) r-r:rrrrendswithappendextend)r<rcleanedwordsfiltered_wordsrprocessed_words hyphen_partsr2r2r3rs(      z0SeleniumLicenseSearcher._clean_name_for_matchingcsv_name website_namec Cs|dkrdS|r |s dS||}||}||}||}||kr&dS||kr0dSt||t||t||t||g}t|}||j j kr^dSd|ddS)z.table_updated N)rr~rrorrlr)r<rPrQrr2rr3rs  z)SeleniumLicenseSearcher._wait_for_resultslicence_numberc CsFz|tjd}t|dkrtWS|ddD]q}z]|tjd}t|dkr*Wq|dj}||vr7Wq|dj}|dj}z|dtjd } | j} Wn |dj} Yt|rf|nd | rk| nd |rp|nd d WWSt y} zWYd} ~ qd} ~ wwtWSt y} z t d t | d} ~ ww) z5Parse search results and extract relevant informationrr$NtdrrrwdivrA)rBrCrDzError parsing search results: ) r~rrorr@TAG_NAMEtextr: find_elementrqr-) r<rPrrowsrowcells licence_cellrexpirylicence_type_div licence_typertr2r2r3r sH      z-SeleniumLicenseSearcher._parse_search_resultsN)r)r*r+r,rrNrWr-rrrrr4r@rrr rerrrrr2r2r2r3rs )( ) 7rc@s.eZdZdZedededeefddZ dS)SimpleCSVHandlerz"Reads employee data from CSV files file_pathrOr>c sztj|tddjjjdd_gd}fdd|D}|r5tdd |d tjd g} D]9\}}t |d |d |d|d|dd}| | rt| ||jrtt||jkrttd|jdnq;|s{tdg}t} d} |D]}|j| vr| |j| |q| d7} q| dkrtd| d|WSty} z tdt| d} ~ ww)z Load employee data from CSV fileF)dtype na_filterurH)Payroll Number Employee NameLicense NumberExpiry/Update Datecsg|] }|jvr|qSr2)columns)rzcoldfr2r3rOsz7SimpleCSVHandler.load_employee_data..z" ERROR: Missing required columns: rz Your CSV file MUST have these exact column names: - Payroll Number - Employee Name - License Number - Expiry/Update Date Current columns in your file: z rrrrr)r5r6r7r8r9z!TEST MODE: Processing only first z recordsz2ERROR: No valid employee records found in CSV filerr$zNOTE: Removed z duplicate license numberszERROR reading CSV file: N)pdread_csvr-rr:replace ValueErrorrlistiterrowsr4r=r?rr&rr'rXsetr7addrq) rrOrequired_columnsmissing_columns employeesindexrrunique_employees seen_licensesduplicates_removedrtr2rr3load_employee_dataDsZ       z#SimpleCSVHandler.load_employee_dataN) r)r*r+r, staticmethodr-rrr4rr2r2r2r3rA rc@steZdZdZdefddZddZdedefd d Z dd e d ede fddZ de de fddZ de de fddZdS)SimpleProgressTrackerz1Shows nice progress bar and processing statisticstotal_employeescCs<||_d|_d|_d|_d|_t|_d|_t |_ dS)Nr) r  completed successfulfailedcachedrr last_updaterSrTrU)r<r r2r2r3rWs zSimpleProgressTracker.__init__c Csdtddtd|jdtdtddddd dd ddd d dd tddS)zShow the processing header U=====================================================================================z!TAS LICENSE CHECKER - PROCESSING z EMPLOYEESProgress<15rEmployee<25LicenseStatus<12SpeedzU-------------------------------------------------------------------------------------N)rXr r;r2r2r3 show_headers  ,z!SimpleProgressTracker.show_headerrrc Csl|j-|jd7_|jr|jd7_n|r"|jd7_n|jd7_Wdn1s3wYt}||jdkrEdS||_|j|j d}| |}t ||j d}|j|d}t |jdkrt|jddd n|j}t |jd kr|jdd d n|j} |jrd } n |rd } nd} td|d|dd| dd| dd|dd ddddS)zUpdate progress displayr$Nr"dg{Gz?<z..r CachedSuccessFailed rrrr.0fz/minrHT)endflush)rUr rLrrMr r rrr _make_progress_barrrrr6r7rX) r<rr current_timepercent progress_barelapsedspeed display_namedisplay_licensestatusr2r2r3update_progresssN  . z%SimpleProgressTracker.update_progressr r)widthr>cCs8t||d}d|d||}d|d|ddS)zCreate ASCII progress barrXr[z] z5.1f%r/)r<r)r1filledbarr2r2r3r'sz(SimpleProgressTracker._make_progress_bar elapsed_timeexcel_file_pathcCstddtdtdtd|jtd|jtd|jtd|jtd|jt|jd d d d td ||td|j|dd dtd|tddS)zShow final processing summaryz rzPROCESSING COMPLETED!zTotal employees processed: zSuccessful searches: zFailed searches: zCached results: zSuccess rate: r$rrr4z Total time: zProcessing speed: rz employees/minutezExcel report saved: N)rXr r r rr _format_time)r<r8r9r2r2r3show_final_summarys $z(SimpleProgressTracker.show_final_summarysecondscCsp|dkr |ddS|dkr"t|d}t|d}|d|dSt|d}t|dd}|d|dS) z"Convert seconds to readable formatrr$z secondsizm szh mr5)r<r<minutessecshoursr2r2r3r:s    z"SimpleProgressTracker._format_timeN)r )r)r*r+r,r/rWrr4r@r0r0r-r'r;r:r2r2r2r3r s ,r c@s.eZdZdZedeededefddZdS)SimpleExcelGeneratorz'Creates Excel reports with color codingresultsoriginal_file_pathr>c Cszt|}td}|j|jd|d}|}t|}tj |dd}}d}|j |d|d|j } |j |} | d d d d d dd d} t|jD]5\} } | d| | | ttt| |jsn|| tjnd}tt|ddd}| | | |qM| dd| ddt|t|jd | d d Wdn1swYt|WSty}z tdt|d}~ww)z?Create Excel report - ALL RECORDS (complete processing results)z %Y%m%d_%H%M%S _TAS_Results_z.xlsx xlsxwriter)engine TAS_ResultsF)r sheet_nameTz#8C1E31whiter$centervcenter)boldbg_color font_colorborderalignvalign text_wraprrw 2r NzERROR creating Excel report: )rrnowstrftimeparentstemcopyr DataFrame ExcelWriterto_excelbooksheets add_format enumeraterwriterrr-emptyastypemin set_columnset_row autofilter freeze_panesrqrX)rCrD original_path timestamp excel_path all_resultsrwriterrIworkbook worksheet header_colorcol_numheader max_length col_widthrtr2r2r3create_excel_reportsL /      (z(SimpleExcelGenerator.create_excel_reportN) r)r*r+r,rrrr-rvr2r2r2r3rBrrBc@s|eZdZdZddZddZddZdd Zd efd d Z d e d e fddZ de ed e efddZded efddZdS)SeleniumTASLicenseCheckerzEMain application class - using reliable Selenium with CAPTCHA supportcCst|_t|j|_d|_dSrx)rrOrNrprogress_trackerr;r2r2r3rWls  z"SeleniumTASLicenseChecker.__init__c CsDzzC||}t||j}|t|s%tdWW|j dS| | |}t ||}t|jj}|j||Wn?tyQtdYn;ty}z'tdt|tdtdtdtdtd td WYd}~nd}~wwW|j dSW|j dSW|j dS|j w) z0Main function - this is where everything happenszProcess cancelled by userNz* Process stopped by user (Ctrl+C pressed)z ERROR: z TROUBLESHOOTING TIPS:z1. Make sure your CSV file has the exact column names: 'Payroll Number', 'Employee Name', 'License Number', 'Expiry/Update Date'z)2. Check that Chrome browser is installedz)3. Make sure you have internet connectionz74. Solve any CAPTCHA that appears in the browser windowz<5. Try running as administrator if you get permission errors) _show_welcome_get_csv_file_from_userrrrO_ask_user_to_continuerrXrr_show_captcha_warning!_process_all_employees_sequentialrBrvrrxrr;KeyboardInterruptrqr-)r< csv_file_pathrrCrlr8rtr2r2r3runqsB     zSeleniumTASLicenseChecker.runcCs tddS)zShow welcome messagez!TAS License Checker - Starting...N)rXr;r2r2r3rys z'SeleniumTASLicenseChecker._show_welcomecCsLtdtdtdtd|jjrtdtdtdtddS) zShow CAPTCHA warning to userru#⚠️ IMPORTANT: CAPTCHA HANDLINGrz(The TAS website uses CAPTCHA protection.z5A browser window will open - please be ready to solvez+any CAPTCHA that appears during processing.z/Processing will pause when CAPTCHA is detected.N)rXrOr(r;r2r2r3r|s z/SeleniumTASLicenseChecker._show_captcha_warningr>cCsttjdkrtjdd}td|ntdtdtdtdtdtd d}|ssR          !SOhqL3*