o jh6@sBdZddlZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZedZeejZd d Zd d Zd dZe defddZdefddZdefddZdedefddZdedefddZdYdefdd Zdefd!d"Z d#efd$d%Z!d&efd'd(Z"dedefd)d*Z#dedefd+d,Z$d-efd.d/Z%defd0d1Z&defd2d3Z'd4efd5d6Z(d&efd7d8Z)d9ed&efd:d;Z*d&efdefd?d@Z,d&efdAdBZ-dCefdDdEZ.d&efdFdGZ/dHefdIdJZ0d&efdKdLZ1dMefdNdOZ2d&efdPdQZ3dRefdSdTZ4dZdVdWZ5GdXdUdUZ6dS)[a data_store.py All persistence lives here: premarket snapshots, intraday snapshots, trade records, and bot state (positions, watchlists, cooldowns). Design notes: - Uses simple JSON-lines files per day for premarket/intraday data so they can be analyzed later with pandas without a database dependency. - State files (positions.json, watchlist.json, cooldowns.json) are written atomically (write to tmp file, then os.replace) to avoid corruption if the process is killed mid-write. - A single filelock-style guard (using a lock file + fcntl) protects state writes in case of accidental concurrent processes, matching a lesson learned in prior systems where duplicate processes raced on shared state. N)datetimedate)Path)contextmanager) get_config) get_logger data_storecCsLtd}t|dt|dt|dt|dt|dt|ddS) N data_storage premarket_dir intraday_dir trades_dirfast_engine_dir state_dirbreakout_scanner_dir) premarketintradaytrades fast_enginestatebreakout_scanner)rBASE_DIR)cfgr data_store.py_dirs#s       rcCs"tD] }|jdddqdS)NTparentsexist_ok)rvaluesmkdir)drrr ensure_dirs/sr!cCs tSN)rtoday isoformatrrrr _today_str4s r% lock_pathc csz|jjdddt|d%}t|tjz dVWt|tjnt|tjwWddS1s6wYdS)NTrw)parentropenfcntlflockLOCK_EXLOCK_UN)r&lfrrr _file_lock8s ""r/pathcCsl|jjddd||jd}t|d}tj||dtdWdn1s)wYt ||dS)NTrz.tmpr')indentdefault) r(r with_suffixsuffixr)jsondumpstrosreplace)r0datatmpfrrr_atomic_write_jsonCs  r>c Cs|s|Szt|d}t|WdWS1swYWdStjtfyG}ztd|d|d|WYd}~Sd}~ww)NrzFailed to read z: z. Returning default.)existsr)r6loadJSONDecodeErrorOSErrorlogwarning)r0r3r=errr _read_jsonKs (rGrecordcCsX|jjdddt|d}|tj|tddWddS1s%wYdS)NTra)r3 )r(rr)writer6dumpsr8)r0rHr=rrr _append_jsonlVs "rMsymbolcC:tdtd}t|d|}t||dS)Nr.jsonl timestamprNrr%rutcnowr$rMrNrHr0rrrappend_premarket_snapshot`rVcandidates_20.json candidatescCs(tdtd|}t||dS)Nr_rr%r>)rYfilenamer0rrrwrite_premarket_candidatesfsr]cCs$tdtd}t||dS)a[FEATURE 2026-09-16] breakout_scanner.py's own ranked candidate list -- deliberately a SEPARATE file from write_premarket_candidates above (different directory, different name) so it can never be confused with or overwrite sip_bot's own live candidate output. Date-stamped filename, same convention as write_premarket_candidates, so a day's breakout_scanner run and sip_bot run can be diffed later.rz _scanner.jsonNr[)rYr0rrr!write_breakout_scanner_candidatesksr^top_listcCs.tdd}t|d}t||dS)zDWrites the canonical top_stocks.json referenced by the whole system.rtop_stocks.json) updated_atstocksN)rrrTr$r>)r_r0payloadrrrwrite_top_stocksvsrdreturncCs(tdd}t|dgi}|dgS)Nrr`rb)rrGget)r0r;rrrread_top_stocks}s rgcCrO)NrrPrQrSrUrrrappend_intraday_snapshotrWrhcCsrtdd}ttddt|i}dti|||<t||WddS1s2wYdS)zMirrors the latest intraday state for a symbol into intraday.json (a single mutable snapshot file, distinct from the historical .jsonl log), matching the project's requested intraday.json state file pattern.rz intraday.jsonzintraday.json.lockraN)rr/rGrrTr$r>)rNrHr0all_datarrrwrite_intraday_jsons   "rjtradecC$tdtd}t||dS)Nr _trades.jsonl)rr%rM)rkr0rrrappend_trade_recordrncCs8tdtd}dti|}t||dS)a^ [FEATURE 2026-09-11, extended 2026-09-11 v3] One JSON line per ranked candidate, every poll cycle, for EVERY health-eligible premarket_20 symbol -- not just the top-5 shortlist that gets a real fast_entry_gate decision. `shortlisted`/`confidence_rank` distinguish "ranked but not evaluated this cycle" (`shortlisted: False`, `state`/`should_enter`: None) from "made the top-5, got a real BUY/WAIT/REJECT" (`shortlisted: True`). Per explicit instruction ("log all bot decisions so we can review and adjust what is not working", later extended to "make sure we have all the indicator values logged during the life cycle of the symbol"): the whole point is being able to load this file into pandas afterward and see exactly what the engine saw for every symbol at every poll, whether or not it ever became a real trade (which append_trade_record above already covers) or even made the shortlist. Each record also carries a full raw-indicator snapshot (fast_pipeline.snapshot_dict() -- every stream_features.StreamFeatures/FastPredictionReading field, not just the classified labels) alongside the classified summary fields kept here for quick filtering without unpacking the nested snapshot. Same per-day .jsonl pattern as premarket/intraday snapshots -- pandas-readable, no database dependency. Volume note: at ~20-30 eligible symbols x poll_interval_seconds_intraday=5s, this is meaningfully larger than the old shortlisted-only version (was ~5 records/cycle, now ~20-30/cycle) -- expect a materially bigger file per trading day. rrPrRNrSrHr0rrrappend_fast_engine_decisionsrqcCs:tddtd}dti|}t||dS)a [FEATURE 2026-09-11] One JSON line per open-position poll where position_manager._check_momentum_fade_exit() successfully computed features -- added the same day that check was introduced, after discovering there was no way to validate its thresholds (velocity_stall_threshold_pct_per_sec / pressure_threshold / min_ticks_sub, see config.json's stop.momentum_fade_exit) against real data: a full replay of 2026-09-11's 16 trades against the new check turned out to be impossible, because append_fast_engine_ decision() above only dumps CANDIDATE-scan snapshots, which stop the moment a symbol becomes an open position (fast_pipeline.py has no reason to keep ranking something already held) -- so no tick-level velocity_sub/pressure.score history exists for any open-position hold window from that day. This closes that gap going forward: every poll where the check actually evaluates a reading (not the early-return guard cases -- disabled, missing bars_sub/quote, insufficient_data -- which aren't informative for tuning) gets one row here, whether or not it fired, so the thresholds can be tuned against real fills instead of Yahoo-1-minute-bar approximations next time. Separate file from fast_engine's per-day dump -- different schema/purpose (a single open position's fade check vs. ranking all eligible candidates) even though it lives in the same directory. rmomentum_fade_rPrRNrSrprrrappend_momentum_fade_snapshotsrsrcCrl)Nrz _summary.jsonr[)rr0rrrwrite_trades_summaryrortc Cstdtd}|sgSg}t|d1}|D]%}|}|s$qz |t|Wqtjy@t d|YqwWd|S1sLwY|S)u Reads every trade record appended today via append_trade_record() — the complete, per-trade history for the day. [BUGFIX 2026-08-17] This is deliberately separate from PositionManager.positions, which is keyed by symbol and therefore only retains the MOST RECENT trade for any symbol traded more than once in a day. monitor.py's end-of-day summary previously built itself from positions.values() and silently undercounted days with repeated same-symbol round trips (verified: a session with 19 actual closed trades logged a summary of only 10 — one per unique symbol). Use this function instead for anything that needs a true count of the day's trades or an accurate total P/L. rrmr?z(Skipping malformed trade record line in N) rr%r@r)stripappendr6loadsrBrDrE)r0rr=linerrrload_today_tradess(    rynamecCstd|S)Nr)r)rzrrr _state_path r{cCs`ttdi}t}i}|D]\}}|ddvr |||<q|dd|r-|||<q|S)Npositions.jsonstatus)r)closing entry_time)rGr{r%itemsrf startswith)rawr#filteredsymprrrload_positionss r positionscC@ttdttd|WddS1swYdS)Nzpositions.json.lockr}r/r{r>)rrrrsave_positions"rcCsttdggdS)Nwatchlist.json) premarket_20final_10rGr{rrrrload_watchlist!sr watchlistcCttd|dS)Nrr>r{)rrrrsave_watchlist%rcCttdiSNzcooldowns.jsonrrrrrload_cooldowns)r|r cooldownscCrrr)rrrrsave_cooldowns-rrcCr)Nintraday_health.jsonrrrrrload_health_state6r|r health_statecCr)Nzintraday_health.json.lockrr)rrrrsave_health_state:rrcCr)Nsession_state.jsonrrrrrload_session_state?r|rrcCr)Nzsession_state.json.lockrr)rrrrsave_session_stateCrr SingletonLockcCs ttdS)uEnsures only one monitor.py instance runs at a time — prevents the duplicate-process-racing-the-same-account failure class.zmonitor.pid.lock)rr{rrrracquire_singleton_lockHs rc@s*eZdZdefddZddZddZdS) rr0cCs||_d|_dSr")r0_fh)selfr0rrr__init__Os zSingletonLock.__init__cCst|jjjdddt|jd|_z t|jtjtjBWn t y(t dw|j t t |j|S)NTrr'z{Another monitor.py instance already holds the singleton lock. Refusing to start a second instance against the same account.)r0r(rr)rr*r+r,LOCK_NBBlockingIOError RuntimeErrorrKr8r9getpidflush)rrrr __enter__Ss  zSingletonLock.__enter__cCs(|jrt|jtj|jdSdSr")rr*r+r-close)rexc_typeexc_valexc_tbrrr__exit__aszSingletonLock.__exit__N)__name__ __module__ __qualname__rrrrrrrrrNs )rX)rer)7__doc__r6r9r*timerrpathlibr contextlibr config_loaderr logger_setuprrD__file__resolver(rrr!r%r/r>rGdictrMr8rVlistr]r^rdrgrhrjrnrqrsrtryr{rrrrrrrrrrrrrrrrsZ            $