Commit Graph

88 Commits

Author SHA1 Message Date
towards-a-new-leftypol 3ba873b984 Sync minor refactor - Remove a level of indent 2025-02-05 05:17:50 -05:00
towards-a-new-leftypol bffc851999 Sync - create priority queue 2025-02-05 05:14:24 -05:00
towards-a-new-leftypol 6a557c7c5d Sync - ensure we have all the boards 2025-02-04 23:44:38 -05:00
towards-a-new-leftypol 430a199190 sync ensures sites 2025-02-04 22:22:22 -05:00
towards-a-new-leftypol 1113539321 Sync starts by getting latest post for each board 2025-01-30 18:00:42 -05:00
towards-a-new-leftypol 518467c7eb Add sql function to get the last post on each board 2025-01-29 22:50:28 -05:00
towards-a-new-leftypol 7bf61c0dd2 fetch_catalog should be in sql/initialize.sql
- copy it from the experimental remake_fetch_catalog.sql script
2025-01-29 18:27:57 -05:00
towards-a-new-leftypol 34753c176a Move more things into Lib from Main 2024-04-17 07:44:34 -04:00
towards-a-new-leftypol d3495a9d2d Stub in Sync module to eventually keep the db in sync with sites 2024-04-17 07:11:20 -04:00
towards-a-new-leftypol 04abd71582 Implement a simple priority queue mechanism
- pseudo-random index generator to that favours priority indices
2024-04-16 23:49:33 -04:00
towards-a-new-leftypol 8fcea9c84b Explicitly make attachment thumbnail optional 2024-04-12 02:15:42 -04:00
towards-a-new-leftypol cb2da26c64 Fix legacy style attachment file paths 2024-04-11 14:43:15 -04:00
towards-a-new-leftypol 359869984e Handle errors when creating phash 2024-04-11 13:47:44 -04:00
towards-a-new-leftypol 93d789fa65 Can now update archive from url 2024-04-09 21:01:06 -04:00
towards-a-new-leftypol fc321d8531 Abstract out everywhere that references reading src files from local dir
- Now we have two binaries: one to read from local dir one from http
    - both work on the same json api files
2024-04-09 19:40:29 -04:00
towards-a-new-leftypol 67870cab36 Add a way to save a file by doing a GET request over http to DataClient 2024-04-08 02:14:06 -04:00
towards-a-new-leftypol 1c6c1250e3 Begin generalizing backfill code to use either local or http functions 2024-04-05 19:07:20 -04:00
towards-a-new-leftypol 0086dab7f8 Move everything except for main method from Backfill into Lib (for reuse by consumer Main module) 2024-04-05 01:31:26 -04:00
towards-a-new-leftypol 2588724b8c Build another binary, parse settings 2024-04-03 19:57:13 -04:00
towards-a-new-leftypol 02d35a051d Move http-conduit nix build script into Common 2024-03-22 20:29:24 -04:00
towards-a-new-leftypol ac32f29c83 Move JSONSettings into Common.Server 2024-03-22 20:01:02 -04:00
towards-a-new-leftypol 64736a8011 Update common submodule 2024-03-22 19:45:00 -04:00
towards-a-new-leftypol ef7ba97d09 remove debug statements 2024-03-20 15:00:58 -04:00
towards-a-new-leftypol 7c7c6df955 Fix ensuring correct local_idx for posts
- query each thread's max local_idx before processing
2024-03-20 04:56:53 -04:00
towards-a-new-leftypol 335be14f7a Compute the local_idx of each Post 2024-03-19 23:09:40 -04:00
towards-a-new-leftypol 2b1940b401 Update Common 2024-03-18 17:44:25 -04:00
towards-a-new-leftypol 3c3fec27b3 Add attachment index 2024-03-18 09:26:49 -04:00
towards-a-new-leftypol 34bfd77ba6 Remove duplicate search function definition in sql 2024-03-18 08:53:25 -04:00
towards-a-new-leftypol 081f334fad Move get/post definitions out of DataClient into Common (to be used by chandlr-server) 2024-03-18 08:34:35 -04:00
towards-a-new-leftypol c04f8ebe45 make sure search_posts is the same in both scripts 2024-03-02 03:09:34 -05:00
towards-a-new-leftypol 1e0289750d Speed up fetch_catalog
- also add limit to number of rows search returns (to let front-end
  decide how many to display)
2024-03-02 03:02:31 -05:00
towards-a-new-leftypol 09bb17f19c limit search results to 2000 items 2024-03-01 23:50:27 -05:00
towards-a-new-leftypol db6f168c2f commit new search function to initialize, update comment 2024-03-01 19:28:57 -05:00
towards-a-new-leftypol c1a59dc4a6 sql: Return the same rows in search as we do in fetch_catalog
- create a new catalog_grid result type that these functions share
2024-03-01 19:25:47 -05:00
towards-a-new-leftypol 7cb508eb8a Add attachment_idx column, constraint and populate it 2024-02-28 00:29:28 -05:00
towards-a-new-leftypol e73293c4e0 Common submodule, update 2024-02-26 19:26:40 -05:00
towards-a-new-leftypol e290dac0c3 increase POST request timeout 2024-02-26 19:24:42 -05:00
towards-a-new-leftypol 8bb7f413bf Add embed field
- alternative post insert function to set the embed field on existing
  posts that don't have one
2024-02-26 18:20:30 -05:00
towards-a-new-leftypol 9e3c054c53 sql: improve search_posts function 2024-02-19 14:51:56 -05:00
towards-a-new-leftypol 96546a8f7e Fix thumbnail paths 2024-01-25 23:28:59 -05:00
towards-a-new-leftypol acb6f5ddb4 Copy files to destination
- build destination filepath from site, board, thread and old board
  filename
2024-01-24 19:47:07 -05:00
towards-a-new-leftypol 762a60f399 Add board filename and spoiler to attachment 2024-01-24 18:33:07 -05:00
towards-a-new-leftypol 6485acf12b Move AttachmentType into Common (separate git repo) 2024-01-22 13:59:27 -05:00
towards-a-new-leftypol 6f139779ea Additional attachment fields, don't enforce having a phash 2024-01-22 02:19:30 -05:00
towards-a-new-leftypol 5a0cece388 Allow same file 2024-01-22 00:35:39 -05:00
towards-a-new-leftypol 7265bc871b Additional checks to find non-existant files
- runs (may have bugs)
2024-01-21 21:58:27 -05:00
towards-a-new-leftypol 29c8f4eedb WIP: Save Attachments
- nothing works well but the code to compute sha256 and perceptual hash
  on files is there...
2024-01-20 21:04:13 -05:00
towards-a-new-leftypol 576db35727 Remove unnecessary sql file 2024-01-20 09:37:29 -05:00
towards-a-new-leftypol 97884ee425 WIP: Saving attachments
- handle legacy json case for attachement parsing
- compute sha256
- computing phash is TODO
2024-01-19 19:05:02 -05:00
towards-a-new-leftypol a78464fa81 WIP: Saving attachments - refactor procedural stuff into own method for processing files 2024-01-18 14:37:32 -05:00