towards-a-new-leftypol
c1a59dc4a6
sql: Return the same rows in search as we do in fetch_catalog
...
- create a new catalog_grid result type that these functions share
2024-03-01 19:25:47 -05:00
towards-a-new-leftypol
7cb508eb8a
Add attachment_idx column, constraint and populate it
2024-02-28 00:29:28 -05:00
towards-a-new-leftypol
e73293c4e0
Common submodule, update
2024-02-26 19:26:40 -05:00
towards-a-new-leftypol
e290dac0c3
increase POST request timeout
2024-02-26 19:24:42 -05:00
towards-a-new-leftypol
8bb7f413bf
Add embed field
...
- alternative post insert function to set the embed field on existing
posts that don't have one
2024-02-26 18:20:30 -05:00
towards-a-new-leftypol
9e3c054c53
sql: improve search_posts function
2024-02-19 14:51:56 -05:00
towards-a-new-leftypol
96546a8f7e
Fix thumbnail paths
2024-01-25 23:28:59 -05:00
towards-a-new-leftypol
acb6f5ddb4
Copy files to destination
...
- build destination filepath from site, board, thread and old board
filename
2024-01-24 19:47:07 -05:00
towards-a-new-leftypol
762a60f399
Add board filename and spoiler to attachment
2024-01-24 18:33:07 -05:00
towards-a-new-leftypol
6485acf12b
Move AttachmentType into Common (separate git repo)
2024-01-22 13:59:27 -05:00
towards-a-new-leftypol
6f139779ea
Additional attachment fields, don't enforce having a phash
2024-01-22 02:19:30 -05:00
towards-a-new-leftypol
5a0cece388
Allow same file
2024-01-22 00:35:39 -05:00
towards-a-new-leftypol
7265bc871b
Additional checks to find non-existant files
...
- runs (may have bugs)
2024-01-21 21:58:27 -05:00
towards-a-new-leftypol
29c8f4eedb
WIP: Save Attachments
...
- nothing works well but the code to compute sha256 and perceptual hash
on files is there...
2024-01-20 21:04:13 -05:00
towards-a-new-leftypol
576db35727
Remove unnecessary sql file
2024-01-20 09:37:29 -05:00
towards-a-new-leftypol
97884ee425
WIP: Saving attachments
...
- handle legacy json case for attachement parsing
- compute sha256
- computing phash is TODO
2024-01-19 19:05:02 -05:00
towards-a-new-leftypol
a78464fa81
WIP: Saving attachments - refactor procedural stuff into own method for processing files
2024-01-18 14:37:32 -05:00
towards-a-new-leftypol
d48e6e4fe3
minor renaming refactor
2024-01-17 16:59:14 -05:00
towards-a-new-leftypol
df2155a4e4
WIP: Save attachments - Clean up bad approach
2024-01-17 16:56:12 -05:00
towards-a-new-leftypol
b2d03b59c7
WIP: saving attachments - parse api into list of Attachments
2024-01-17 16:53:55 -05:00
towards-a-new-leftypol
f07907df1b
WIP: saving attachments
2024-01-17 16:36:49 -05:00
towards-a-new-leftypol
103bf86017
WIP: actually adding attachments
...
- attachments table was already defined
- change the hash type to sha256
- define sql function to upsert attachment rows
2024-01-16 17:13:55 -05:00
towards-a-new-leftypol
f03d11aafb
Label attachment field as sha256 instead of md5 for futureproofing
2024-01-15 21:00:30 -05:00
towards-a-new-leftypol
ecb444ac21
Use PostsType from Common
2024-01-15 20:59:26 -05:00
towards-a-new-leftypol
54a14156d0
Move PostsType into submodule, commit local changes
2024-01-13 11:22:06 -05:00
towards-a-new-leftypol
af424fb887
Apply linter hints to Backfill script
2023-11-20 03:09:25 -05:00
towards-a-new-leftypol
8820330ff5
return site_id from query_catalog
2023-11-10 02:22:35 -05:00
towards-a-new-leftypol
458573792d
db: add search querying function, fix permissions
2023-11-02 23:22:54 -04:00
towards-a-new-leftypol
8a10ce4798
sql - fetch_catalog needs ORDER BY bump time after all those operations
2023-11-01 20:23:47 -04:00
towards-a-new-leftypol
e83df86725
Add email field to db, data defs
2023-11-01 19:33:26 -04:00
towards-a-new-leftypol
2fcd1d0582
Add subject and name to posts table text index
2023-11-01 19:14:50 -04:00
towards-a-new-leftypol
c297bae697
- Change fetch_catalog and fetch_top_threads functions
...
- neither was correct
- fetch_top_threads doesn't need to be a loop, it will be up to the
application whether or not to load more results. This simplifies
the function, which was sometimes returning more than one row for
each thread_id anyway
- fetch_catalog is now fairly performant, and fetches all relevant
information
2023-10-26 18:19:32 -04:00
towards-a-new-leftypol
7b8b1b2255
commit some sql experiments
2023-10-17 08:19:33 -04:00
615ef6e80b
Rename script (its a shell script not an sql file!)
2023-10-13 23:48:57 +00:00
df94847946
Expect board might have no threads
2023-10-13 20:47:00 +00:00
towards-a-new-leftypol
718ec98475
Add backfill script
2023-10-13 16:42:37 -04:00
towards-a-new-leftypol
86507b4377
Continue even if the thread-specific json file does not exist
2023-10-13 16:42:27 -04:00
towards-a-new-leftypol
6df2ab5757
Only process those directories that actually exist
...
- could be a subset of what's in the database
2023-10-13 16:05:08 -04:00
towards-a-new-leftypol
c18be31289
Can read in the whole site from the api json files
2023-10-13 14:43:02 -04:00
towards-a-new-leftypol
7bab7ea3f3
Change some fields to use Text, more parsing
2023-10-11 14:37:55 -04:00
towards-a-new-leftypol
ce097414db
Can create all the threads on a board now
2023-10-07 16:58:59 -04:00
towards-a-new-leftypol
d78f235c90
light refactoring
2023-10-07 15:04:07 -04:00
towards-a-new-leftypol
8d3a2c05d0
Creating boards if they don't exist
2023-10-07 01:12:30 -04:00
towards-a-new-leftypol
0f77c17e5c
Getting list of threads now
2023-10-06 14:53:39 -04:00
towards-a-new-leftypol
ce76a22e3d
Clean up the code a little bit
2023-10-06 12:45:16 -04:00
towards-a-new-leftypol
5d63d23c79
create site if not exists
...
- get and post implemented in DataClient, and some helper methods
2023-10-06 02:05:58 -04:00
towards-a-new-leftypol
26c57a3bc8
Parse catalog json from file
2023-10-05 02:25:00 -04:00
towards-a-new-leftypol
d6ce32187b
add postgresql to have psql to shell
2023-10-04 18:01:41 -04:00
towards-a-new-leftypol
5af7924779
Read settings.json and list the directory
...
-basically chat-gpt bootstrapping this script
2023-10-04 15:23:58 -04:00
towards-a-new-leftypol
5466849e33
minor thing - rename settings file for script
2023-10-03 22:22:23 -04:00
phil@volguine.com