[ / / / / / / / / / / / / / ] [ dir / baaa / choroy / dempart / jenny / leo / polder / vg / vietnam ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog   Archive

Winner of the 72rd Attention-Hungry Games
/otter/ - The Church of Otter

February 2019 - 8chan Transparency Report
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 1,590 items

The program is now on Python 3! Check v335 release post if you need to update from before then!

Current big job: OR search


YouTube embed. Click thumbnail to play.

893c3e  No.11975[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v344.tar.gz

I had a mixed week, but I got some good hydrus work done. It is basically all misc this week.

highlights

The Client API v1.0 got its last polish this week. I fixed some bugs and added a new parameter to control page selection to the /add_urls/add_url command. Future Client API work will be entirely in regular weekly work. I'd like to add wildcard and namespace tag searching, more system predicates, autocomplete tag searches, optional https to encrypt communication outside a LAN, cookie.txt import, and I am sure many other things. The basic bones are decent now though–I now need only hang new things off it.

Animation scanbars (thePost too long. Click here to view the full text.

12 posts and 2 image replies omitted. Click reply to view.

c9c79e  No.12006

>>11997

>2) It is too fucking big. About 450 million mappings atm, which makes for a gigantic db.

Is this DB size generally problematic to manage on your machine / server?

I don't mind splitting these up at all, seems like a great idea to me. But I'll also guess that there are a good bit more than 450m pieces of CG/drawn art with a known artist out there, never mind photographs and so on. And I'm sure the rest of the world getting internet access will only add more.

Wouldn't it be wise to assume that even if you split the DB up, even some of the primary "objective" databases will hit something like 450m again in relatively short order as more tag sources are ingested?




File: 3f8a8d22364b1de⋯.jpg (111.13 KB, 400x400, 1:1, 1394958218797.jpg)

d5679d  No.9327[Reply]

Here you can ask questions so that the board is not clogged with small threads.

>>6021 has reached its bump limit, so I made a new thread.

612 posts and 116 image replies omitted. Click reply to view.

edc21c  No.11995

What is the most efficient method of shitting on my motherboard?


7ecce6  No.11999

Is time between updates controlled by the server or the client? How trivial is it to change the update rate from 100k seconds to, say, 300 or 600?


f9a293  No.12000

>>11993

Thanks for the help! Now the double number makes more sense.

What I did with tabs is new file tabs looking at everything, and the search is with "inbox" and a copy paste of the string used in the downloader. It happens that if I use "inbox" only as a search there's extra files that should've been in the full…. I'm confusing myself. Let's simplify this: 2 tabs, one is inbox, the other is both inbox and Bulma Briefs, so it should show me (after a refresh) new yet to archive files from the bulma downloader. Let's say it grabbed 10 new files, I can see them all in the "inbox" tab, but the Bulma one might not show some or worse all of them (after a refresh). The way I get them out of the inbox is opening the sorting interface and then right or left clicking as needed.

Hopefully this isn't as confusing :D


7ecce6  No.12002

>>11999

I guess a follow-up question is: is there a technical reason for the 100k second update delay?


f2d6fa  No.12007

Is it possible to merge two separate databases? Is it something I would need to use outside programs for? Do I just open one of the databases as a server?




File: 7a56ef4350b0bb9⋯.jpg (64.84 KB, 445x488, 445:488, 7a56ef4350b0bb96c5b6a4eb80….jpg)

ff2c17  No.11542[Reply]

BUGS THREAD

69 posts and 31 image replies omitted. Click reply to view.

ff2c17  No.11982

>>11974

>>11976

>>11977

>>11978

Thank you, these are great. I will try to check these out this week.


2ab4a9  No.11983

>>11651

Dude, I'm sorry. I am dummy depressed all the time and I feel stupid even making this post.

I didn't reply to you for now over a month cause I was kinda panicking over your reply. I don't use Linux, and I didn't get all the stuff you mentioned, and I wasn't doing so well anyway. I figured maybe later it'd be easier.

But I still can't do it. It's hard. That doesn't justify anything but I don't know.

I'm on version 341 now, and I was about to install version 344, which is the latest right now. I am on Windows 7 Ultimate.

I don't really fetishize ignorance so I don't really need you to say "it's ok to not provide context". I'm not really saying anything for any purpose


460147  No.11984

I'm having an issue with the API. When I GET /get_files/file_metadata with the parameters file_id and only_return_identifiers, I get this error:

Traceback (most recent call last):
File "threading.py", line 916, in _bootstrap_inner

File "threading.py", line 864, in run

File "site-packages\twisted\_threads\_threadworker.py", line 46, in work

File "site-packages\twisted\_threads\_team.py", line 190, in doWork

--- <exception caught here> ---
File "site-packages\twisted\python\threadpool.py", line 250, in inContext

File "site-packages\twisted\python\threadpool.py", line 266, in <lambda>

File "site-packages\twisted\python\context.py", line 122, in callWithContext

File "site-packages\twisted\python\context.py", line 85, in callWithContext

File "include\ClientLocalServerResources.py", line 1255, in _threadDoGETJob
file_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', file_ids = file_ids )
File "include\HydrusController.py", line 588, in Read
return self._Read( action, *args, **kwargs )
File "include\HydrusController.py", line 183, in _Read
result = self.db.Read( action, *args, **kwargs )
File "include\HydrusDB.py", line 931, in Read
return job.GetResult()
File "include\HydrusData.py", line 1577, in GetResult
raise e
include.HydrusExceptions.DBException: TypeError: _GetHashIdsToHashes() got an unexpected keyword argument 'file_ids'

Database Traceback (most recent call last):
File "include\HydrusDB.py", line 557, in _ProcessJob
result = self._Read( action, *args, **kwargs )
File "include\ClientDB.py", line 9265, in _Read
elif action == 'hash_ids_to_hashes': result = self._GetHashIdsToHashes( *args, **kwargs )
TypeError: _GetHashIdsToHashes() got an unexpected keyword argument 'file_ids'

The request returns OK if I get rid of the only_returPost too long. Click here to view the full text.


ff2c17  No.11991

>>11983

No worries m8. I think I read your error report wrong in the first place and assumed you were having 'finding ffmpeg executable' problems, whereas it looks actually like your ffmpeg is ok but could not find the file hydrus was pointing it to. If you had a batch rename thing going on at the time, this may well have been related–maybe hydrus queued the file up to be imported, and then irfanview renamed it before hydrus got around to trying the import. Or maybe hydrus had a hiccup when trying to do some temporary storage stuff and lost track of the file mid-import for other reasons.

If you are still seeing this error over and over, then I am happy to chase it up. But if this was a one-time thing that may have been because of two programs accessing the same space at the same time, no need to worry about it.

You are good to update to 344. Let me know if you run into any more trouble.

Sounds like you are having a shit time. I understand this completely. If you can get a few bucks together, please consider getting this book: https://www.amazon.com/Feeling-Good-Handbook-David-Burns/dp/0452281326/ . It saved my life about ten years ago. You can find it in any big corporate bookstore as well, just say you are buying it for a friend.


ff2c17  No.11992

>>11984

Thank you for this report. I fucked up the variable name here, and this is now fixed for 345. I have a background job already to improve my testing to catch this shit, which has been a blind spot previously.




File: 1426721772716.png (100.78 KB, 1624x1081, 1624:1081, 1327614072601.png)

7f2c0e  No.471[Reply]


Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
643 posts and 191 image replies omitted. Click reply to view.

2532ec  No.11930

Do Hydrus have the ability to auto-follow pruned/deleted chan threads?


3d3331  No.11931

Will this also work with my current version of Windows 3.1?


2bf658  No.11959

>>11931

is this bait?


295e17  No.11968

File: 8dfc7d1abcc0aa8⋯.png (392.22 KB, 1000x563, 1000:563, 8dfc7d1abcc0aa8d07c71bc0de….png)

>>11959

>using any version of Windows after the ones they put NSA keys in


ba4e0d  No.11986

>>471

Table of every artist as a row, and every social media platform as a column




File: 214bc076862edc6⋯.gif (102.07 KB, 450x450, 1:1, 214bc076862edc61448ea65b72….gif)

08b601  No.11953[Reply]

I had a mixed week, but I got some good hydrus work done. I added some neat x/y timestamps to the animation scanbar, made a way to launch file urls with a shortcut, and added barebones .psd import support.

The release should be as normal tomorrow, but perhaps a little late if some IRL stuff comes up.

2 posts omitted. Click reply to view.

4a76c8  No.11967

>>11962

>I wrote this specification of the XCF format in mid-2006. Later it got included in the official Gimp sources, and unless you're explicitly looking for historic data, you should go there and get the specification instead.

Try here instead

https://gitlab.gnome.org/GNOME/gimp/raw/master/devel-docs/xcf.txt


a21095  No.11969


a21095  No.11970


862d70  No.11972

File: 79d8445d2b4039a⋯.png (502.41 KB, 1100x587, 1100:587, 79d8445d2b4039ad72ee107763….png)

There may not be a formal specification for the .kra format it looks like, so source code might be your best bet, or just reversing it yourself as it's just zipped up junk.

Here's the Krita stuff if it's useful:

https://github.com/KDE/krita/blob/v4.1.8/plugins/impex/kra/kra_converter.cpp#L56

https://github.com/KDE/krita/blob/v4.1.8/plugins/impex/libkra/kis_kra_loader.cpp#L196

And here's GIMP's XCF loader code which may be of use if the specification is ever unclear:

https://github.com/GNOME/gimp/blob/GIMP_2_10_8/app/xcf/xcf-load.c#L152


08b601  No.11973

>>11969

>>11970

>>11972

Thanks lads. I'll queue these up and I'll see what I can figure out. Adding 'I can recognise it at least' .psd support wasn't a massive hassle this week, so I expect to do more of this sort of work. But I suspect I'll have to refigure some UI like the system:mime panel checkbox hell if we go too nuts here.




YouTube embed. Click thumbnail to play.

dbbfa9  No.11863[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v343.tar.gz

I had an ok week. The Client API v1.0 is essentially finished, and I did some code cleanup as well.

client api

The Client API now has full 'file search' capability, including fetching file metadata like size and resolution and tags, and also file and thumbnail fetch. As before, the help for these new commands is here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

But for some tiny last cleanup jobs, the Client API v1.0 is now complete. From now on, I will fitPost too long. Click here to view the full text.

27 posts and 5 image replies omitted. Click reply to view.

ad5a96  No.11955

File: 067c964e23215e2⋯.jpg (97.36 KB, 640x641, 640:641, f7215076-af0f-442c-a17b-53….jpg)

>>11947

I agree. at the moment im only testing on lan. I have no problem waiting for https support before releasing. By the way I had a quick question on the file metada endpoint. The example on the site shows the following bit of json for tags :

"service_names_to_statuses_to_tags" : {

"local tags" : {

"0" : [ "blonde hair", "blue eyes", "looking at viewer" ]

"1" : [ "bodysuit" ]

}

}

I was wondering, why are the local tags split up into two separate arrays? Why is bodysuit separate from the other tags? Why not have a single array like so :

"service_names_to_statuses_to_tags" : {

"local tags" : [ "blonde hair", "blue eyes", "looking at viewer", "bodysuit" ]

}

Should I expect any number of arrays or is it always 2?


0a87d5  No.11956

>>11947

Yeah, encryption would be pretty important. I personally don't care because lol openvpn server at home, but that's not normal.


dbbfa9  No.11964

>>11955

The numbers there are the 'status' of that list of tags. In hydrus, a file's tags can have four different statuses:

0 - current - the file has the tag for this service

1 - pending - this tag is pending to the service

2 - deleted - this tag was current but has now been deleted

3 - petitioned - this tag is current and is petitioned for deletion

Pending/petitioned relates to the 'pending' menu in the main hydrus client gui. Repository tags are not fixed as current/deleted until pending/petitioned are commited through that menu.

In writing this out, I realise I fucked up here because a 'local tags' service won't ever have 'pending' tags. I will rewrite this help section to have it refer to a hypothetical tag repository.

Post last edited at

dbbfa9  No.11965

>>11964

To follow up here, if you wish to edit a file's tags with /add_tags/add_tags, you will need to use this more detailed tag status information to figure out if you are 'petitioning' or 'rescind pending' or whatever.

For practical purposes, if you want to calculate what tags the file currently 'has' for easy phone display, you can go:

tags = ( current ∪ pending ) - petitioned

Petitioned tags are pretty rare, so you can probably just go ( current ∪ pending ). You'll then be unioning those sets across all given service_names to get a file's 'all known tags' as you would see in the client.


b58bd7  No.11971

>>11964

>>11965

Ah. I forgot tags can have statuses. Thanks for clarifying




File: ed3f745dbd39b5d⋯.jpg (4.66 MB, 4000x2715, 800:543, shutterstock_89245327.jpg)

12f73e  No.4475[Reply]

How about a thread for discussing/creating/sharing parsing scripts?

I made one for md5 lookup on e621.net (actually I just modified Hydrus_dev's danbooru script). Let me know if I did anything wrong with it, I'm pretty clueless… but it seems to work fine.


[32, "e621 md5", 1, ["http://e621.net/post/show", 0, 1, 1, "md5", {}, [[30, 1, ["we got sent back to main gallery page -- title test", 8, [27, 1, [[["head", {}, 0], ["title", {}, 0]], null]], [true, true, "Image List"]]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-general"}, null], ["a", {}, 1]], null]], ""]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-copyright"}, null], ["a", {}, 1]], null]], "series"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-artist"}, null], ["a", {}, 1]], null]], "creator"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-character"}, null], ["a", {}, 1]], null]], "character"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-species"}, null], ["a", {}, 1]], null]], "species"]], [30, 1, ["we got sent back to main gallery page -- page links exist", 8, [27, 1, [[["div", {}, null]], "class"]], [true, true, "pagination"]]]]]]

45 posts and 16 image replies omitted. Click reply to view.

354bf6  No.11616

>>4475

Can any custom parsers handle logins? Like the twitter gallery situation is still out of the picture and has been for a few months now. Fur Affinity and InkBunny if parsers are made but without logins will barely scrape any content as well. I know Hdev said FA gallery parser is coming but without login support it's hardly worth the work to make one imo.


d54ab2  No.11696

>>11616

You can make your own login scripts but IMO it's not worth it, especially when the site makes heavy use of javascript or captchas.

Instead, just copy the cookies from your browser session to get logged in.

>network>data>review session cookies

Inkbunny needs "PHPSESSID"

For other sites just copy anything that has any login looking things like username, base64 or hex string values until it works.


bb00d3  No.11698

What do I need to learn about HTML or JSON so I can make downloaders?


601c51  No.11814

I'm trying to use the iqdb-tagger python script, but there is a PermissionError when it tries to write to windows temp folder. Anyone know how to fix? I tried setting the iqdb-tagger-server.exe, iqdb-tagger.exe and python.exe to run as administrator but it doesn't help. I'm on Windows 10.

https://github.com/rachmadaniHaryono/iqdb_tagger


9d5f02  No.11886




File: 4ef4a595735e470⋯.gif (1.51 MB, 500x375, 4:3, 4ef4a595735e4708935b207a7d….gif)

72d6d5  No.11858[Reply]

I had an ok week. The Client API v1.0 is just about finished, and as hoped I did some long-overdue code cleaning. Also are some tiff/webm detection fixes and slightly smoother media viewer browsing.

The release should be as normal tomorrow.



YouTube embed. Click thumbnail to play.

a51686  No.11805[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v342.tar.gz

I had a good week. There are several new ui features and bug fixes, and webp and tiff now have basic support.

tiff and webp

Tiff files are now supported! Tiff is an old format typically used for photographic and scientific uses (think 12,000x12,000px images of the moon). They'll import to hydrus like any other image and will work on the duplicate file system. I tested 24bit and 8bit (monochrome), but could not find a 48bit example–even the easily accessible NASA stuff was all 24bit–so I don't know if high colour-depth tiffs work. If you have an example of one, please send it to me, and if it doesn't work in hydrus, I'll see if I can figure it out. EDIT: Some tiffs are not supportPost too long. Click here to view the full text.

25 posts and 5 image replies omitted. Click reply to view.

a51686  No.11853

>>11838

A user helpfully pointed me to some here:

http://www.brucelindbloom.com/index.html?RGB16Million.html

They render ok, so I guess a clever sRGB conversion is happening. I also added support for 'MM' tiffs for tomorrow. Should be complete tiff support.


a51686  No.11854

>>11840

>>11841

If you have a lot of processing still to do, I recommend you do not do explicit processing from that review services frame. It can only be cancelled through the UI, and after a long time, there is a decent chance the UI will deadlock until the job is done (this is due to some shit code by me that will take a bit to clear up). If the job still has 15 hours of work left, the whole program can hang that long.

I recommend you only let processing happen during shutdown, where it has a timer, and idle time, where moving the mouse will tell it to safely cancel.

That review services button is hidden behind advanced mode and is only ever really a pleasant experience when I am doing some testing or when on a fast SSD without much total processing left to do.


a51686  No.11855

>>11842

Yeah, for some complicated websites, an import 'item' produces more than one file or a new page to pursue. It is a technical pain in the ass, but for now, the x/y progress on an importer refers to 'import items processed' not 'potential files imported'.

If you would like to check those 'already in db' files, you should be able to right-click on them and say something like 'open these in a new page'. Since in that note it says '10 seconds ago', they are almost certainly duplicates from above. (I don't know anything about nijie.info, but yeah, it looks like the _0 url without diff/main is a 'cover image' for the subsequent mini-gallery?)

Again, some galleries give the same file twice, even on their nice APIs. I don't know why this parser pulls that 'cover', but I did not write it, so I can't confidently say this strategy isn't needed to catch edge cases. The nuts and bolts of this stuff are frequently fuckery duckery doo, particularly on Japanese sites.


a51686  No.11856

>>11844

Thanks, that is interesting. I assume the lossless compression mode is functionally 4:4:4, right? But any lossy encoding is coerced to 4:2:0?

My hesitance for webp in the past is that it still is sRGB, so I don't know how it is useful as we move to HDR in the coming years. Maybe they will do like webm and add new accepted encoders, but I dunno. It doesn't seem to be taking the world of internet browsers and phones by storm as it is. HEIF and FLIF seem to have their good points, but they are still similarly meme status for now.

I'll play with animated webps a bit when I find time, as I'd really prefer not to use shitty gifs for animated thumbs, and I don't want to go completely nuts with apngs either.


a51686  No.11857

>>11846

Thanks, I am glad you like it. When you get an app ready and want to share it about, please send me a link to it and I'll put it up in my release posts and help!

Current plan for Client API is to get a simple 1.0 out the door. This should be done tomorrow for 243, where basic file search and file/thumbnail retrieval are finished. You'll be able to find all the files in a client with the tags 'blue eyes' and 'blonde hair' and then shows thumbs and files and tags. It should be possible to replicate very basic booru functionality. After the release tomorrow, please check the updated help here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

Which will have the last '/get_files/…' queries all filled out.

With the 1.0 done, Client API work will then be fit into the regular weekly 'small jobs' schedule. If someone wants the ability to archive files or search by duration, I'll see if I can squeeze it into a small job. If you have ideas for your app, please submit them any way that is convenient–these release threads are fine, and you can also email me or even DM me on discord on Saturday afternoons US time if you want to talk live.




YouTube embed. Click thumbnail to play.

d76d70  No.11732[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v341.tar.gz

I had a great week after being ill at the end of the previous week. There is a bit more than usual in this release because of the extra time. The Client API has some new commands, and there are a variety of fixes and stability improvements.

client api

The Client API now does some fun stuff. If you would like to play with one user's browser extension, which recognises hydrus-compatible URLs–including whether you have the page's file already–and lets you send them straight to hydrus for downloading, please check it out here:

https://gitgud.io/prkPost too long. Click here to view the full text.

41 posts and 3 image replies omitted. Click reply to view.

d76d70  No.11827

>>11783

Thanks. I see one of those definitions is unhappy with UPnP. Feel free to delete the three upnpc executables from install_dir/bin–they aren't needed to run the program, only if you want to explictly tell some hydrus service to keep a NAT port open for itself. They are just copies of the exes available here:

http://miniupnp.free.fr/

The others, I suspect, are as I said before. I haven't had a true positive here yet, but I'd appreciate any future information you have here.


d76d70  No.11828

>>11785

Thank you, I really appreciate this recommendation and your thoughts about it. I will definitely revisit this when work on hydrus multi-page support starts.


d76d70  No.11829

>>11789

Thank you for this report. My dark mode is a bit of a fudge–I only have control over the colour of my custom objects, whereas everything else (like window borders and backgrounds and button colours) is supposed to be inherited from your OS's current system colours. If you have a window manager that changes your OS colours, hydrus's dark mode switching can be helpful to make sure that my custom objects also change with it.

There's a very speculative long-term plan to move from wxPython to Qt. If this ever happens, theming support will get significantly better, but until then it is all duct-tape, I am afraid.


d76d70  No.11830

>>11793

Thank you for this report. Are the updates processing during idle time or shutdown? If you have problems during idle time, can you change your settings under options->maintenance and processing to be shutdown-only, and, say, only 5 minutes? If that relatively small amount of work can go through, can a bit more, like 15 mins? Is there a significantly 'cleanup' lag, like 5 mins of work takes 3 mins to commit?


d76d70  No.11831

>>11810

Thank you. That is very strange. Here's SQLite's and Python's info on this:

https://sqlite.org/tempfiles.html#temporary_file_storage_locations

https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir

So, for some reason, a different folder was not permitting a program launched from it to access any system temp location. Windows has simpler temp structure than Linux, so perhaps this really just meant stopping you from accessing the one under AppData/Local/Temp.

Could somehow that directory (or say the client.exe) have got a compatibility mode or some other protected state applied to it? I am afraid this is way beyond my expertise.

Post last edited at



File: e3d80057357881e⋯.gif (826.95 KB, 500x275, 20:11, e3d80057357881e6faf3fc41e3….gif)

4552a2  No.11798[Reply]

I had a good week. I fixed several bugs (including I think the Linux >0 distance similar files search crash, finally!), did some neat ui work like better tag autocomplete responsivity, more stable and slightly smoother video rendering, and more 'open urls' options on multiple selections, and added some basic webp and tiff file import support.

The release should be as normal tomorrow.

1 post omitted. Click reply to view.

7f1978  No.11801

Sauce, devbro?


4552a2  No.11803

>>11801

Dunno, sorry! It was just in my inbox this week.

But perhaps source could be your life, one day.


a3f3b7  No.11804

>(including I think the Linux >0 distance similar files search crash, finally!)

Was that at all related to any of my posts:

>>11736

>>11758

>>11797

Because I was having an issue on linux crashing. In the process for searching for duplicates an SQL statement similar to

SELECT phash_id, phash, radius, inner_id, outer_id FROM shape_perceptual_hashes NATURAL JOIN shape_vptree WHERE phash_id IN (1, 2, 3);
would be executed and cause a segfault, not just for hydrus but anything that could execute an sqlite statement. Apparently Sqlite 3.27.1 (and possibly some earlier versions) had a bug where if the size of a list exceeded two items it just crashed. This appears to be fixed in sqlite 3.27.2.

I imagine based on the code I looked at having a search distance greater than 0 would make it far less likely to happen if not impossible.


4552a2  No.11806

>>11804

There are quite a few places where the client fetches things with lists up to 256 items long, so if you found that just loading some files and tags was ok, I _think_ you weren't getting hit by that exact issue.

But yeah, your posts are the bug I think I fixed. Only affected some Linux users, and I could not reproduce it. I worked one on one in the week with a user who had the crash and we ultimately figured it was a problem with SQLite hanging on to the phash while I did a Hamming Distance search on it (which requires some low level C++ struct calls and then some mickey-mouse CPU-level binary counting). Something to do with memory access violation is my best guess. Python usually can't crash by itself. Previously, I was doing hamming searches interleaved with the sqlite call iterating each row, but when I separated it into completed batches, so sqlite was 'done' with the rows, no crash happened.

The 0 search distance could do a direct lookup without the hamming check, and that was running fine. It just seemed to be the combination of iterator on sql results and then low-level access on 'bytes' type response from that that caused the crash. Core dumps suggested the crash was in sqlite code, so I guess it was still hanging on to a bytes buffer or something that wasn't cleaned up. Could indeed still be a (different) bug in sqlite, or python's wrapper for it, but I dunno for real.

Let me know how 342 works for you!


62a2f0  No.11808

>>11801

Owarimonogatari

>>11803

wholesome




File: d7e685360192f2a⋯.png (20.52 KB, 722x449, 722:449, api-image-for-blog.png)

b45f6a  No.11626[Reply]

ITT: We propose new features that can be solved by using the API, and recommend new API commends for it

b45f6a  No.11627

>>11626

*commands


59ad1a  No.11761

Here are some features that I can think of that are useful

1. Image tag and metadata "hash"

Image data hash, so if you were to download metadata from the same file twice, and the metadata has not changed, the hash would be the same, telling the user to not download it again and waste precious bandwidth.

2. IPFS H2H API

IPFS operational API, so that you can create folder or file hash of any search results, and be able to pin or unpin them at will. This will be useful in the long term when people use it for Hydrus-2-Hydrus (H2H) P2P file share.

3. De-duplication API

internal APIs for file de-duplication software like Pippy360 for images and Chromaprint for music, and then allow external APIs be called for IQDB-like file search and hash download capabilities. (requires APIs for uploads first)

4. Headless Drone API

Headless "master+drone" Hydrus APIs, that allows one "master" to distribute downloader scripts and "download orders" to them, and the "drones" will return either downloaded images and tags, or an error message, back to the "master"


59ad1a  No.11764

Also having an API similar to this would be useful https://github.com/mserajnik/hydrusrv/blob/master/API.md


d9f85b  No.11802


d9f85b  No.11843

>>11764

This is a one-upping https://github.com/rr-/szurubooru/blob/master/API.md now THIS is an API that we can learn from




File: d49758fd3e9ccfe⋯.png (54.11 KB, 2105x826, 2105:826, logo.png)

4ab0be  No.11035[Reply]

I was thinking since Pixiv has such a weird, complicated, and sometimes awfully designed site, us Pixiv bros should help each other out on figuring out ways to make it easier to use with Hrdrus. Post any tips, scripts, regexes, setups, etc.

4 posts omitted. Click reply to view.

1aa8d8  No.11087

Speaking of pixiv, I'm new to hydrus but I can't seem to download via artist lookup. I tried using both the downloaders on the easy-import pngs but still no luck. My pixiv account is linked properly.

Heres the ones I was using.

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/blob/master/Download%20System/All-in-Ones/Single-Sites/easy-import-new-pixiv-layout-artist-gallery-2018.09.21.png

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/blob/master/Download%20System/All-in-Ones/Single-Sites/easy-import-pixiv-artist-lookup-2018.11.29.png


4ab0be  No.11759

File: 19594f5e0a1c881⋯.jpg (56.6 KB, 446x446, 1:1, pic 1.jpg)

File: 45fdedcae48bb05⋯.jpg (111.88 KB, 865x503, 865:503, pic 2.jpg)

Man this is pretty annoying needing to translate all this using siblings in pic 1. It would be nice if Hydrus could pick up the already translated tags on pixiv in pic 2 instead.


52c4b5  No.11760


323f16  No.11786

>>11059

>>11054

>>11041

Not just that, but "artists' favorite tags" and "tags' main users", from that we can create a network graph for better tag and artist discovery


4ab0be  No.11787

>>11760

Those are nice scripts but my main issue is I need to keep turning new Japaneses tags into English using sibling. It just seems like such a clutter to keep turning Japaneses tags into English when they are already translated in that >>11759 2nd pic, unless there's a better way to deal with this and I'm going something wrong.




File: 60e0e08ae58bb15⋯.jpg (334.26 KB, 443x575, 443:575, 60e0e08ae58bb15563f3b41807….jpg)

47970f  No.11716[Reply]

I had a great week after being ill at the end of last week. 341 is a big release because of the extra time. There are several new Client API commands, bug fixes and more memory reduction, and some important stability improvements, particularly for Linux.

The release should be as normal tomorrow.

4 posts omitted. Click reply to view.

1f27c4  No.11721


1f27c4  No.11722

>>11721

Trying to find a kanban for people to put their ideas on the board.


d8312b  No.11724

>>11718

the program tends to eat memory when you heavily use it, let me give you a picture of what I mean

I save every thread on 4chan that catches my interest, at some point the reposts start to overtake the new images to a point that this is a viable option, thy im getting close to the edge of I need some delete functionality for telling me what the file was deleted for. that aside, right now, over the coarse of 2 months I have created around 6400 watchers, adding somewhere around 150-200k images, almost none of them are displayed, with around 100k displayed.

the 100k displayed images takes about 1.5-2gb, I know this from the time I culled the watchers, my current session is 4.3gb and bloats out to 6.5gb on saves.

I have absolutely no idea what takes ram, if they were all current watchers, ok, I would believe that, but from experience current watchers for all the threads adds up to around 500mb or less, so for some reason, the program is using an ungodly amount of ram for reasons that are unclear because just storing all of the the data in text form could not possibly add up to over 20mb much less what I estimate at being over 2gb with a doubling in save.

I should also note that I come from a session that was hitting 14gb on startup and when a save happened would bloat to 18+gb

depending on where hdev is focusing, memory reduction has many different ways to happen.


251015  No.11725

>>11724

>the program tends to eat memory when you heavily use it

I think that goes for most programs.

When I had 8GB of RAM my average overall usage was around 3GB, but now that I have 16GB it averages 6 or 7 even though I'm doing the same things.


47970f  No.11730

>>11718

Bare client is about 115MB ram on Windows, so the absolute minimum is pretty low compared to what many users see. I am about 490MB on my IRL laptop that has about 17k items in its main session (mostly thread watchers).

The python 3 change lead to some pseudo-memory-leaks, so that was a recent chunk I was able to recover, and I am making an important db-memory change in 341 today.

I would appreciate continued feedback here. There isn't a huge amount I can do atm about clients that have hundreds of thousands of pending URLs in their session, but PTR update processing and some other db stuff should be much less traumatic now. There's still more to do, and I still plan to make a memory profiler this year to draw some pie charts or something and squeeze this further.




YouTube embed. Click thumbnail to play.

9ecea0  No.11606[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v340.tar.gz

I had a great if busy week. The Client API does more, several bugs are fixed, some new features and options are in, and stability and memory use should be a bit better.

client api

This is still somewhat advanced, but users who are interested may want to start looking in.

The first version of the Client API last week went ok! There were a couple of bugs, but thanks to feedback from some advanced users, I've improved reliability and carved out a better spec. This week brings JSON everywhere, fixes the add_file crash, and adds two neat calls:

/add_urls/get_url_files nowPost too long. Click here to view the full text.

30 posts and 17 image replies omitted. Click reply to view.

9ecea0  No.11704

File: 4e55e5c587407d0⋯.png (27.1 KB, 1135x188, 1135:188, p.png)

>>11688

Hey, Pixiv is a slightly odd site, so there are two caveats here:

1) Ugoiras are not supported, so they will most likely get 'ignored' status.

2) Any 'manga' page will branch out to create new file import entries in the queue, leaving a stub entry.

Pic related has an example of 2. It is an unfortunate artifact of how Pixiv have decided that some 'mode=medium' URLs refer to one file while others refer to multiple. Although the initial https://www.pixiv.net/member_illust.php?illust_id=73135351&mode=medium url here gets 'successful', meaning it was a queue item processed successfully, it doesn't have any files itself.

Could the manga surplus explain your bad file counts here? It sounds maybe reasonable for the 2033 vs 1731, but not for the 307 to 21. Can you tell me more about what that 307 means? Where did you see it, and where did the 21 come from?


9ecea0  No.11705

>>11699

>>11700

Thank you. Your situation does not sound crazy–2k thumbs is fine, for example. I have now done more for this for 341. In particular the db will now spool temp data to disk if it gets too big. I expect many users to see significantly less memory use during PTR work.

Please keep me updated here.


a4c48c  No.11711

File: 4722a45faf3ad08⋯.png (45.6 KB, 1081x532, 1081:532, client_2019-02-25_16-07-41.png)

>>11704

Its possible that its manga and ugoria, but the problem I have is that it says there are 307 new images, but when I tell it to present new, only 21 show up.

my concern is that it's filing things that may, in the future, be parseable under an I have it, and when it rechecks it thinks I have it and skips over it.

>>11703

for this, sorry, im not getting them from exhentai, im finding new artists, usually in a gallery for an artist they will post where you can find their work directly, and I plug it in to download.

effectively, I download a gallery of something in the gallery downloader, and a few weeks later I see an update to it, so I regrab it in the gallery, the best way to put it is these are impulse downloads, not something I want to set a subscription up for. I think this shows off what I would like to avoid fairly well, if there was a way to recheck without adding the same search again, that would be great.


9ecea0  No.11712

>>11663

>>11636

I did some work on this today and it went well! I have a 'thumbnail experiment mode' in for v341 under help->debug->gui so you can try it out yourself. It loads the fullsize and resizes in memory. There is not a human-noticeable difference in speed. On my dev machine here, which is a few years old, I added some timers and got (once the file data was in memory) approx 500µs to load a resized thumb and 1.8ms to load a fullsize thumb and scale it, which I was impressed by. I have a Nvidia 1030 here to drive my 4k monitor, so perhaps that is accelerating this.

I am willing to experiment more here, so I will mention it in my release post and see what you and other users find.

After looking at the code, I think that in exchange for simplifying the file system by only having one set of thumbs, I could justify making that single thumb more intelligent in how it swap outs bad-size thumbs when needed (i.e. basically removing the master rather than the resized). So, I think we can get the best of both worlds here, saving space and keeping things fast. I have a job set out for it now, so I'll try to chip away at this in the coming weeks. Thank you for bringing this up.


e6d33c  No.11713

>>11705

Last bit of info is I currently have about 100-110 gallery download ques (none running), that would send the pending tags through the roof but since Hydrus only displays 1 at at time then I dont think this would do much for thumbnails displayed. It would certainly push up high use of the CPU, disk writing and some ram but only when Hydrus is busy. Despite all the stuff I listed I'm only around 450 MB of Ram.




Delete Post [ ]
[]
Previous [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Cancer
[ / / / / / / / / / / / / / ] [ dir / baaa / choroy / dempart / jenny / leo / polder / vg / vietnam ]