Fan made projects related to MLP

Search /collab/ threads

Password  (for post and file deletion)

File 132950976172.png - (163.61KB , 1000x800 , derpysonic.png )
34178 No. 34178
Well, figured I should put a post in here to see if anypony's interested in this.

Basically I got annoyed over Christmas with ponibooru and other Shimmie/Danbooru clones all being pretty slow and not particularly clever. I write webapps as a bit of a hobby (compsci student) so figured I'd hack something together. is the result of that. It's a completely new bit of booru software which I'm going to be open-sourcing soon but I'm basically using derpibooru as a testbed and to kick the tyres. There's not a lot of content there (~3000 images at time of writing) but it's full of features and neat stuff, as well as being crazy fast.

Some highlights:

* Fast. Really fast. Written in a way which avoids it becoming slow with a large DB - it'll stay fast
* Scalable. See above. Avoids the messy relational DB approach, and can be sharded across multiple servers with ease
* Very flexible tag spoilering, hiding, and lots of user-tweakable view options including thumbnail sizes and images per page
* Fuzzy (and specific) tag searching capabilities
* Automatic exact and perceptual (fuzzy) deduplication of uploads (no more dupe posts!)
* Automagic import helpers from deviantArt, Tumblr, and other oEmbed-enabled sources, with artist tagging etc
* SVG support (with server-side PNG thumbnailing) and GIF animation support (for thumbnails, too)
* Server-side image optimization of PNGs, GIFs and JPEGs
* Intelligent resizing (combination of server and client side)
* Watched tag support, with automatic user-specific RSS feed generation and instant-update page
* Grid views for all image lists, which look like (and just try scrolling down to the end)
* Comments, voting, keyboard shortcuts, licensing for original content, and more!

I'd love to get some feedback to further improve on the software. And the booru itself exists and isn't going anywhere, so if anypony's interested in making use of it, go for it! Hope you enjoy. It's been a lot of fun writing the software.
Unspoiler all text  • Expand all images  • Reveal spoilers
>> No. 34189
File 132952389373.png - (447.02KB , 1631x902 , R - Tray-zhure.png )
That's all I needed to hear. I'll tell you if I run into any problems.
>> No. 34238
Fair enough. Speed's been a big concern throughout development. Most page requests are fulfilled within ~30-60 milliseconds, and we split up complex pages into multiple requests where it makes sense.
>> No. 34566
I'm encountering some issues when switching between pages. Trying to go to the next, previous, last or a specific page, I will sometimes be brought to a general search page.

For instance, instead of going to this page:

I'm redirected here:

Going back and retrying solves it. It occurs every now and then, not sure why.

Great job on this anyhoof, much faster than sites like ponibooru, and much easier to navigate.
>> No. 34582
File 133006985914.jpg - (59.23KB , 315x352 , Liar smile2.jpg )
I have this same problem.

Like this site so far, it's the first booru I've bothered to register on or upload something to.
>> No. 34629
Ah, that looks like a bug with the page scoping - I'll have a tinker and sort it out post-haste.


Glad you both like it, thanks for the feedback!
>> No. 34634
File 133014001815.png - (39.64KB , 501x501 , 132464900469.png )
very nice, I'll keep a tab on it
>> No. 34663
hm, it's even slower than PB over here in Hong Kong.
>> No. 34672
Hey, the page actually loaded! Already way ahead of ponibooru in quality~
>> No. 34802
How do you make it avoid slowness with a huge DB?
>> No. 34814
Well, that's mostly down to the database type. Relational databases are quite a poor tool for answering most of the questions we want to ask our data structures and for storing associations - things like querying for images based on tag inclusion/exclusion can get very complex in relational DBs like MySQL/PostgreSQL, as used by other booru software like danbooru and ponibooru (shimmie). MongoDB (the DB used in derpibooru) is a document DB, which lets us store array data objects alongside images and query based on arrays and hashes stored on those objects natively, rather than having to abstract it all to fit the DB. This means that "Show me all images except those tagged with x, y or z" is now a simple $nin query where in MySQL/PostgreSQL it's a few queries and a join/subquery, and our MongoDB queries stay linear where the relational DB queries grow with DB size.

That's a sort of basic top-level view of how it stays fast, anyway. On top of this fundamental difference in DB structure, we make use of HTTP fragment caching throughout - never serving stale data, but always avoiding rebuilding chunks of pages where we can just load it out from an in-memory cache for frequently grabbed chunks of page. We also do HTTP streaming to start delivery of page content before it's finished being generated server-side, and where pages contain multiple complex elements where we can break things up without disrupting user interaction we make multiple HTTP requests using AJAX to load things like comments and image view statistics. This along with server-side image resizing for huge images means that for end-users, things nip along a fair bit faster, in perceptual terms.

Hope that answers your question!
>> No. 34820
This booru-on-rails is a very impressive and humbling achievement, and I think I could learn quite a few things from you. I'd like to ask a question about these remarks:
> things like querying for images based on tag inclusion/exclusion can get very complex in relational DBs like MySQL/PostgreSQL
> in MySQL/PostgreSQL it's a few queries and a join/subquery
Would you happen to know if this inefficiency is endemic to relational databases in general (i.e. does it all happen, abstracted, beneath the surface of a single SQL query that uses AND/OR/LIKE to match text)? Or is this assessment based on the particular way in which the other boorus perform searches, i.e. with the multiple queries, joins etc. spelled out explicitly in the call(s) to the DB querying functions?

I ask because thinking of using Boolean mode fulltext searches in MySQL for tag searches in my project--
I'm mainly just trying to ascertain how well it would scale.

>> No. 34826
File 133048583628.png - (1.68MB , 1856x898 , derpibooru.png )
Just scrolled down to the end... and damn that took some while to accomplish! anyways great job with your webpage: i never liked boorus but this is quite an amazing job you did here
>> No. 34827
> dat picture at the very end

"I need pictures! Pictures of Ponies"

>> No. 34829
Thanks! Now to try and answer that...

If you store tags on an object merely as text strings then fulltext searching will scale relatively well, but the flexibility with which you can include/exclude tags will be limited. Not sure how well it'd scale with increasing tag query complexity, though - probably linearly, but I've not used MySQL fulltext enough to be sure.

Additionally, you have to pull out tag strings and match them to another DB table if you want to store tag metadata, which can add overhead at the application layer - for derpibooru's tags we can mark them as being system-wide spoilered/hidden by default and store custom spoiler images for each tag. This all gets loaded along with images very rapidly in a low-complexity way (because we're storing IDs in images and just doing quick "Give me a list of images, and include any tags referenced by them" queries, directly grabbing IDs).


You, sir, deserve a medal. Or something.
>> No. 34830
Thanks! That answers my question. I thought that would be so (inability to have tag metadata right there) but I think that I could find a way to easily make tags into links to their respective views (by putting the tag slug into the URL and then having the controller find it by that attribute).

Odd question, out of curiosity: how does MongoDB avoid redundancy, if there aren't any relations defined? Or does it simply store everything it needs to make up an image post in that one database record, sacrificing in the area of saving disk space?
>> No. 34831
Not sure what you mean by redundancy - the only redundancy as it were is in the keys. There is a relation, it's just implemented at the application layer, I suppose, depending on how you look at things. Specifically we store:

Image: image info, metadata, [tag ID, tag ID, tag ID], {hash of dedupe metadata}

Tag: ID, name, metadata

To make, for instance, the tag list that you get on mouseover of a thumbnail, we load the image (or rather already have the image), and from that we just ask for all tags which have an ID in the list from the image. We do filtering and spoilering the same way - users have several arrays of tag IDs on their collection. We don't have any relational models, effectively.
>> No. 34832
File 133049863820.jpg - (48.08KB , 501x525 , 're awesome.jpg )
No sir; you deserve all of the internetz
>> No. 34834
File 133050051255.jpg - (34.53KB , 500x500 , fspleased.jpg )
But how will I hold all these internets?

Thanks, anyway. Really glad people like it and hopefully will find some use in it. Mostly the trick is going to be getting people to upload to it, so... :-)
>> No. 34838
loving this web page
user friendly and quick too boot
heres too you Clover
Cheers mate
>> No. 34873
File 133058053865.png - (97.32KB , 347x658 , 132100111290.png )
Question to OP: are the users allowed to post rule 34 and grimdark (as long as is tagged properly) or do you want to keep it free of those kind of pics?
>> No. 34878
Not OP, but admin. All users can post anything MLP related as long as it's properly tagged.

We have 4 default tags for Suggestive (mildly sexualized), Questionable (clearly sexualized / mild violence), Grimdark (in regards to theme, possibly upsetting, etc), and Explicit (blatant r34, gore, etc).
>> No. 34879
Kinda ironic that you're calling it derpibooru and the main feature is that it's fast. If anything, something named after Derpy should be slower.
>> No. 34889
File 133062114184.png - (119.26KB , 516x519 , derpythinker.png )
Yonder are the rules: - upload what you like so long as it's not illegal in the United Kingdom. >>34878 guidelines on tagging are relevant, though - tag properly upon upload!

I like to think that Ms. Hooves has some hidden depths.
>> No. 35247
File 133135386783.png - (63.34KB , 500x500 , 131190887601.png )
I just went to grid view and zoomed out. Then, after the thumbnails had been scaled down, they suddenly moved about and rearranged themselves in a smooth and graceful motion, fitting more and more pony into one browser window.

Holy shit.

I mean, this is so incredibly awesome. It deserves far more attention. Have you considered requesting it be featured on Equestria Daily? You might catch the attention of some more skilled Ruby / RoR developers who would be able to make it even better.
>> No. 35275
Hah, never did try zooming out on the grid. That is a lot of pony. The reflow stuff is just a small side feature of the library used to achieve the grid layout, but neat nonetheless.

I've not really been fussed about throwing it at news outlets - DerpyHoovesNews featured it a while ago and I dropped EqD an email about it recently, but not too fussed either way. We'll see!
>> No. 35319
Does this have tag history? Because tag history is critical to the long-term health of a *booru.
>> No. 35321
File 133151027202.jpg - (74.42KB , 294x312 , dear-diary-today-op-was-awesome.jpg )
You sir win several internets

I imagine the next step would be creating a tool to convert danbooru style boards to derpibooru?
>> No. 35360
It has tag history, yes, though it's not very good. I'm going to improve that at some point. It's good enough to let you see who did what and revert changes, however.

Thanks. I don't think I'm going to write a converter, though. Tags differ too much, for one thing.
>> No. 35426
My only suggestion to you is this. Please, please, PLEASE FOR THE LOVE OF BABY MOSES HIMSELF, follow danbooru/gelbooru's tagging system. The Ponibooru tagging system is irritating to no end. If I wanted a picture of Twilight using levitation to move a book, I'd much rather search "twilight_sparkle book levitation" than to just search "twilight_sparkle" and sift through page after page of irrelevant images or risk missing what I'm looking for because the uploader couldn't tag for shit.

Just a suggestion, of course.
>> No. 35472
File 133175414084.jpg - (38.51KB , 500x500 , clover.jpg )
>> No. 35498
File 133179150134.png - (59.32KB , 501x736 , 131171385532.png )
> Mostly the trick is going to be getting people to upload to it, so... :-)
> ...
> I've not really been fussed about throwing it at news outlets - DerpyHoovesNews featured it a while ago and I dropped EqD an email about it recently, but not too fussed either way. We'll see!

I really wish I could have your 'tude - to not be worried about publicity too much, despite having built something pretty decent and wanting to see it get wider exposure.

Well, at any rate, you've done what you can. It's only a matter of time before Derpy takes over.

Page 0 bump!
[Return] [Entire Thread] [Last 50 posts]

Delete post []
Report post